Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Rayhana, Rakiba
Doctor of Philosophy - PhD
University of British Columbia
Water transmission networks are a community's most expensive and critical urban assets. Management of these pipelines is becoming one of the significant concerns as some of the water pipelines are at the end of their service lifespan. Nowadays, robotic platforms equipped with cameras and a wheeled system are used to videotape the internal state of the water pipelines. The acquired videos then require certified human personnel to assess the conditions of water pipelines manually. However, manual assessment is often error-prone, challenging, and expensive. Additionally, while inspecting the pipeline, the robotic platform experiences difficulties to pass through the valves installed inside the pipeline. This disrupts the inspection process and adds labor to a human operator. Hence, this research aims to mitigate the aforementioned issues of the inspection process by developing automated video data analytical frameworks to enhance inspection operations. At first, an extensive literature review was conducted, showing that automated frameworks display better performance when developed using deep learning-based techniques. Therefore, a deep learning-based defect characterization framework and a user-friendly defect characterization interface are developed to simultaneously detect, classify and segment the defects of water pipelines. The framework is developed by employing three residual networks (ResNet), a channel-spatial attention mechanism (CSA), Mask Canny-Regional Convolution Neural Network (MaskC-RCNN), and an ensemble mechanism. The results from this experiment depicted that the proposed model improved the defect detection rate and saved time for human operators. Then, a deep learning-based valve detection framework is developed to detect the valves by combining MobileNet-160 and Feature Pyramid Network (FPN). The experimental results and analysis shows that the developed algorithm detected the valves efficiently and accurately. Lastly, a real-time and super-lightweight valve detection algorithm with an edge device (NVIDIA Jetson TX2) is developed by combining MobileNetv3 and YOLOv4. The results from the proposed approach show a satisfactory performance and pave the door for its integration with the robotic platform. The research outcomes of this thesis contribute to the inspection process of the condition assessment of the water pipelines. The proposed algorithms can significantly improve the inspection process and aid in the decision-making process of pipe replacement/renewal plans.
VIDEO DATA ANALYTICS FOR THEAUTOMATION OFWATER PIPELINEINSPECTIONbyRakiba RayhanaB.Sc., BRAC University, 2015M.Sc., The University of Strathclyde, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)December 2022© Rakiba Rayhana, 2022The following individuals certify that they have read, and recommend to the Col-lege of Graduate Studies for acceptance, the dissertation entitled:VIDEO DATA ANALYTICS FOR THE AUTOMATION OF WATER PIPELINEINSPECTIONsubmitted by Rakiba Rayhana in partial fulfillment of the requirements of the de-gree of Doctor of Philosophy.Dr. Zheng Liu, School of EngineeringSupervisorDr. Rehan Sadiq, School of EngineeringSupervisory Committee MemberDr. Jahangir Hossain, School of EngineeringSupervisory Committee MemberDr. Mohamed Shehata, Department of Computer Science, Physics, Mathematics,and StatisticsUniversity ExaminerDr. Lucio Soibelman, Sonny Astani Department of Civil and EnvironmentalEngineering, University of Southern CaliforniaExternal ExamineriiAbstractWater transmission networks are a community’s most expensive and critical urbanassets. Management of these pipelines is becoming one of the significant concernsas some of the water pipelines are at the end of their service lifespan. Nowadays,robotic platforms equipped with cameras and a wheeled system are used to video-tape the internal state of the water pipelines. The acquired videos then requirecertified human personnel to assess the conditions of water pipelines manually.However, manual assessment is often error-prone, challenging, and expensive.Additionally, while inspecting the pipeline, the robotic platform experiences diffi-culties to pass through the valves installed inside the pipeline. This disrupts the in-spection process and adds labor to a human operator. Hence, this research aims tomitigate the aforementioned issues of the inspection process by developing auto-mated video data analytical frameworks to enhance inspection operations. At first,an extensive literature review was conducted, showing that automated frameworksdisplay better performance when developed using deep learning-based techniques.Therefore, a deep learning-based defect characterization framework and a user-friendly defect characterization interface are developed to simultaneously detect,classify and segment the defects of water pipelines. The framework is devel-oped by employing three residual networks (ResNet), a channel-spatial attentionmechanism (CSA), Mask Canny-Regional Convolution Neural Network (MaskC-RCNN), and an ensemble mechanism. The results from this experiment depictedthat the proposed model improved the defect detection rate and saved time forhuman operators. Then, a deep learning-based valve detection framework is de-iiiveloped to detect the valves by combining MobileNet-160 and Feature PyramidNetwork (FPN). The experimental results and analysis shows that the developedalgorithm detected the valves efficiently and accurately. Lastly, a real-time andsuper-lightweight valve detection algorithm with an edge device (NVIDIA JetsonTX2) is developed by combining MobileNetv3 and YOLOv4. The results fromthe proposed approach show a satisfactory performance and pave the door for itsintegration with the robotic platform.The research outcomes of this thesis contribute to the inspection process of thecondition assessment of the water pipelines. The proposed algorithms can signif-icantly improve the inspection process and aid in the decision-making process ofpipe replacement/renewal plans.ivLay SummaryRecently, robotic platforms have been employed to inspect and videotape the in-ternal condition of water pipelines. The videos are then manually assessed byhuman operators to identify the defects. The robotic platforms also face issueswhile traversing the valves, interrupting the videotaping process. Therefore, thisresearch analyses the acquired video data to facilitate the condition assessmentprogram of the water municipalities.Firstly, a deep-learning-based framework and a user interface are developedto characterize the defects from the acquired video files. This reduces the bur-den on human operators and saves significant time. Then, an automated valvedetection framework is developed to detect the valves before the robotic platformapproaches the valves. This helps the robotic platform to ensure uninterruptedinspection. Lastly, a real-time valve detection with an edge device frameworkis developed to be integrated with the robotic platforms to detect the valves inreal-time and enhance the inspection process.vPrefaceThis thesis is based on the research work conducted in the School of Engineeringat The University of British Columbia (Okanagan campus) under the supervisionof Dr. Zheng Liu. I conducted major parts of the research, including the literaturereview, problem formulation, algorithm development, and manuscript writings.Dr. Zheng Liu helped me with the research ideas, suggestions, and feedback toimprove the quality of the manuscripts. Mrs. Angie Wu from Pure Technologies-Xylem guided me with her expertise in the water pipeline industry, which facili-tated me in identifying the research gaps. Ms. Yutong Jiao joined the discussionon the research and gave her valuable feedback on the manuscripts.Chapter 2 is based on the following published paper and used with permissionof IEEE:• R. Rayhana, Y. Jiao, A. Zaji, and Z. Liu, “Automated vision systems forcondition assessment of sewer and water pipelines,” IEEE Transactions onAutomation Science and Engineering, vol. 18, no. 4, pp. 1861 - 1878,2020.Chapter 3 is based on the following submitted paper:• R. Rayhana, H. Yun, and Z. Liu, X. Kong “Automated Defect Detec-tion System for Water Pipelines based on CCTV Inspection Videos of Au-tonomous Robotic Platforms,” IEEE/ASME Transactions on Mechatron-ics. (Status: Under Review).viChapter 4 is based on the following published papers and used with permissionof IEEE and SPIE:• R. Rayhana, Y. Jiao, Z. Bahrami, Z. Liu, A. Wu, and X. Kong, “Valve de-tection for autonomous water pipeline inspection platform,” IEEE/ASMETransactions on Mechatronics, vol. 27, no. 2, pp. 1070 - 1080, 2022.• R. Rayhana, Y. Jiao, Z. Bahrami, Z. Liu, A. Wu, and X. Kong, “SmartValve Detection System for Water Distribution Networks,” 22nd IEEE In-ternational Conference on Industrial Technology (ICIT), vol. 1, pp. 1176–1181, 2021.• R. Rayhana, Y. Jiao, Z. Liu, A. Wu, and X. Kong, “Water pipe valve detec-tion by using deep neural networks”, Smart Structures and NDE for Indus-try 4.0, Smart Cities, and Energy Systems, vol. 11382, pp. 1138205, SPIE,2020.Chapter 5 is based on the following published paper and used with permissionof Springer:• R. Rayhana, Y. Jiao, Z. Liu, A. Wu, and X. Kong “Real-time embeddedsystem for valve detection in water pipelines,” Journal of Real-Time ImageProcessing, vol. 19, pp. 247–259, Springer, 2021.Other publications that I authored and co-authored are summarized as follows:• R. Rayhana, G. Xiao, Z. Liu, “Printed sensors for Monitoring Applicationsin Smart Farming: A Review,” IEEE Transactions on Instrumentation &Measurement, vol. 70, pp. 1-19, September, 2021.• R. Rayhana, G. Xiao, Z. Liu, “RFID Sensing Technologies for Smart Agri-culture,” IEEE Instrumentation & Measurement Magazine, vol. 24, no.3, pp. 50-60, 2021.vii• R. Rayhana, G. Xiao, Z. Liu, “Internet of Things Empowered Smart Green-house Farming,” IEEE Journal of Radio Frequency Identification, vol. 4,no. 3, pp. 195–211, 2020.• Y. Jiao, R. Rayhana, J. Bin, Z. Liu, A. Wu, and X. Kong, “A steerable pyra-mid autoencoder based framework for anomaly frame detection of waterpipeline CCTV inspection” Measurement, vol. 174, pp. 109020, Elsevier2021.• L. Jian, R. Rayhana, L. Ma, S. Wu, Z. Liu , H. Jiang, “Infrared and Vis-ible Image Fusion Based on Deep Decomposition Network and SaliencyAnalysis,” IEEE Transactions on Multimedia, 2021. (Early Access).• Z. Bahrami, R. Zhang, R. Rayhana, and Z. Liu, “An HRCR-CNN Frame-work for Automated Security Seal Detection on the Shipping Container,”IEEE Transactions on Instrumentation & Measurement, vol. 70, pp. 1-13, 2021.• H. Yun, R. Rayhana, S. Pant, M. Genest, and Z. Liu, “Nonlinear ultrasonictesting and data analytics for damage characterization: a review,” Measure-ment, vol. 186, pp. 110155, Elsevier, 2021.• J. Lin, J. A. Ferna´ndez, R. Rayhana, A. Zaji, R. Zhang, O. E. Herrera,Z. Liu, and W. Me´rida, “Predictive analytics for building power demand:day-ahead forecasting and anomaly prediction”, Energy and Buildings, pp.111670, Elsevier, 2021.viiiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Problem Statement and Motivation . . . . . . . . . . . . . . . . . 41.3 Research Questions and Objectives . . . . . . . . . . . . . . . . . 51.4 Thesis Structure and Organization . . . . . . . . . . . . . . . . . 72 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9ix2.1 Robotic Platforms for Water Pipeline Inspection . . . . . . . . . . 102.1.1 PIRAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1.2 MAKRO . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.3 PANORAMO . . . . . . . . . . . . . . . . . . . . . . . . 122.1.4 Responder . . . . . . . . . . . . . . . . . . . . . . . . . . 122.1.5 KANTARO . . . . . . . . . . . . . . . . . . . . . . . . . 132.1.6 Sahara . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.1.7 PipeDiver . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2 Machine Vision Inspection Systems for Water Pipelines . . . . . 152.2.1 CCTV Technique . . . . . . . . . . . . . . . . . . . . . . 162.2.2 Zoom Camera Technique . . . . . . . . . . . . . . . . . . 172.2.3 Laser-based Scanning Technique . . . . . . . . . . . . . . 182.2.4 Digital Scanning Technique . . . . . . . . . . . . . . . . 182.2.5 Electro Scanning Technique . . . . . . . . . . . . . . . . 192.3 Video Data Analytics for the Robotic platforms . . . . . . . . . . 192.3.1 Defect Detection in Water Pipelines . . . . . . . . . . . . 202.3.1.1 Traditional Defect Detection Techniques . . . . 222.3.1.2 Deep Learning based Defect Detection . . . . . 242.3.2 Valve Detection in Water Pipelines . . . . . . . . . . . . . 262.3.2.1 Traditional Object Detection Techniques . . . . 272.3.2.2 Deep Learning based Object Detection Techniques 292.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 Automated Defect Characterization for Water Pipelines . . . . . . . 373.1 Defect Characterization . . . . . . . . . . . . . . . . . . . . . . . 383.2 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 413.3 Proposed Framework . . . . . . . . . . . . . . . . . . . . . . . . 423.3.1 Image Pre-processing Module . . . . . . . . . . . . . . . 423.3.2 Defect Characterization Module . . . . . . . . . . . . . . 433.3.2.1 Channel-Spatial Attention (CSA) Module . . . . 443.3.2.2 MaskC-RCNN . . . . . . . . . . . . . . . . . . 46x3.3.2.3 CSA-MaskC-RCNN . . . . . . . . . . . . . . . 473.3.2.4 Ensemble Mechanism . . . . . . . . . . . . . . 503.3.3 Defect Characterization User Interface . . . . . . . . . . . 503.4 Experimental Results and Analysis . . . . . . . . . . . . . . . . . 513.4.1 Model Implementation . . . . . . . . . . . . . . . . . . . 523.4.2 Model Visualization . . . . . . . . . . . . . . . . . . . . . 563.4.3 Comparative Studies . . . . . . . . . . . . . . . . . . . . 573.4.4 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . 603.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644 Valve detection for robotic water pipeline inspection platform . . . 664.1 Valve Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2 Proposed Framework . . . . . . . . . . . . . . . . . . . . . . . . 694.2.1 Data Pre-processing Module . . . . . . . . . . . . . . . . 694.2.2 MFPN Module . . . . . . . . . . . . . . . . . . . . . . . 714.2.3 Valve Detection Module . . . . . . . . . . . . . . . . . . 744.3 Experimental Results and Analysis . . . . . . . . . . . . . . . . . 754.3.1 Data Acquisition and Description . . . . . . . . . . . . . 754.3.2 Model Training and Testing . . . . . . . . . . . . . . . . . 764.3.3 Comparative Studies with State-of-the-Arts . . . . . . . . 774.3.4 Model Visualization and Ablation Studies . . . . . . . . . 804.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825 Real-time Valve Detection System with Edge Device . . . . . . . . . 865.1 Real-time Valve Detection System with Edge Device . . . . . . . 875.2 Proposed Framework . . . . . . . . . . . . . . . . . . . . . . . . 875.2.1 Image pre-processing . . . . . . . . . . . . . . . . . . . . 885.2.2 Real-time valve detection algorithm . . . . . . . . . . . . 895.2.3 Edge Device . . . . . . . . . . . . . . . . . . . . . . . . . 935.3 Experimental Results and Analysis . . . . . . . . . . . . . . . . . 955.3.1 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . 96xi5.3.1.1 Precision . . . . . . . . . . . . . . . . . . . . . 975.3.1.2 Recall . . . . . . . . . . . . . . . . . . . . . . . 985.3.1.3 F1-Score . . . . . . . . . . . . . . . . . . . . . 985.3.1.4 Intersection over Union (IoU) . . . . . . . . . . 985.3.2 Results and Comparative Analysis for State-of-the-art Al-gorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 995.3.3 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . 1005.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1026 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109xiiList of TablesTable 2.1 Differences between popular non-visual techniques . . . . . . . 16Table 2.2 Types and features of the defects inside the water pipelines [1] . 21Table 3.1 Data set description for each types of defects and features . . . 54Table 3.2 Confusion Matrix (CM) for testing set . . . . . . . . . . . . . . 55Table 3.3 Comparative results of our proposed model and state-of-the-artmodels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Table 3.4 Performance assessment of the proposed model with state-of-the-art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Table 3.5 Ablation studies for the proposed model. . . . . . . . . . . . . 65Table 4.1 Ablation studies for the proposed network . . . . . . . . . . . 82Table 4.2 Performance evaluation with the state-of-the-art models . . . . 85Table 4.3 Performance evaluation for the valve detection models . . . . . 85Table 4.4 False Positives, False Negative and Performance on the videosobtained from robotic platforms . . . . . . . . . . . . . . . . . 85Table 5.1 Specifications for NVIDIA Jetson TX2™ . . . . . . . . . . . . 93Table 5.2 Initialization parameters of YOMO for valve detection . . . . . 96Table 5.3 Confusion Matrix (CM) for typical classification task . . . . . 98Table 5.4 Performance evaluation for the proposed real-time valve detec-tion algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 102xiiiTable 5.5 Performance evaluation for valve detection in NVIDIA JetsonTX2™ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102Table 5.6 Performance assessment (with respect to time) between themanual and automated detection system . . . . . . . . . . . . . 102xivList of FiguresFigure 1.1 The inspection procedure of water pipeline © Rakiba Rayhana 3Figure 1.2 Steps of the post-processing of the inspection videos of waterpipelines © Rakiba Rayhana . . . . . . . . . . . . . . . . . . 3Figure 1.3 The structure of the thesis. . . . . . . . . . . . . . . . . . . . 8Figure 2.1 Different types of robotic platforms for pipeline inspection[1]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Figure 2.2 Principle of Electro Scanning [2, 3]. . . . . . . . . . . . . . . 20Figure 2.3 Architecture of a typical CNN © Rakiba Rayhana . . . . . . . 25Figure 2.4 Steps for traditional object detection framework © Rakiba Ray-hana. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 3.1 The overview of the proposed automated defect characteriza-tion framework © Rakiba Rayhana . . . . . . . . . . . . . . 43Figure 3.2 The illustration of CSA module [4] . . . . . . . . . . . . . . . 49Figure 3.3 The illustration of the proposed framework ©Rakiba Rayhana 49Figure 3.4 The Defect Characterization User Interface (DCUI) (Imagesreproduced with the permission from the Pure Technologies-Xylem, Mississauga, ON, Canada). . . . . . . . . . . . . . . . 52Figure 3.5 The Defect detector window (Images reproduced with the per-mission from the Pure Technologies-Xylem, Mississauga, ON,Canada). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53xvFigure 3.6 Training and validation loss curve of the proposed model ©RakibaRayhana . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Figure 3.7 Visualization of the last convolutional layer of “Conv4” blockfor ResNet50, ResNet101, and ResNet152. (a) The last layerof “Conv4” block for ResNet50 (b) The last layer of “Conv4”block for ResNet101 (c) The last layer of“Conv4” block forResNet152 (d) The last layer “Conv4” block after attention isadded for ResNet50 (e) The last layer of “Conv4” block afterattention is added for ResNet101 (f) The last layer of“Conv4”block after attention is added for ResNet152. (The images areresized for the purpose of illustration) ©Rakiba Rayhana. . . . 58Figure 3.8 Receiver operating characteristic (ROC) curve for the pro-posed model and state-of-the-art models ©Rakiba Rayhana . 60Figure 3.9 Precision Recall (PR) curves of the proposed model and state-of-the-art models©Rakiba Rayhana . . . . . . . . . . . . . . 62Figure 3.10 Experimental results from the proposed model. From top tobottom: corrosion, crack, outlet, metal loss and joints defectsare presented. From left to right: (a) original image (b) groundtruth (c) masks (d) predictions. Note that the images are re-sized for the purpose of this illustrations. ©Rakiba Rayhana . 63Figure 3.11 Results for different types of defects. From left to right: (a)corrosion (b) crack and corrosion (c) broken pipes (d) corro-sion (e) outlet and pipe joint. Note that all the images areresized for the purpose of this illustration. ©Rakiba Rayhana . 64Figure 4.1 The video acquisition platform, PipeDiver®. (Images repro-duced with the permission from the Pure Technologies-Xylem,Mississauga, ON, Canada) . . . . . . . . . . . . . . . . . . . 67Figure 4.2 The automated valve detection framework [5] . . . . . . . . . 70Figure 4.3 Some of the synthetic valve images generated by the modifiedBigGAN [5] . . . . . . . . . . . . . . . . . . . . . . . . . . . 72xviFigure 4.4 An illustration of the architecture for the proposed valve de-tection system [5] . . . . . . . . . . . . . . . . . . . . . . . . 74Figure 4.5 The training and validation loss curve for the proposed frame-work [5] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77Figure 4.6 Some of the valve testing sample results for our proposedmodel [5]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Figure 4.7 Visualization of the feature maps for the DCNN architecturefor an image containing valve (Images are resized for this il-lustration) [5]. . . . . . . . . . . . . . . . . . . . . . . . . . . 81Figure 4.8 The filters for the corresponding DCNN layers (Images areresized for this illustration)[5] . . . . . . . . . . . . . . . . . 82Figure 4.9 Comparison of the Receiver Operating Characteristics (ROC)graph for the proposed model [5]. . . . . . . . . . . . . . . . 83Figure 4.10 Comparison of the Precision Recall (PR) curve for the pro-posed model [5]. . . . . . . . . . . . . . . . . . . . . . . . . 84Figure 5.1 Overview of the real-time valve detection system with theedge device ©Rakiba Rayhana . . . . . . . . . . . . . . . . . 88Figure 5.2 The proposed architecture (YOMO) for real-time valve detec-tion with edge device ©Rakiba Rayhana . . . . . . . . . . . . 90Figure 5.3 An illustration of the prediction process for valve detectionwith YOMO ©Rakiba Rayhana and [6] . . . . . . . . . . . . 92Figure 5.4 NVIDIA Jetson TX2 module (Image Courtesy: NVIDIA [7])[ USB: Universal Serial Bus, M.2 E: Next Generation FormFactor, HDMI: High Definition Multimedia Interface, SD: Se-cure Digital, PCIE: Peripheral Component Interconnect Ex-press, STATA: Statistical Package, JTAG: Joint Test ActionGroup, UART: Universal Asynchronous Receiver-Transmitter 94xviiFigure 5.5 End-to-end pipeline of NVIDIA Jetson TX2 [7] [LIDAR: LightDetection and Ranging, RF: Radio Frequency, USB: Univer-sal Serial Bus, PCIE: Peripheral Component Interconnect Ex-press, MIPI CSI: Mobile Industry Processor Interface and Cam-era Serial Interface, ISP: Internet Service Provider, H.265:High Efficiency Video Coding, CUDA: Compute Unified De-vice Architecture, cuDNN: Compute Unified Device Archi-tecture Deep Neural Network, Open GL: Open Graphics Li-brary, CANbus: Controller Area Network bus]. . . . . . . . . 95Figure 5.6 The training and the validation loss curves of the proposedmodel ©Rakiba Rayhana . . . . . . . . . . . . . . . . . . . . 97Figure 5.7 The precision recall (PR) curves of the models ©Rakiba Ray-hana. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100Figure 5.8 Some testing samples for the real-time valve detection withedge device, Jetson TX2 ©Rakiba Rayhana . . . . . . . . . . 101xviiiGlossaryANN Artificial Neural NetworkAUC Area Under the CurveCANbus Controller Area Network BusCBAM Convolutional Block Attention MechanismCCTV Closed-circuit TelevisionCNN Convolutional Neural NetworkCSP Cross Stage PartialCUDA Compute Unified Device ArchitectureCuDNN NVIDIA CUDA® Deep Neural Network libraryDCNN Deep Convolutional Neural NetworkDI Ductile IronDNN Deep Neural NetworkDPM Deformable Part-based ModelEPA Environmental Protection AgencyFC Fully ConvolutionalxixFCN Fully ConvolutionalNetworkFPN Feature Pyramid NetworkFPS Frame Per SecondGAN Generative Adversarial NetworkGPU Graphics Processing UnitHDMI High Definition Multimedia InterfaceHMM Hidden Markov ModelHOG Histogram of Oriented GradientHSB Hue, Saturation and BrightnessH.265 High Efficiency Video CodingIoU Intersection over UnionISP Internet Service ProvideJTAG Joint Test Action GroupLBP Local Binary PatternsLIDAR Light Detection and RangingmAP mean Average PrecisionMIPI CSI Mobile Industry Processor Interface and Camera Serial InterfaceM.2 E Next Generation Form FactorNASSCO National Association of Sewer Service CompaniesNMS Non-Maximum SuppressionxxOpen GL Open Graphics LibraryPACP Pipeline Assessment Certification ProgramPCA Principal Component AnalysisPCIE Peripheral Component Interconnect ExpressPCCP Pre-stressed Concrete Cylinder PipePIG Pipe Inspection GaugePIRAT Pipe Inspection Rapid Assessment TechniquesPVC Polyvinly ChlorideRCNN Region-based Convolutional Neural NetworkResNet Residual NetworkRBF Radial Basis FunctionRBN Radial Basis NetworkRDA Redundancy AnalysisReLU Rectified Linear UnitRF Random ForestRGB Red, Green, BlueROC Receiver Operating Characteristic CurveROI Region of InterestRPN Region Proposal NetworkSD Secure DigitalxxiSE Squeeze and ExcitationSGD Stochastic Gradient DescentSIFT Scale Invariant Feature TransformSPP-Net Spatial Pyramid Pooling NetworkSSD Single Shot-Multibox DetectorSTATA Statistical PackageSVM Support Vector MachineUAV Unmanned Aerial VehicleUART Universal Asynchronous Receiver-TransmitterUPVC Unplasticised Polyvinyl ChlorideUSB Universal Serial BusVAE Variational Auto EncodersVGG Visual Geometry GroupYOLO You Only Look OncexxiiAcknowledgmentsI want to express my deepest gratitude to my supervisor Dr. Zheng Liu, for hisconstant support, guidance, and encouragement in conducting this research. Ithank him for his careful supervision, suggestions, and valuable insights, whichmotivated me to be enthusiastic about my research. It was an immense honor forme to work under his supervision.I am also grateful to Dr. Rehan Sadiq and Dr. Jahangir Hossain for being onmy supervisory committee. I want to thank Dr. Mohamed Shehata for serving asthe university examiner. I want to thank Dr. Lucio Soibelman for his willingnessto serve as my external examiner. I appreciate their valuable suggestions, time,and critical comments on my research and thesis.I want to thank my colleagues at the Intelligent Sensing, Diagnostics, andPrognostics Research Lab (ISDPRL). Especially, I would like to thank Ms. Yu-tong Jiao for being an excellent colleague and for all her support during my Ph.D.journey.Pure Technologies-Xylem, Canada, and NSERC are acknowledged for theirfinancial support through the “Collaborative Research and Development Grants”(CRDPJ 523761 - 18). I want to thank Mrs. Angie Wu, and Dr. Xiangjie Kongfrom Pure Technologies-Xylem, Canada, for their kind assistance in conductingthis research.xxiiiDedicationI dedicate this thesis to my beloved family, especially my mom, dad, siblings,husband, mother-in-law, father-in-law and my sister-in-laws. Without their un-conditional love, prayers, encouragement, and support, I could never pursue mydream. Thanks to all my friends who supported me during my Ph.D. journey.xxivChapter 1IntroductionThis chapter presents the research background, motivation, objectives, and thecorresponding research tasks to complete this research.1.1 BackgroundWater, one of the invaluable assets of human existence, is distributed to house-holds through a vast array of pipelines. As the pipe materials age, they are proneto deterioration, leading to pipe breakage, water loss, contamination, and eventhreatening public health. During 2012 to 2018, the total mains break rate forwater pipes in the United States (US) and Canada has increased from 11.0 to 14.0breaks/ (100 miles)/year [8]. It is estimated that by 2050, the US would exceed$1.7 trillion of investment [8] for the replacement of the water mains. This im-plies that the conditions of most of the water pipelines are degrading and are onthe verge of failure.With economic constraints and new regulatory policies, the water utility opera-tors are now moving from the active repair/ rehabilitation process toward proactivemaintenance methods. Therefore, there is an urge to develop advanced conditionmonitoring and assessment techniques for effective asset management and prior-itization of the maintenance schedules [9]. Generally, the condition assessment1of water pipelines requires pipe information, including construction information,maps, valves, and past inspection records. Once the information is obtained, a riskassessment is carried out, which estimates how much risk is associated with theintervention and if the pipes require urgent inspection. Based on the risk assess-ment, in-pipe inspection is carried out by employing a various range of inspectiontechnologies to understand the internal condition of pipelines [10]. Traditionally,flow tests and steady-state pressure analysis techniques are employed, as both arecapable of identifying the general problems for the pipelines. However, thesetechniques pose a limitation in providing in-depth knowledge and accurate in-ternal condition of the pipes (types of fault, location, etc.) [9]. To overcomethe aforementioned problems, direct methods, which includes visual inspection,pipe sampling, and non-destructive testing, have been included in the proactivemaintenance routine [11]. Amongst all the techniques mentioned above, visualinspection is the most popular technique for the systematic condition assessmentof water pipelines as it can provide comprehensive knowledge of the internal con-dition of the pipelines [12]. Visual inspection is an effective technique as it candetect the macroscopic flaws in pipelines and composite structures [13]. It is con-sidered as a simplest and a reliable form of non-destructive testing as it requiresless interpretive skills, time and cost to perform the analysis [14].At present, robotic platforms (also known as in-pipe inspection tools) areequipped with cameras and optical fibers are visually used to inspect the waterpipelines [15]. Fig.1.1 shows a typical inspection process for the water pipelines.The advantage of these inspection tools is that they can operate inline withoutinterrupting the service to the consumers [16]. It can traverse through small di-ameter pipelines and videotape the internal conditions of the pipe with the aid ofhigh-resolution cameras and an illumination system. Once the run-time for therecording of the inspection tool is completed, the acquired video data are thentransferred to the municipal office for post-processing. The post-processing ofthe files is manually assessed to identify the distress indicators by the inspectionoperators [17]. Fig. 1.2 in the following shows the steps of the post-processing of2the inspection videos. The operators carefully observe each frame of the videos,and halt to assess and record the structural discrepancies in the pipes. This manualassessment to post-process the video data are often error-prone, expensive, labor-intensive, and time-consuming. Hence, there is a high demand for automation inthe post-processing of the video data obtained from the cameras of the inspectiondevices. This will not only aid the inspection process for the operators, but it cansave time and money and achieve higher accuracy and consistency in the faultdiagnostics process.Figure 1.1: The inspection procedure of water pipeline © Rakiba RayhanaThis research aims to develop automated frameworks for water pipeline in-spection process by using the video data obtained from the robotic platform. Theprimary focus has been placed on the investigations of algorithms and techniquesfor deep learning. The proposed automated algorithm aims to overcome the defectcharacterization and navigational issues of the robotic platforms.Figure 1.2: Steps of the post-processing of the inspection videos of waterpipelines © Rakiba Rayhana31.2 Problem Statement and MotivationUntil recently, the condition assessment of the water pipeline was limited by thelack/delay of the regular inspection cycles, which mainly occurred due to the im-proper maintenance plans and budget constraints [18]. A standard, comprehen-sive, and reliable inspection can identify the extensive degradation and can inhibitsevere damage and unreasonable expenses [19]. The US Environmental Protec-tion Agency (EPA) has mandated the municipal authorities to provide regular in-spection reports of the buried pipelines so that the operation and maintenance is-sues can be tackled immediately [20]. Therefore, developing a reliable automatedinspection technique, which can use the visual inspection data/videos to interpretthe internal conditions of the water pipelines efficiently, is crucial.Visual inspection of the internal conditions of the pipelines is conductedthrough the robotic platforms, which are equipped with camera sensors and anillumination system. The robotic platforms are launched in the water pipeline viaan access point or fire hydrants, which then traverses along the water pipelinesto videotape the internal condition of the pipelines. However, while the roboticplatform is traversing through the pipelines, it often faces difficulties to negotiatethrough the butterfly valves. Butterfly valves are installed after a certain distancein water pipelines, so that the flow of the water can be controlled. Sometimes,valves are even used to isolate a complete section of pipeline for maintenance orrepair issues. As robotic platforms experience issues to traverse the valves, it in-terrupts the inspection process and it generates number of motionless videos. Thehuman operator then manually analyze the recordings to analyze defects, debris,or attempts to understand the sudden interruption in the video recording process.The post-processing process of the videos require long hours, which delay theoverall condition assessment process. Throughout these years, the automated vi-sual inspection process for water pipelines has not developed as much as it hasdeveloped for the sewer and wastewater infrastructure. This indicates that there isa good scope to improve the post-processing techniques for the water infrastruc-ture. However, the existing models to post-process the videos have some major4limitations. The limitations are listed in the following.• There is no comprehensive automated defect characterization frameworkdeveloped so far for water pipeline inspection. Till date, the assessmenthas been manually carried out by the expert pipeline operators. The man-ual inspection process is often error-prone, expensive, tedious, and time-consuming.• It can not provide any information about the navigational problems, such asobstacles like valves, during the video capturing process. The obstacles inthe navigation process interrupt not only the video recording but also gen-erate a huge number of static (still) video files, which increases the burdenon the pipeline inspectors.• The valve’s status, i.e., whether the valve is open/closed/partially closed,remains unknown for the robotic platform until it reaches the valves. Thiscreates a delay in activating the control mechanisms of the robotic platform,which interrupts the video recording and the inspection process.• The current post-assessment process of the videos is conducted offline. Theoffline assessment can not provide the information to the robotic platformto make intelligent decisions, such as activating the control mechanismthrough the pipeline.1.3 Research Questions and ObjectivesThis research aims to assist the robotic platforms with advanced data analyticcapabilities (intelligence) for inspection interpretation and system operation. Theaforementioned limitations can be defined with the following research questions.• Can an efficient and reliable automated defect characterization frameworkto detect and classify the defects for water pipelines is feasible to design?5• Can the valves in the water pipelines be detected from the videos obtainedfrom the robotic platforms ?• Can a real-time video data analytic solution with an edge device applied tovalve detection is feasible to develop so that robotic platform can maintainuninterrupted video recording (inspection) process?To address the aforementioned research questions, the objectives of this researchis defined as the following.• To characterize the defects inside the water pipelines, which will providethe information regarding defect types, location, and geometrical shapes ofthe defects so that the pipeline operators can accurately identify the defectsand schedule the maintenance plan.• To enhance the navigational process of the robotic platforms by identify-ing the valves and status of the valves (open/partially open/closed) so thatthe robotic platforms can ensure the smooth operation for water pipelineinspection.• To develop a portable solution to integrate the developed algorithms withan edge device for real-time valve detection. Integration of the edge devicewith the robotic platforms can facilitate the robots to become intelligent andimprove the pipe inspection process.The research objectives will be achieved by completing the following tasks.• By developing an efficient, reliable, and automated deep learning-based de-fect characterization framework from the inspection videos, so that the con-dition assessment process for the water pipeline is enhanced. This wouldhelp to reduce the burden of the pipeline inspectors and save time.• By developing a deep learning-based valve detection algorithm so that thevalves and the status of the valves (open/close/partially open/close) can6be detected from the inspection videos. This would help the robotic plat-form activate the control mechanism on time and assist in the uninterruptedvideotaping process.• By developing lightweight algorithms for both valve detection frameworksso that they can be deployed in an edge device to achieve a portable solutionof the developed frameworks. Integration of the edge device with the roboticplatforms can improve the decision support system for automated conditionassessment of the water pipeline.This study can help to enhance and facilitate the inspection process of thewater pipelines. This will also improve the overall condition assessment processand asset management for the water utilities.1.4 Thesis Structure and OrganizationChapter 1 of this thesis includes the background studies, problem statement, re-search motivation, research objective, and overview of the research methodologiesto complete this research. Chapter 2 describes the literature review for the con-dition assessment of the water pipelines, highlights the research gaps, and pro-vides potential solutions to overcome the current problems. Chapter 3 providesthe framework for defect detection in water pipelines. Chapter 4 provides theframework for automated valve detection in water pipelines. Chapter 5 presentsthe system implementation of real-time valve detection with edge device. Lastly,Chapter 6 presents the conclusions of this research and discusses the key findingsof this study. Figure 1.3 in the following presents a flowchart for structure of thisthesis.7Figure1.3:Thestructureofthethesis.8Chapter 2Literature ReviewThe management of underground buried utilities has been a constant interest forthe municipalities as buried pipes are highly vulnerable to unanticipated circum-stances such as alternation in soil pressure, pipe material, decay, and so on. Watertransmission networks, one of the buried utilities, are one of the nation’s mostessential and exorbitant infrastructures. A regular inspection can avoid unwanteddisasters (pipe bursts, flooding, etc.). Nowadays, advanced data acquisition toolsand visual techniques are developed to interpret and understand the internal con-dition of water pipelines. In the meantime, the evolution of computer-vision tech-niques on video analysis has played a significant role in automating water pipedefect detection. However, the literature survey shows few studies on automatedwater pipe defect defection and analysis compared to the sewer and waste-waterpipelines.This chapter is a minor modification of the paper R. Rayhana, Y. Jiao, A. Zaji,and Z. Liu, “Automated vision systems for condition assessment of sewer andwater pipelines,” IEEE Transactions on Automation Science and Engineering,vol. 18, no. 4, pp. 1861 - 1878, September, 2020, and it is reproduced here withthe permission of IEEE, and the copyright is reserved with the authors.92.1 Robotic Platforms for Water PipelineInspectionRobotic platforms comprise of the structural aspect of pipes, sensors, effectivecommunication, and an efficient energy management system. Some of these as-pects are in the full implementation stage regarding structural and sensor designs[12]. However, research is still going on to establish effective communication,such as using the cloud as automated communication systems [21] and makingthe robots more energy efficient [1, 12].Studies depict that the first robotic system was developed in the 1970s [12, 22].Based on that, different robotic system designs have been developed. There aretwo types of motions for the robotic platforms: active and passive. Active motionrefers to the approaches with one or more driving mechanisms according to theirdesigned structure. Active motions include screw types, wheeled, walking, track-ing, snake, and inch-type robotic platforms. Figure 2.1 shows the different typesof robots. On the other hand, passive motion refers to situations where the flowof fluids drives the platform. Pipe Inspection Gauge (PIG) is one of the examplesof passive motion, which helps to clean the internal parts of the pipes throughgrazing, and carries instruments and sensors inside the pipelines [12]. Some pop-ular active motioned in-pipe inspection tools for sewer and water pipelines aredescribed below. However, tools that are used for pipeline rehabilitation and re-placement will not be covered in this review [1].The main objective of this section is to provide a general idea of how the re-search could address the challenges for the condition assessment of sewer andwater pipelines and how the robotic platforms are evolving with time and havebeen advanced with diverse technologies to tackle those challenges.2.1.1 PIRATPIRAT (Pipe Inspection Rapid Assessment Techniques) was developed byCSIRO, an Australian and Melbourne Water company, in the year of 1995 [23].10Figure 2.1: Different types of robotic platforms for pipeline inspection [1].PIRAT consists of multiple integrated systems, such as machine vision, artificialintelligence, and robotics. The PIRAT aims to develop an automatic sewer in-spection system. It is a compact, self-driving device equipped with scanners andsensors to evaluate the geometry inside the sewer pipelines. The device has beendeveloped to operate under harsh (e.g., explosive and hazardous) environments.The device has advanced data communication technology and uses a scanner toassess the condition of the pipelines [24]. Also, it can auto-generate defect reportsfrom the collected data by identifying the faults of the sewer pipes. The CSIROcompany has just developed a prototype with this technique and no product hasbeen manufactured for commercial purposes so far [1, 24].2.1.2 MAKROMAKRO was developed by a group of German researchers in the year 2000 [25],to conduct automatic sewer inspections. This device automatically detected thefaults’ location, type, and size in sewer pipelines. MAKRO is a tethered devicethat is made up of six segments. Such flexible structure enables it to glide throughthe pipe as a snake and subdues any obstacles a probe may experience duringits deployment [24]. It is mainly designed to operate in the range of 300 mmto 600 mm [12] pipe diameter. It is also equipped with an ultrasound sensor,and an optical zoom camera [26], which measures an approximate distance of theobstacles, such as blockages in pipes, the wall thickness, and cracks, even if the11pipes are embedded with a layer of mud. Additionally, it can detect the externaldamage of pipes up to 10 cm behind the pipe wall and the leakage pits in the soils.However, MAKRO is not a commercially available tool in the market, and it wasdeveloped to operate only as a prototype [1, 24].2.1.3 PANORAMOPANORAMO was developed in 2003. PANORAMO is a device that uses anoptical scanner with two combined scanning units. One of the scanners is placed atthe rear, and the other is placed at the front. Both scanners are equipped with highdefinition (HD) cameras and a fish-eyed lens of 186 degrees. The two scannerstake hemispherical images and generate 360-degree spherical profiles [27].The three-dimensional perspective of the pipe allows the operator to adjust theangle of view in different areas. It does not require the camera to operate duringthe inspection period, and the data analysis is often conducted in the office. Themain advantage of PANORAMO is the virtual camera, which can be used by theoperator just like a real camera is used. Also, the speed of the device is 0.3 m/s,which is higher than many of the inspection tools [27].2.1.4 ResponderResponder was developed in 2005 by the RedZone company to perform conditionassessments for different types of pipelines. It is designed to operate in remoteconditions to inspect large diameter sewer pipelines or tunnels in service [28]. Itis equipped with a 3-D LIDAR scanner, sonar, gas and temperature sensors, anddigital CCTV, and it is capable of collecting pipe data in the most challengingconditions [28]. An operator then assesses the data thoroughly to identify anyfaults. The weight of the Responder is almost 300 kg, and it can traverse nearly2,400 m in a single run [12]. Similar to the Responder, RedZone also offersSolo, MD sub, MD profiler, ICOM, and integrated multi-sensor platforms such ascamera, laser, and sonar to perform the condition assessment of pipelines.122.1.5 KANTAROKANTARO was developed in 2007 in Japan (Kyushu Institute of Technology).It is a passive-active intelligent, untethered prototype with intelligent sensors andmodular structure [29]. KANTARO is designed for 200 mm and 300 mm diame-ters [12], with a flexible structure that allows it to bend and pass through obstacles.It can detect special landmarks, such as inlets and maintenance holes. A fish-eyedcamera and a 2-D scanner are attached to the device to identify the condition andfault of the pipelines. Also, the software is used to process the data collected bythe robot. The software can classify nine different types of defects. KANTAROcan identify the invaded substance, crack, root intrusions, water infiltration, gaps,dislocations, break, landslides, and mortar breakages [12].KANTARO now has a new updated version, which has an embedded comput-ing system and an emergency release mechanism. This is named as KANTAROII. Using an embedded computing system has enabled KANTARO to reduce thedevice’s size and operate with lower power, making it a more reliable device forinspection [30].2.1.6 SaharaSahara is a pipe inspection device that has been commercially brought to the mar-ket by the Pressure Pipe Inspection Company (PPIC). It is mostly used for activewater mains inspection. It is a passive-active device that has a tethered sensor.Once the device is inserted inside the pipeline, a parachute is deployed, whichmoves along the pipe with the water flow to activate the sensor. Real-time re-sults allow maximizing the sensitivity through the tethered sensor. Sahara can beplaced in any pipeline with a diameter greater than or equal to 50 mm [31]. Re-cently, the Sahara platform has adopted video technology and named it as “Saharavideo.” This additional feature enables the operators to see inside the pipelineswithout de-watering and expensive shutdown of the networks. Sahara video iseasily inserted in live pipelines via tap greater than or equal to 50.8 mm and canrecord up to approximately 1829 m inside pipelines. It helps to identify debris and13obstacles, valves, and leaks to assess the internal condition of pipe walls, such ascorrosion within the 500 mm of the device [31]. It can also determine any illegalconnections that are made from the mains transmission system [32].However, Sahara has limitations with the diameter length and the pipes’ tightbending. The maximum turn or bending it can bear is 270 degrees before frictioncauses any intrication. Also, the equipped sensor has limitations similar to anyacoustic listening sensors [12].2.1.7 PipeDiverPipeDiver was first displayed in 2009 at a No-Dig Show by PPIC. The first gen-eration was a prototype using the flow of fluids to navigate. Three modules wereused to track the vehicle and supply power to carry the assessment in pipelines[11]. Nowadays, the second-generation PipeDiver is commercially used by PureTechnologies to perform condition assessments in water pipelines. PipeDiver is anuntethered pipe wall assessment tool installed in major pipelines that are difficultto remove from services. The second-generation is deployed similar to Sahara,and it is equipped with a camera to record videos inside the pipes. It has a flexi-ble fish-like structure that can locate corrosion, dents on metal pipes, and brokenwire enfolds. The device can traverse across the apertures and valves. The batteryhours for PipeDiver is 30 hours, which helps it to traverse 29 miles in a singledeployment. The data for condition assessment is collected through the camerasensors. Additionally, electromagnetic sensors are used to facilitate the data col-lection and to identify the conditions when the pipe is on the verge of failure [33].Similar to Sahara, PipeDiver avoids the hassle of shutting down the networkand de-watering, making the device more cost-effective. It is relatively safe andhas assessed more than 1000 pipelines around the world [33].Apart from the aforementioned robotic platforms, many companies and in-dividual researchers have exploited the vision-based design of the robot for thecondition assessment of sewer and water pipelines, such as Prokasro, Reduct, andIspek [12]. One common issue for the robotic platforms is their capability for14autonomous navigation, especially through the pipelines’ T, elbows, and bendingzones. This is because the current robotic platforms are primarily propelled bythe flow of the liquid inside the pipelines. The traveling speed and body-centeringcontrol of the robots remain a challenge for field operation. Also, the currentrobotic platforms are dependent upon trained operators for the deployment andevaluation process.Therefore, future robotic platforms should be able to carry multiple sensorsto provide relevant and reliable information about the networks to inspectors andoperators. The robotic platforms should also be equipped with an autonomousnavigation system to reduce the hours of human labor and perform a comprehen-sive and reliable condition assessment of the pipelines.2.2 Machine Vision Inspection Systems for WaterPipelinesGenerally, condition assessment of pipelines can be classified into three categoriese.g., internal surface inspection, pipe structural assessment, and bedding materialassessment. The internal surface assessments can be categorized into non-visualnondestructive testing (NDT) methods and visual NDT methods. Non-visual NDTmethods include ultrasonic inspection, sonar, eddy current testing, and groundpenetrating radar (GPR) [34]. The difference between popular non-visual tech-niques are presented in Table 2.1.The visual NDT methods is one of the widely used inspection technique for toinspect the inner surface of the water pipeline. The advantage of visual inspectionis that it is cheap, easier to interpret, and better for decision making of pipe reha-bilitation [11].. The visual inspection include closed-circuit television (CCTV),SSET, laser-based scanning, zoom camera, digital scanning technique, and elec-troscanning. This section presents the specifications, limitations, and tolerancelevel of the well-known visual systems used to evaluate the internal conditions/-faults of the pipelines with the expert knowledge of the operators [45]. As thisresearch is focused on the automated vision system for the condition assessment15Table 2.1: Differences between popular non-visual techniquesTechnology Applications Limitations Suitableformetallicpipes?Suitablefor con-cretepipes?Ultrasound Location of thedefects can be de-termined [34]Prior cleaning ofpipes to removedebris and wax arerequired [34]. Itis also expensive[34]Yes [11] No [11]Sonar andlaserInternal profileof pipes can bedetermined [34].Pipe deflectionand corrosionof pipes can beestimated [34]It cannot operate inboth medium (airand water) simul-taneously [34]Yes [11] Yes [11]Eddy cur-rentSuitable for smalldiameter pipes[34]Estimation of skindepth of pipes arevery difficult [34]Yes [11] Yes [11]GroundpenetratingradarConditions of ex-ternal pipes canbe estimated [34]Requires skilledoperator [34]Yes [11] Yes [11]of internal surface of water pipelines, only visual NDT techniques of internal sur-face assessment techniques will be covered.2.2.1 CCTV TechniqueClosed-circuit television (CCTV) was introduced in the 1960s [35], and it is nowconsidered as one of the widely used visual inspection techniques to assess bothwater and sewer pipelines [36]. CCTV inspections are generally administered byusing a distant employed system and with a portable platform, such as Sahara,PipeDiver, and Responder, which are accompanied by high illumination settings16[37]. When the robotic platforms equipped with CCTV are placed inside thepipeline, they move along the pipelines and records the internal condition of thepipelines. Once the flight/run for the inspection is completed, the video files arethen transferred to the external ground facility. Generally, the speed of the plat-form is maintained at 15 cm/s [11], and data for site location can be retrieved.Also, the standard coding system provided by NASSCO Pipeline AssessmentCondition Protocol (PACP) is used to record the defects [1, 37].CCTV is one of the cheapest techniques to inspect the water pipelines’ con-dition. Recently, different types of CCTV cameras, such as High-Definition(1080P), and ultra-High-Definition (4K), have been commercially available. Cur-rently, the inspection results are manually analyzed through expert personnel. Thismakes the process tedious, error-prone, and time-consuming [11].2.2.2 Zoom Camera TechniqueZoom camera was developed to serve as a screening device that could preemi-nence the pipes for maintenance. In contrast to conventional CCTV, this camerais fastened on a fixed arrangement, which could zoom down the pipe up to 15.24m to 30.48 m without any movement along the pipelines. The zoom camera canrecord both videos and still images of the pipelines, providing preliminary inter-nal conditions of the pipelines. The main advantage of this technology is that itcan provide better video/image qualities than CCTV. It is cost-efficient, suitablefor all types of pipe materials, and can offer faster service for inspectors [38].The limitation of the zoom camera is that it cannot provide substantial visualfeatures, such as fault locations or pipe details like what CCTV does. It cannotbe used underwater, and it has constraints on measuring the distance from a faultlocation [38]. Studies show that zoom camera has been used to inspect a few feetthrough sewer pipelines and maintenance holes. However, the latest zoom cameracan go deep down the pipelines and also has a pan of 360 degrees [39].172.2.3 Laser-based Scanning TechniqueLaser-based scanning (also known as LiDAR) is one of the effective techniques todetect the finer faults, which are often missed by CCTV or other visual techniques[40]. It can detect cracks, corrosion, and missing bricks by reflecting the laserlight on the surface of the pipes. If there is any crack in the pipes, the laser lightwill be reflected at reduced intensity. The information from laser scans is readilyrecorded and evaluated simultaneously with computers. This reduces the chanceof having errors in detection.The limitation of laser-based scanning is that the equipment cost is higher thanthe CCTV and other visual systems. Hence, its usage is limited to the inspectionof pipelines.2.2.4 Digital Scanning TechniqueDigital scanning was developed to anticipate a detailed visual assessment ofpipelines above the water surface by using High Definition (HD) cameras [41].It uses multiple high-resolution cameras to capture the rear, sideways, and frontalviews of the platform. The videos are recorded with parallel illumination settingsand with extensive lens angle [42] to produce definitive images. In the upgradedversion of digital scanners, some are equipped with a consigned gyroscope, incli-nation meter, and bent capacities [43].One important issue with its performance is that the resolution decreases withan increased diameter of pipeline [38]. Other than that, digital scanning is de-ployed as a CCTV is deployed inside a pipeline. Likewise, pre-processing of thevideo/images or defect codes is done with the aid of software in the office, whichpermits the inspectors to virtually pan, zoom, and tilts the camera for fault detec-tion. The software generates color codes to categorize each class of faults [44]182.2.5 Electro Scanning TechniqueElectro scanning technique is mostly used for concrete, reinforced concrete, brick,plastic, and clay pipes (all non-metallic pipes) [45]. The electro scan can deter-mine the location and the degree of the possible leakages. A potential difference isapplied between the electrode, called “sonde,” which is placed between the pipes,and another sonde placed on the surface of the ground. The two sondes gener-ate a high electrical resistance, and a current flow occurs through the pipe wall.However, an interruption in the current flow occurs if there is any fault, such as acrack, dislocation of the welded joints, or shape distortion. This interruption canbe monitored through a computer system by providing a spectrum of signals orspikes in the electrical signal found between the sondes, which can identify thefault types such as cracks, dislocated joints, pipe features, and other faults that oc-cur due to maintenance hole and service connections [46, 47]. The amplitude andthe shapes of the signals aid in understanding the severity of the faults. However,to understand which current signals belong to the normal condition, the operatorneeds to have prior knowledge and experiences [2, 3].Nevertheless, there are some limitations to this technique. Electro scanningcan not identify any faults that occur within the external part of the pipelines.Also, it can only detect faults that are submerged underwater. Therefore, to useelectro scanning, one needs to fill up the pipes with water at the location wherethe sondes are installed. The current flow between the two sondes is recorded atevery 12.7 mm distance. It can detect faults for both sewer and water pipelines[38].2.3 Video Data Analytics for the Robotic platformsRecently, there has been an increasing trend to use computer vision-based algo-rithms to analyze the video data obtained from the robotic platforms [48]. Thealgorithms aid in automating the assessment process of the inspection videos withhigher accuracy, speed, and consistency [49]. Once the videos are obtained, hu-19Figure 2.2: Principle of Electro Scanning [2, 3].man experts assess the videos to detect defects and other discrepancies. Thissection reviews the automated defect detection analysis and obstacle detection inwater pipelines.2.3.1 Defect Detection in Water PipelinesThe purpose of the machine vision system is to inspect and collect imaging-basedinformation on the internal condition of the pipelines. One of the essential com-ponents of the pipeline inspection process is to identify the defects, types, and lo-cation of the defects inside the pipelines [1]. Defects of the pipelines are definedas the event at which the pipe failure takes place [50]. The pipelines with differ-ent geometry, construction, and transfer materials are subject to varying types ofdefects [1].The diameter of the water pipelines usually ranges from 12 mm - 3.65 m [1].The most popular construction material for water pipelines are cast iron, copper,steel, polyvinyl chloride (PVC), unplasticized PVC (UPVC) concrete, reinforced20concrete pipes, asbestos cement, and fired clay. The deterioration of the pipesis mainly dependent upon the designed material and size of the pipelines. How-ever, natural calamities or the sudden alteration in pressure inside the soil caninduce leakage, dislocation, cracks, and corrosion. Sometimes, the deteriorationmay occur due to human intrusion, leading to permanent damage to the pipelines.Therefore, the inspection robotic platforms, equipped with CCTV cameras/scan-ners, are used to identify and locate the defects inside the pipelines [1].Generally, pipe defects in water pipelines can be classified into five main cate-gories. Table 2.2 in the following presents the categories with a brief description.Table 2.2: Types and features of the defects inside the water pipelines [1]Defect Types Defect FeaturesLeakageOutflow of the liquid through the surfaceor through the holesCorrosionAccumulation of hydrogen sulphide gason the surface of cast iron/metallic pipes,which forms the sulphuric acid to corrodepipesDeformationAlteration in the shape of pipeline due tosoil pressureDislocationDisplacement of the pipe joints(elbows/ T section)CracksDevelopment of the fracture line aroundthe circumference or axial length of pipesBy definition, pipe defects are easy to define, but they are very challengingfor pipeline inspectors to identify in practical scenarios. This is because the pipedefects have random shapes and sizes, making it difficult to categorize them intoa particular defect class. The details of the water distribution networks are un-certain, especially the pipes’ properties and the discontinuities’ location, makingdefect detection a challenging process [51]. To mitigate these issues, automatedapproaches, especially computer-vision technologies, are considered as solutions.Therefore, this section will review the up-to-date computer-aided defect detection21techniques for buried pipelines.2.3.1.1 Traditional Defect Detection TechniquesThe videos/images captured by the robotic platforms are often highly subjectiveto the noise and illumination conditions inside the water pipelines. Hence, theacquired images require to undergo image pre-processing operations to preparethe data for further analysis. Traditionally, image processing algorithms are em-ployed to pre-process the image by removing noise, obtaining defect features, andresizing the input images [52].Image Enhancement: Generally, the image enhancement process is conductedto accomplish an improved perception for both human and computer-aided anal-ysis [53]. The image enhancement process helps to visualize the distinct featuresof the pipe defects. Over time, various image enhancement filters have been em-ployed, but the most popular pipe image enhancement filters were bi-lateral de-noising filters [54], unsharp masking [55], and Gaussian filters [56]. A bilateraldenoising image filter smooths out the images without compromising the edges ofthe images [53, 54]. The unsharp masking removes the noise and artifacts in theimage. It also performs a histogram adjustment technique to enhance the contrastof the images [53, 56]. Gaussian filter is a 2D filter that helps to remove the noiseby obtaining the weighted mean of each pixel in the neighborhood. The pixelsthat are far away from the central pixel are assigned in smaller weight, which canclearly define the edges of the images [56]. In addition to the above-mentionedfilter, details on other filters can be found in [57].Edge Detection: The edge detection technique aims to provide information re-garding any variations or discontinuities in the photo-metrical or geometricalshape or the characteristics of the objects present in the image [58]. It is usuallyperformed by calculating the rate of change in the intensity of gray level betweeneach pixel both in the vertical and horizontal direction [59]. Some popular edge22detectors are Sobel [60], Canny [61], Hough transform [62], and so on [63]. Nu-merous studies were conducted based on edge detection techniques to identify thedefects in the pipelines. Amongst all the defects, crack detection remained to beone of the popular fields to study [64]. This is because crack is often challengingto identify as it can appear very fine thin line in structure. Thus, the backgroundnoises in the acquired images from the robotic platforms are filtered out to detectdefects and cracks. Then, the edges are detected with the edge detection algo-rithm, such as the Sobel method. Hough transform is then used to remove theedges of the images. After that, morphological-segmentation techniques are usedto identify the crack-containing regions. Once the area is identified, filters areadded to suppress the noise of the images and to extract useful information aboutcrack [65].Morphological Segmentation: Morphological segmentation is one of thewidely used image processing techniques, which is used to obtain useful inter-pretation. In the morphological segmentation technique, dilation and erosion arethe two essential operators [66, 67]. Iyer and Sinha [68] and Sinha and Fieguth[69] employed opening operation by using erosion and dilation operators togetherto obtain opening top-hat images to obtain pipe defects. The images are thensegmented to identify the patterns, crack, lateral joints, and hole using Otsu’stechnique [68, 69]. Yang and Su [70] also used morphological segmentation tech-niques to identify crack, fracture, broken, and open joints of pipes from the CCTV-based images. Su et al. [71] presented a morphological segmentation based onedge detection (MSED) to correctly segment and identify the sewer pipe defectseffectively. Many hybrid approaches, which include edge detection, and contoursegmentation, were also presented in [66, 72]. Besides these techniques, thresh-olding [73] and thinning [69] techniques are also used for segmenting pipe im-ages. Pixel-based segmentation techniques are also employed to interpret the pipedefects. Mashford et al. [74] proposed automatic pipe inspection techniques byutilizing a pixel-based detection. Pixel-based segmentation segments the image23in terms of the pixel connectivity, and it is grouped according to the connectivityand labeled with respect to the gray level pixel. The difference between the in-tensity of pixels and brightness provides a threshold to segment the images. Thesegmented pixels are then identified by using colors. However, one limitation ofpixel-based detection techniques is that it does not consider the pixel values of thelocal neighborhood and is not very accurate in segmentation.2.3.1.2 Deep Learning based Defect DetectionFor the past few years, the trend of research has moved to enhance the algo-rithms’ accuracy, speed, and robustness. Hence, deep learning-based defect de-tection algorithms gained enough attention to solve the aforementioned problems.Convolutional Neural Networks (CNN) or Deep Convolutional Neural Networks(DCNN) are employed to perform automatic defect detection in buried pipelines.CNN comprises of convolutional layers, pooling layers, fully-connected lay-ers, and activation functions. At first, the input images are fed to the networkand passed to the convolutional and pooling layers. The rich semantic featuresof the pipe defects are extracted with each convolutional layer and pooling layer.For instance, if a two-layer CNN is employed, the first layer extracts the low-levelfeatures, and the second layer extracts the high-level features. The fully connectedlayers then stack the extracted features and flatten the features into a feature vec-tor. Afterward, the network uses activation functions to estimate the defect classprobabilities. Some common activation functions are sigmoid, Rectified LinearUnit (ReLu), leaky-ReLu, and softmax. Fig. 2.3 presents a typical CNN architec-ture to detect the defects.A deep convolutional neural network (DCNN) framework to detect the defectsinside the concrete pipelines was proposed by Cha et al. [75]. The experimentalresults showed a tremendous improvement from the prior studies. Meijer et al.[76] proposed a DCNN-based “leave-two-inspections-out” technique, which aimsto reduce overfitting issues when fewer images are available. A fused algorithmwas proposed by Duran et al. [77], which combined image processing algorithms24Figure 2.3: Architecture of a typical CNN © Rakiba Rayhanawith the neural networks to obtain the defects inside the sewer pipelines. However,the training time and the computational resources for the model were very high. Toovercome these issues, region-based proposal networks were proposed to use. Asdiscussed before, region-based proposal networks are developed with RCN. TheR-CNN uses the selective search method to extract useful features. The extractedfeatures are then fed onto the SVM classifier, which classifies and generates aprediction score for each of the defects. The only problem with R-CNN is themodel is very slow as each stage needs to be trained separately [78].To combat the aforementioned issues, Fast R-CNN was proposed by Girshick[79]. This helped to enhance the processing time, but the detection score or theaccuracy was still not reliable as the results were heavily dependent upon the se-lective search method. Therefore, Faster R-CNN was developed to enhance theperformance. Faster R-CNN aided in reducing the computational cost and im-proved the detection accuracy score [80]. A study employed the Faster-RCNN[49] framework to categorize the multiple faults (cracks, deposition, and root in-trusion). The experimental results were evaluated in terms of mean Average Pre-cision (mAP), precision, recall, and F1-score, and they were obtained as 86.2%,90.6%, and 87.7%, respectively. Another study found in [81] employed FasterR-CNN to classify the defects and process the CCTV images at high speed. Theframework was able to train 3,000 frames, and the performance was satisfactoryin terms of the evaluation metrics. The authors also proposed a hyperparameteroptimization algorithm, which increased the overall accuracy up to 83%.25Recently, Hassan et al. [82] proposed an automatic defect detection and recog-nition framework for sewer pipelines. The model was trained with a large numberof CCTV images (48,274). The result concluded that DCNN is better than thehand-crafted feature engineering techniques and, with other optimization tech-niques, has excellent potential to enhance the detection results.Nowadays, deep learning techniques are popularly used to solve defect detec-tion problems in water pipelines. However, to obtain the high accuracy of theimages, a large number of annotated data set is required. Due to the complexityof the inspection process, obtaining high data becomes a significant issue in someinstances. Hence, a comprehensive framework that can perform satisfactory per-formance should be developed to improve the defect detection process for pipelineinspectors.2.3.2 Valve Detection in Water PipelinesIn the last two decades, the robotic platforms for pipeline inspection have focusedon designing robotic prototypes. While the design of the robotic platforms hasprogressed very well, the autonomous navigation of the robotic platforms remainsas a challenge to solve. Most of the in-pipe inspection robotic platforms operatewith limited sensors and knowledge of the network structure of the pipelines be-forehand [83]. This causes the robotic platforms to be stuck in a place, especiallyin front of the valves, which controls the water flow of the pipelines and interruptsthe autonomous navigation along the water pipelines. Therefore, there is a needto develop an autonomous valve detection system so that autonomous navigationis enhanced.With the advent of computer vision, especially deep neural networks, sev-eral autonomous frameworks have been developed for different applications. Tothe best of our knowledge, this is the first study to detect the butterfly valves in-side water pipelines. For this study, we considered the valve an object detectionproblem. Therefore, in this section, only computer-aided algorithms for objectdetection will be reviewed.26Over the past two decades, the object detection algorithms have undergonetwo historical generations, namely traditional object detection generation (before2014) and convolutional neural network-based object detection generation (after2014) [84]. Hence, this section reviews the state-of-the-art object detection tech-niques and applications that are used to solve different types of problems in waterand a closely related field such as sewer.2.3.2.1 Traditional Object Detection TechniquesThe aim of the object detection techniques in an image is to identify the target ob-ject’s accurate information and mark the object’s precise location by employingimage processing and pattern detection algorithms [85, 86]. The traditional objectdetection techniques were hand-crafted feature engineering. Fig. 2.4 in the fol-lowing shows the flowchart of the conventional object detection framework. Thetraditional hand-crafted object detection models could be categorized into threephases, namely, (a) generation of the candidate regions, (b) extraction of features,(c) classification. A brief description of each of the phases is described in thefollowing.Figure 2.4: Steps for traditional object detection framework © Rakiba Ray-hana.Generation of the candidate regions: In this phase, the location of the targetobject is obtained. The multi-scale sliding window is employed as the target ob-ject can appear in any of the images with varying size and aspect ratios. Themulti-scale sliding window slides through each part of the image with a series of27aspect ratios to detect the possible locations. However, this method is relativelyinefficient as it has many redundant windows, which increases the framework’scomplexity. Besides, the sliding window has a high computational cost which caninfluence the performance of the next phase, which is the extraction of the features[87, 88].Extraction of the features: The extraction of visual information and featuresare essential to detect different types of object to ensure a robust and semanticrepresentation [88]. However, extracting the robust visual features becomes chal-lenging as the features are subjective to the change in illumination and the varyingenvironments [87]. To overcome this issues, some hand-crafted feature extractiontechniques such as, Scale-Invariant Feature Transform (SIFT) [89], Local BinaryPatterns (LBP) [90], Histograms of Oriented Gradients (HOG) [91], Haar-likefeatures [92], and Haar-Wavelet [93] is used. Guo et al. [19] employed SIFT todetect defects in wastewater pipelines. Su et al. [94] proposed morphological-based edge detection techniques to detect the defect in sewer pipelines. Halfwayet al. used HOG to perform the feature extraction of the defects in the sewerpipeline. The studies found in [95–99, 99–101] show that the performance ofthe aforementioned feature descriptors is quite reasonable to solve simple appear-ance, backgrounds, and in uniform illumination conditions. Still, these descriptorsmight not be suitable for solving complex environmental issues (multiple objects,varying illumination, etc.).Classification: After the feature extraction phase is completed, the next step isclassifying the objects to detect. Therefore, a classifier is required to categorize thesimilar types of objects in the image so that the representations are more semantic,informative, and hierarchical for the visual identification [88]. The most popularclassifier includes AdaBoost [102], Support Vector Machine (SVM) [103], Ran-dom Forest (RF) [104], and Deformable Part-based Model (DPM) [105]. Studiesshow that Support Vector Machine (SVM) [106–110] and Random Forest (RF)28[106, 111–113] have been widely used in water pipe industries as the classifier,which is trained with the pre-labeled data for detection analysis. For the SVMclassifiers, the margin between each prediction class to the pre-trained models ismaximized by using the hyperplane parameters in higher dimensional space andthe lower dimension of a kernel. The popular kernels that are used with SVM arethe radial basis, polynomial and sigmoid (non-linear kernels) [111]. The trainedclassifier predicts the unseen frame by evaluating the position relative to the hy-perplane dimensional space. The faulty classes are generated through the relativedistance between the hyperplane to estimate the class probability of each type offault. The number of decision trees is used for the RF classifier to predict theunseen frames by voting. The proportion of the voting for each class defines theestimated probability of the class identifying the faults [114].However, the aforementioned hand-crafted feature engineering often lackedgeneralization ability and robustness [87]. The generation of the candidate regionsand the bounding boxes through the sliding window approach are inaccurate, in-efficient, and redundant. This results in a semantic gap in the object detectionprocess, as the integration of poor feature descriptors to train shallow frameworksresults in poor detection performance [88].2.3.2.2 Deep Learning based Object Detection TechniquesThe neural network was first proposed in the 1940s [115] to solve learning prob-lems by modeling the human brain system. During the 1980s-1990s, the neuralnetwork gained popularity with the back-propagation algorithm [116]. However,in the early 2000s, the popularity of deep neural networks dropped due to a lackof extensive data, insignificant performance compared to other algorithms, com-putation power, and resources [88].In 2006 [117], the neural network, especially the deep neural network, be-came a popular research field. Deep neural networks (DNNs) have a deeper layerof convolutional neural network layers. It can quickly learn a network’s com-plex features and have better generalization ability than the conventional machine29learning algorithms [118]. With the advanced high-performance computing tech-nology such as parallel computing (GPU clusters), availability of large-scale an-notated data [119], and significant improvement in neural network frameworks,the deep neural network has revolutionized over other traditional algorithms forautomated tasks.A convolutional neural network (CNN) is one of the typical models to de-fine deep learning [118]. A CNN comprises many layers, and each of the layersrepresents the feature map. The feature maps of the input layer are defined asa 3-Dimensional (D) matrix of the pixels obtained from the three color chan-nels (RGB). The feature maps obtained in the internal layers can be considered‘multi-channel images,’ where each pixel defines a trained image feature. Thetwo common transformation matrices that are applied to the feature maps are fil-tering and pooling layers [120, 121]. Filtering, also known as the convolutionlayer, employs convolution between the learned weight (matrix) and receptivematrix of the neurons and obtains the final response with the aid of a non-linearfunction (sigmoid, leaky ReLU, ReLU). The pooling layer employs the averagepooling, max-pooling, L2-pooling, and contrast normalization to generate the ro-bust feature descriptors [88]. After the pooling layer, depending upon the tasks,several Fully Connected (FC) layers with activation functions are added to ob-tain the probability from each neuron. The model is then optimized by defininga loss function (cross-entropy, mean-squared error, etc.) with a gradient descentfunction [88].The advantage of using CNN over conventional algorithms is that high-levelsemantic information can automatically be captured through multi-level hierar-chical representation. The deeper network increases the learning ability andcan construct better feature maps [122]. The increased generalization helps tosolve complex problems such as image classification [123–125], video analysis[126, 127], image retrieval [128, 129], face recognition [130, 131], pedestrian de-tection [132, 133] and so on. For object detection tasks, the deep neural networkscan be classified into (a) region-proposal based detection technique (b) regres-30sion/classification based techniques. A brief description of the relevant literaturestudies regarding the application of the algorithms is discussed in the following.Region Proposal-Based Detection Techniques: The region proposal-based ob-ject detection framework is a two-step process by generating the region proposalnetworks and then focuses on the Region of Interest (ROIs) in the image. Regionproposals networks were proposed by Uijlings et al. [134] to mitigate the candi-date region issues of the traditional object detection techniques. Regional proposalnetworks use the edge, color, and texture information to generate the possiblelocations of the object in the images. This helps to reduce the redundancy, com-plexity of the models and improve the recall and accuracy of the detection process.Some of the most popular regional proposal models are category-independent re-gion proposals [135], constrained parametric min-cuts (CPMC) [136], selectivesearch [134], edge boxes [137], etc.In 2012, Alexnet et al. [138] led a breakthrough of object detection algorithmsby introducing the deep convolutional neural network (DCNN) to solve feature ex-traction and image classification problems. However, the CNN framework had theobject localization issue. To overcome this issue, Girshick et al. [78] proposed aframework that combines the region proposal network with the features of convo-lutional neural networks (R-CNN). The selective search algorithm was employedin the R-CNN for region proposals to solve the localization problem. The CNNswere used to extract the feature maps from each of the regions, and then the ex-tracted feature maps were sent to a classifier, namely SVM [78]. Although theperformance of R-CNN was better than the traditional algorithms, the algorithmwould require a fixed input size of the image. This would result in a loss of scaleand aspect ratio information. Besides, the feature extraction process was tediousas it would take 50 s to process each image. Therefore, this framework was notsuitable for training larger image datasets.To overcome the shortcomings of R-CNN, Kaiming et al. [139] developed aspatial pyramid pooling network (SPP-Net), which scans the whole image only31one time and generate the corresponding feature maps. It employs parallel com-putation, which allows the image to process (24-64 times) faster than R-CNN. Aspatial pyramid pooling layer is used at the final convolutional layer. The loca-tion of the candidate windows and the generated feature maps are fed to the lastconvolutional layer. The features are then connected with the FC layers, and itcan preserve the global and local information with higher accuracy than R-CNN[139]. The limitation of SPP-Net was that it takes almost the same to completethe training process or the training pipeline. So, it requires additional memoryto store the information, which increases the computation cost. Besides, the finalconvolutional layer can not update the weights. This causes a good amount ofdrop in the accuracy values.To tackle the problems mentioned above, Girshick et al. [79] proposed Fast-RCNN, which included a multi-task loss function for each classification categoryand a bounding box regression values. It can also process the images like theSPP-Net and extract the features from the regional proposal via Region of Interest(ROI) pooling layers. ROI layers are single pyramid layer, which takes the featuremaps and splits into two output layers. The first output layer generates softmaxprobabilities for each class of the object and the second output layer obtains therefined bounding boxes with four coordinate values [79]. Although Fast-RCNNreduces the computation cost and improves the accuracy, the candidate box gener-ation with a regional proposal with selective search and edge box methods impactsthe detection speed of the object in the images. Therefore, Ren et al. [80] pro-posed Faster-RCNN, which uses a fully convolutional network (FCN) to developthe region proposal networks. This improves the model’s overall performance andmakes it one of the efficient techniques for object detection. Inspired by the greatsuccess of Faster-RCNN, Cheng and Wang employed Faster-RCNN to detect thedefects such as root intrusion and deposits inside the sewer pipelines. This studyachieved a mean average precision (mAP) (also known as accuracy) of 83%. Ku-mar et al. [140] also applied Faster R-CNN to localize the sewer defects in thesewer pipelines. The results from their study could achieve 74.5% of mAP.32Regression/Classification-Based Detection Techniques: The region-proposal-based detection techniques comprise: (a) region-proposal generation, (b) featureextraction by using CNN, (c) classification, and (d) bounding-box regression, andeach of them is trained individually. Although Faster R-CNN achieves a good per-formance in terms of processing speed of image and detection accuracy, the timeto process the multi-stage pipeline is not efficient to conduct real-time detection[88]. Therefore, one-step architectures developed with regression or classifica-tion models, which can directly map the extracted features to the class probabil-ities and bounding box coordinates values, can be used for real-time application.The two most popular one-step object detection frameworks that are used in thewastewater pipeline industries are You Only Look Once (YOLO) [141] and SingleShot-Multibox Detector (SSD) [142].In 2016, Redmon et al. proposed You Only Look Once (YOLO). The YOLOemploys the top layer of the feature maps and directly maps it to calculate the con-fidence scores for each class and bounding boxes. The main idea of YOLO is tocreate several windows (small partitions) of the images by S× S grid dimension.Each of the defined grids then enumerates bounding boxes and the confidencescore of the object in that particular grid. The YOLO framework comprises 24convolution layers and 2 FC layers. The network can process images at a veryhigh frame rate per second, making it suitable for real-time applications. In 2017,Redmon et al. [143] proposed YOLOv2, an improved version of YOLO. YOLOv2included batch normalization, dimension cluster, anchor boxes, and multi-leveltraining. This improved the accuracy and processing speed of YOLO. In 2018,YOLOv3 [144] was proposed, which was an incremental improvement of theaccuracy. Redmon et al. [144] offered a new feature extractor by combiningYOLOV2, Darknet-19, and residual network. This could help to extract the low-level features and drastically improve the accuracy and efficiency of the model.Kumar et al. [140] employed YOLOv3 to perform defect detection in the sewerpipeline. The results from the study showed 69.5% mAP values, and the aver-age time taken to process time of the images was around 57 s. This delivers a33promising performance to be used in real-time applications.The main limitation of YOLO was that it had issues detecting small objectsin the images [141]. Besides, YOLO has issues generalizing the objects into newaspect ratios as several downsampling operations are used to design the archi-tecture. To overcome the issues as mentioned above, Liu et al. [142] proposedSingle-Shot Multi-box Detector (SSD). SSD was developed by using the anchorsfrom RPN [80], multi-box [145], and multi-scale representations [146]. The ad-vantage of SSD is that it can employ anchor boxes with different aspect ratios. Theaspect ratios can help to discretize the output features to a bounding box. SSD candetect objects of different shapes, sizes, and even images with lower resolution.SSD uses a VGG-16 backbone and combines several feature layers at the lastconvolutional layers to predict the confidence score and bounding box coordinatevalues. Localization loss and confidence loss are obtained through the training ofthe network. In addition to that, Non-Maximum Suppression is used to refine thebounding box detection results. A study regarding SSD was conducted by Kumaret al. [140] to estimate the sewer defects. The results from their experiment showthat SSD could obtain only 53% of mAP, while it took only 37 s to process oneimage with the proposed framework.The aforementioned literature shows that there were no relevant studies to de-tect the valves in the water pipelines. However, numerous attempts were madeto detect the root intrusion or deposition inside the sewer pipelines. Therefore,developing a comprehensive valve detection framework, which could detect thevalve inside the water pipelines, could aid in the pipe inspection process and im-prove the inspection system and contribute towards condition assessment of thewater pipelines.2.4 SummaryIn recent years, there have been substantial accomplishments regarding the au-tomation of defect detection for sewer pipelines compared to water pipelines. Theautomated algorithms aim to reduce the heavy workflow of the operator, which34is the manual assessment of the raw videos/images collected from the in-pipe in-spection tools and by the visual systems. The advantages of the automated systemfor condition assessment include the followings [147]:• Reducing the bias of varying skills of operators and inspection errors due tofatigues;• Improving the rate of detection of faults better than human-eye; and• Controlling the quality and reviewing to corroborate the accuracy of theinspection approach.However, there are many challenges that reduce the aptness of the state-of-the-art models. The robotic platforms are deployed to record and capture theconditions of the pipelines. However, the captured videos/images are always sus-ceptible to poor illumination levels as the movements of the robotic platformsare inconsistent. In other words, the change in any robotic platform changes thecamera angle, which increases the difficulties in defect interpretation as imageswithout any information are captured. Therefore, efficient image pre-processingalgorithms should be employed to remove such images from the data set. Therange/resolution of the camera and the number of images for training a modelplay a crucial role in increasing detection accuracy. The higher the resolution andthe higher the training data set, the better the model will have the generalizationability. Subsequently, the detection accuracy of the model will improve.The existing studies show that defects were identified by using single channelinformation (gray-scale). This reduces the complexity of the model but can missmany important details of fine defects such as fractures or cracks. Therefore, toimprove the detection ability of the model, models should be trained with the RGBchannel information so that the robustness and reliability of the model increase.There have been no studies for valve detection in water pipelines. The valvesare considered to be objects. Hence, the existing object detection algorithms werereviewed. Although there have been tremendous achievements in the object de-tection field, some issues still need to be tackled. Identifying objects in multiple35aspect ratios and aspect sizes is challenging as the current object detection al-gorithms can not detect the same objects from different views and scales. Thesimultaneous classification and object localization is also a challenge. However, aformulation of the multi-task loss function may resolve this issue to a certain ex-tent. The computational demand for real-time training of the current object detec-tion model is very high and cost-intensive. Designing lightweight models mightmitigate the issues, but there might be a trade-off between the model performanceand accuracy. The real-time video processing of objects is also challenging as theobjects need to be detected very fast with respect to the frame speed of the videos.Considering the above mentioned issues, it is envisioned that a fully automatedinspection system for data acquisition (hardware) and interpretation algorithms(software) would be developed. This will benefit the condition assessment ofwater pipelines and enhance the asset management of the water infrastructure.36Chapter 3Automated Defect Characterizationfor Water PipelinesIn recent times, the condition assessment of the water pipelines are conducted bythe robotic platforms. Once the robotic platforms are deployed inside the pipelinethrough the access point, the cameras mounted with the robotic platform starts torecord the internal condition of the pipeline. These recordings are then manuallyassessed by the human inspector to identify the pipe defects inside the pipelines,which is time-consuming, inefficient and unreliable.In this chapter, an efficient and reliable automated defect characterizationframework is presented to facilitate the inspection process for the water pipelines.With the aid of the proposed defect characterization framework, a defect char-acterization tool is developed. This automatic defect characterization tool canautomatically detect the defect detection rate, reduce the burden and time for thehuman operators, which can aid them in making timely decisions for pipe re-pair/rehabilitation. This chapter is submitted in IEEE/ASME Transactions onMechatronics (status: under review), and all rights are reserved to the authors.373.1 Defect CharacterizationWater is an indispensable resource for the existence of human life, social welfare,and economic development for a nation [148]. With population growth, urbaniza-tion, and industrialization, the demand for water consumption spiral up every day[149]. A report from U.S. Environmental Protection Agency (EPA) states that inthe U.S., every year, around 1.7 trillion gallons of water volume is lost through thewater transmission system, which could be an equivalent of losing USD 2.6 billionof U.S. revenues [150]. Therefore, regular inspections of the water pipelines arerequired to identify and anticipate the location and the type of defects [151]. Iden-tifying the defects can also aid in determining the severity/rank of the defects sothat pipe repairs/rehabilitation decisions can be made before a catastrophic eventtakes place.Presently, most water utility companies employ closed-circuit television(CCTV) based inspection techniques [1] to assess the condition of the pipelines.The inspection technique comprises of a robotic crawler, equipped with a cam-era and an illumination system to record the inner surface of the water pipelines[140]. The pipe inspectors then examine the recorded videos offline or in real-time(while the robotic crawler is recording) to manually identify the defects inside thewater pipeline. This manual approach to detect the defects is often subjective tothe knowledge of the operators, inconsistent, labor-intensive, and time-consuming[1, 152]. Hence, there is a constant urge to develop a faster, reliable assessmenttechnique to help resolve the above problems.An automated defect characterization system to interpret CCTV recordings isthought to be one of the efficient ways to tackle the existing problems. With thecomputerized defect detection techniques, the defects can be accurately analyzed[153]. The system could also be employed to reassess the historical CCTV dataset to predict the useful remaining life of the pipelines [154, 155]. Besides, inte-gration of the automated system with the CCTV software can aid in the real-timealerts, identify the defects, and location in the pipelines. This can greatly leveragethe strenuous manual process, reduce errors (misinterpretation/missing), and save38time.The prior studies depict that the automated analysis of defect characteriza-tion was mainly conducted on the sewer pipelines. In recent times, computervision and machine learning techniques have been used to design automated sys-tems/tools. However, the conventional computer vision and machine learningtechniques require a large number of images to pre-process and complex fea-ture extractors to obtain the features, which makes the technique inefficient andrepetitive [81]. For instance, Xu et al. [156] and Moselhi and Shehab-Eldeen[157] employed edge detection, Fourier Transforms, and binary image threshold-ing techniques to extract the features and identify the pipe deformations, joint-displacements, cracks, and spalling of the pipes [156, 157]. Guo et al. [158] pro-posed the frame differencing technique, which requires identifying the regions ofinterest by subtracting from the reference image. The region of interest is thenused as an input for feature extraction, which can then classify the defect types[158]. Moradi and Zayed [159] employed Hidden Markov Model (HMM) to iden-tify the defective sewer frames in real-time. Su and Yang [94] and Halfawy andHengmeechai [153] proposed using morphological operations to detect the regionof interest and histogram of gradients (HOG) for feature extraction. However,the major issue with the conventional algorithm is that it requires intricated fea-ture extractor, which can only perform one specific task. Also, the conventionalalgorithms often require complex image pre-processing steps, which makes theprocess quite strenuous and tedious for the inspection operators [81].Nowadays, deep learning-based techniques have revolutionized conventionalalgorithms and have shown remarkable performances for object detection and im-age classification task. The deep learning-based algorithms can extract the fea-tures automatically and minimize the pre-processing time frame. This enhancesthe efficiency and the performance of the model than the traditional algorithms[81]. The breakthrough of the convolutional neural network (CNN) has improvedthe in-domain generalization capability of the models. The in-domain generaliza-tion capability of a model defines the capacity to precisely classify the images with39respect to variations of shape, size, texture, and color [140]. Unlike the conven-tional algorithm (HOG and speeded-up robust features (SURF)), which focusedon the edges and corners of the images, deep learning-based algorithms rely uponthe feature learning process. With the aid of the back-propagation algorithm, theoptimal features of the images can be automatically learned and classified [140].Recently, Kumar et al. [49] proposed a binary CNN-based framework to clas-sify the defects inside the sewer pipelines. The algorithm could only identify thedeposits, cracks, and root intrusion defects in the sewer pipe images. Similarly,multi-class CNN-based deep learning algorithms to classify the defects inside thesewer pipelines were proposed in [82, 160]. Cheng and Wang [81] employed afaster region-based convolutional neural network (Faster R-CNN) to detect cracks,deposits, water infiltration, and root intrusions. Yin et al. [161] used the YOLOv3network, a popular object detection-based deep learning algorithm, to perform thereal-time defect detection to sewer pipelines. Their model could achieve a meanaverage precision (mAP) of 85.37%. Kumar et al. [140] conducted an compara-tive analysis with Faster RCNN [80], YOLOv3 [144], and Single-Shot MultiboxDetector (SSD) [142] to identify defects in sewer pipelines. The results from theirstudy show that Faster-RCNN has a mAP value of 76.2%, YOLOv3 has a mAPvalue of 74.5%, and SSD has a mAP value of 54.4%, respectively.The aforementioned studies primarily focused on the simultaneous localiza-tion and classification of the defects for the sewer pipelines. However, developinga fully automated defect characterization system for water pipelines has remaineda challenge for the water management industry. Therefore, to tackle this industry-based challenge, we attempt to develop a fully automated defect characteriza-tion system for water pipelines that is 1) accurate, 2) efficient, and 3) reliable.The automated defect characterization system aims to achieve higher detectionaccuracy, reduce the human operators’ burden, save significant time and laborcosts. Hence, this paper proposes CSA-MaskC-RCNN based defect characteri-zation framework, which combines channel and spatial attention mechanism toform an adaptive module to emphasize and extract the features of the defect. The40extracted features are then passed through a defect detector, MaskC-RCNN, todetect, classify and segment the defects accurately. In order to make the modelmore efficient and reliable, an user-friendly defect characterization graphical unitinterface is developed so that the operators can obtain reliable results (rather thanthe expert knowledge) and save significant time. In short, the contributions of thispaper are as follows.• A deep-learning based automated defect characterization algorithm, CSA-MaskC-RCNN, is proposed to efficiently detect the fine defects and distinctstructural features in water pipelines. This can achieve higher accuracy andreduce the detection error/misinterpretation of the defects.• An user-friendly and interactive defect characterization interface is devel-oped, to ensure reliable results and reducing significant time for the humanoperators to conduct the manual assessment.• The proposed model outperforms the state-of-the-art model, which indi-cates that it has an excellent potential to be integrated with the pipelineassessment programs.The proposed computational framework will facilitate and enhance the existingcondition assessment programs of the water pipelines. The rest of the chapter isorganized as follows. Section 3.3 presents the defect detection framework for thewater pipeline. The experimental results and analyses are available in Section 3.4.Finally, Section 3.5 summarize the findings from this experiment.3.2 Qualitative AnalysisLoss of water through the pipe failure is considered as a costly global issue. Thefailure of the water pipelines are usually due to the aging infrastructure, third partyinvasion, and severe environmental factors [162, 163]. Therefore, routine periodicinspection is paramount to know the internal conditions of water pipelines. The41traditional pipe defect detection is often performed through a skilled pipeline in-spector, which is often error-prone, laborious, and time-consuming. Therefore,this chapter proposes an easy, efficient and robust algorithm that can characterizethe type, location, and shape of the water pipe defects accurately. The proposedautomated algorithm can considerably reduce the assessment time and allow theto make a strategic plans for pipe replacement and rehabilitation.3.3 Proposed FrameworkThis section describes the proposed automated defect characterization algorithm,namely CSA-MaskC-RCNN. Once the CCTV videos are obtained through theautonomous robotic platforms, PipeDiver® (shown in Fig. 4.1). The videosare processed through image processing algorithms and then fed to the proposedframework to automatically characterize, classify and segment the defects. Thetrained model is then used to develop a defect characterization unit interface sothat the operators can quickly detect the defects. Figure 3.1 presents the overviewof the proposed defect characterization framework for the water pipelines. Thedetails of each section are provided in the following.3.3.1 Image Pre-processing ModuleThe raw CCTV footages/video data are obtained from the in-pipe inspectionrobotic platform, PipeDiver® (owned by Pure Technologies-Xylem, Canada).The video data are then extracted in a sequence of images. These images rep-resent the internal state of the water pipelines. The images may contain defectssuch as crack, corrosion, etc. or construction features such as outlet and pipejoint. Among the extracted images/frames, the defects or the constructive fea-tures are located in a small number of images. Therefore, the images containingthe defects/construction features are separated from the rest. Then, image aug-mentation techniques are employed to increase the number of images for training.Image augmentation also helps to prevent the models from over-fitting [164]. In42Figure 3.1: The overview of the proposed automated defect characterizationframework © Rakiba Rayhanathis experiment, rotation, random zoom, translation, random crop, and shear wereemployed. The resolution of the raw images was 1394× 1080, but for this ex-periment, the images are resized to 224× 224. Once the images are resized, theimages are annotated by using a VGG Image Annotator (VIA) tool [165].3.3.2 Defect Characterization ModuleDefect characterization is defined as simultaneous detection of defects, geomet-rical shape, size, and location of the defects. Once the image pre-processing is43completed, the annotated images are then fed through our proposed CSA-MaskC-RCNN defect detection module. Our proposed defect detection framework con-sists of a channel and spatial attention-based mechanism, so that the representa-tional power of the network to extract the defect features can be improved and alsoa modified Mask-Regional Convolutional Neural Network (Mask-RCNN), whichcan simultaneously detect, classify and segment the defect features efficiently.In recent times, transformer [166] based frameworks have gained popularityto solve many attention-based tasks [167]. Despite its popularity, the transformer-based backbones have some challenges, limiting their usage to some particularapplications. This is because transformers frameworks can not extract low-levelfeatures in the images, especially the edges and corners, as patch tokenization oc-curs from the input images. Besides, transformers are memory and computation-intensive as many overheads are required for training and testing the model. Thehead of the transformer, also known as multi-head self-attention, can only mapone embedded image dimensions, which can vitiate the efficiency of the net-work. Additionally, the transformer backbones can operate on a fixed scale ofthe input tokens and positional encoding, which makes it less suitable for denseprediction-based applications [168]. Hence, our proposed framework is basedon the convolutional neural network (CNN), which can increase/decrease in di-mensions, is less computational intensive (parameter sharing [168]), robust, andprovides faster inference time. The following section briefly describes the con-cepts of the channel-spatial attention mechanism and the defect characterizationalgorithm, MaskC-RCNN.3.3.2.1 Channel-Spatial Attention (CSA) ModuleThe attention mechanism is an effective algorithm that allows us to know whichregion to focus on and improve the representation of the areas. In other words,to focus more and learn the representation features of a specific part is the aim ofusing the attention mechanism [169–171]. In our case, it specifically concentrateson the pipe defects and ignores the rest of the irrelevant objects in the frame.44The attention mechanism improves the characterization capabilities for defectsthat look similar. Nowadays, different types of attention mechanisms are widelyapplied to many applications. The channel-spatial attention (CSA) module, alsoknown as the convolutional attention block module [4] is a type of attention mech-anism that combines the channel’s informative features and spatial dimensions. Asequence of the channel and spatial modules are applied so each branch can de-termine where and what should be learned in the channel and spatial dimensionsand suppress the irrelevant information [4].Let us consider the feature map of input as,F ∈ RC×H×W (3.1)CSA sequentially employs a 1 Dimension (D) channel attention map, Mc and 2Dspatial attention map Ms. They are defined as equation (3.2), and (3.3), respec-tively.Mc ∈ RC×1×1 (3.2)Ms ∈ R1×H×W (3.3)The equations for CSA can be then condensed to (3.5), where ⊗ are used to indi-cates the element-wise multiplication. In the multiplication process, the channelattention values are copied with the spatial attention values and vice versa. TheF′′ represents the final output of the CSA.F′ = Mc(F)⊗F (3.4)F′′ = Ms(F′)⊗F′ (3.5)The channel attention modules, Mc(F) focuses on what is relevant to the givenimage, as each of the channels acts as the feature detector. The spatial informa-tion of the input feature is computed through average pooling and max poolingfunctions. This results in two spatial descriptors such as Fcavg and Fcmax. Thesedescriptors contain the average pooled features and max pooled features and are45transferred to generate the channel attention map of Mc ∈ RC×1×1. By employingthe element-wise summation function, the output vector of the features are mergedalong the aforementioned descriptor and it is expressed as (3.6). In (3.6), σ rep-resents the sigmoid function, W0 ∈ RC/r×C and W1 ∈ RC/r×C are the weights ofmulti-layer perceptron (MLP), RC/r×C represents the activation size of the over-head parameter and, r represents the reduction ratio [4].Mc(F) =σ(MLP(avgpool(F))+(MLP(maxpool(F))=σ(W1(W0(Fcavg))+(W1(W0(Fcmax))(3.6)The spatial attention module, Ms(F) focuses on the location or the region to whereit should focus to learn the parameters. To generate an effective feature descriptor,average pooling and max pooling are used on the channel dimension and the out-puts are concatenated to obtain the feature descriptor. A convolution layer is thenapplied after the concatenated layer to obtain the spatial feature maps. Hence,Ms(F′) can be presented as equation (3.7).Ms(F′) =σ( f 7×7([avgpool(F);(maxpool(F)]))=σ( f 7×7[Fcavg;Fsmax])(3.7)where σ represents the sigmoid function and f 7×7 represents the convolutionallayer with a 7× 7 filter size. Once the channel and spatial modules are defined,they are sequentially applied to the input image so that the feature maps can cap-ture the relevant information better [4].The illustration of the channel-spatial at-tention module is shown in Fig. 3.2.3.3.2.2 MaskC-RCNNThe original Mask-RCNN [172] is an effective object instance segmentationframework, which first detects the target in the input images and then generatesrefined instance segmentation results for the individual target [172]. It is an in-nate extension of the popular object detection algorithm, Faster-RCNN [80]. TheMask-RCNN comprises three stages. At first, a backbone (ResNet in our case)46is employed to extract the features from the input images. We employed ResNetas it requires less memory and computation power than many other dense lay-ered based backbones [173]. Then, the feature maps are transferred to the re-gion proposal network (RPN). The architecture of Mask-RCNN adopts the RPNof Faster-RCNN. However, instead of ROIPool, it adds a branch, ROIAlign, aquantization-free stage to mitigate the misalignment issues to obtain the target’sregion of interest (ROI). The output from this stage is then relayed to the fullyconvolutional network (FCN) to predict the segmentation masks for the object inparallel to the bounding boxes and confidence scores.We have modified the conventional Mask-RCNN framework in this study byadding an edge detection branch. The edge detection branch allows the networkto emphasize the edges of the defect and accurate segmentation of the defects canbe achieved. Therefore, we have employed Canny [174] edge detection filter tothe conventional Mask-RCNN and named as MaskC-RCNN. The canny operatorexecutes the convolution operation with respect to the input, and a new loss func-tion, edge detection loss, is added to the overall loss function of the network. Inthis study, root means squared error (RMSE) between the predicted detection andthe ground-truth, is used to define the edge detection loss function. Equation 3.8presents the edge loss function in the following.ledl =√1N ∑1≤i≤N(|yi− yˆi|2) (3.8)3.3.2.3 CSA-MaskC-RCNNThe architecture of our proposed MaskC-RCNN is presented in Fig. 3.3. Wechose our backbone to be a five-layered residual network (ResNet), which is aconvolutional-based network. For the purpose of this study, we have added theCSA modules {CSA2,CSA3,CSA4,CSA5} with the {C2,C3,C4,C5} convolutionlayer. The output is then sent to the {P2,P3,P4,P5}, top-down lateral connectionarchitecture, Feature Pyramid Network (FPN). The FPN consists of the ‘res5’,47which helps the module to be more efficient with a lower number of filters.The original Mask-RCNN was developed and tested with two backbones,ResNet50 and ResNet101. ResNet [175] is a residual network that enhances thetraining of deep neural networks and accuracy. However for our proposed CSA-MaskC-RCNN, we have employed different types of residual networks, such asResNet18, ResNet34, ResNet50, ResNet101, and ResNet152. We have also at-tempted to combine some residual networks to examine the performances. Theresults of different backbone and combinations can be found in Table 3.3. Afterextensive studies, we have found that a combination of ResNet50, ResNet101, andResNet152 provides the best prediction results. It helps to obtain the in-depth fea-tures required to detect and segment the minute defects of the pipelines. To predictthe Region Proposal Network (RPN), we have employed the last layer (‘Conv4’)for ResNet50, ResNet101, and ResNet152. For this model, a multi-task loss func-tion is employed in each of the regions of interest generated (ROI) by the RPNtruth box ‘b’ is obtained. During the region proposal search, if there is a match,the ROI is assigned as a class label of ya = 1 and a vector value of (φ(ba;a)),which represents the encoded ground-truth bounding box ‘b’. For each ROI ‘a’,the best-collated ground with respect to the ROI, ‘a’. The class label is assigned toya = 0, if there is no event of matching. The ROI losses are comprised of locationloss, lloss for predicted box floss(I,a;θ), classification loss clsloss for predictedclass fcls(I,a;θ), and mask loss, mskloss for predicted mask fmask(I,a;θ) [176].Additionally, we have added the edge detection loss (ledl) to the overall loss func-tion of the proposed model. The equation (3.9) presents the overall loss functionused for this study.L (a, I;θ) =1mm∑i=11nN∑j=11[a|positive] · lloss(φ(ba;a)−floss(I,a;θ))+ clsloss(ya, fcls(I,a;θ))+mskloss(mska, fmask(I,a;θ))+ ledl(3.9)where I represents the image and θ represents the model parameter, m represents48the mini-batch size, N is the number of ROI for each input image. For localizationloss, the L1 loss is used. For classification loss, softmax and for mask loss binarycross-entropy was used [176].Figure 3.2: The illustration of CSA module [4]Figure 3.3: The illustration of the proposed framework ©Rakiba Rayhana493.3.2.4 Ensemble MechanismThe output from Resnet50, Resnet101, and ResNet152 backbones, extracts differ-ent kinds of features as their number of layers in the architecture is different. Forinstance, the deeper layer of ResNet152 can evaluate high-level features. How-ever, some spatial information is lost due to the contraction and pooling layers.Some defects, such as crack, which has very fine details, can be missed by oneclassifier, whereas the other can detect them. An ensemble of the output fromthree classifiers is employed to solve this issue. The combination of the resultscan improve the detection accuracy and the masks for the defects. Among thethree classifiers, only one model is considered the primary model, and the othertwo are the auxiliary model, which can aid in improving the accuracy values. Asthe aim is to increase the mAP values for the defect detection, we have employedweighted average voting to combine the predicted masks into an ensemble forthis study. Weighted average voting is a machine learning technique that mergesthe predictions from the multiple models according to the contribution/weight ofeach of the models. In this study, the weights were chosen from the three selectedmodels’ classification accuracy and averaged to obtain one predicted mask for thedefects.3.3.3 Defect Characterization User InterfaceThe proposed CSA-MaskC-RCNN defect detection algorithm is then trained andevaluated based on the evaluation metrics. Then, the trained model is used todesign a graphical user interface (GUI), which is the defect characterization userinterface (DCUI) in our case. DCUI is an interactive interface that permits thehuman users (pipeline operators) to automatically detect the defects from theCCTV videos at high speed and accuracy. DCUI is mainly developed throughpython scripts and PySimpleGUIQt package. PySimpleGUIQt is one of the effi-cient packages to build a simple GUI as the latest release comprises Tkinter, Qt,WxPython, and Web (Remi) libraries. Therefore, an easy, simple, and reliableuser interface is developed through PySimpleGUIQt packages.50A basic GUI consists of three elements; containers, mains, and widgets. Thecontainers provide a structure for the interactive interface layout, and the mainscontain the developed defect detection algorithm. The output is then displayedwith the aid of the graphical widgets, also known as buttons. As one of the objec-tives of this study is to minimize the burden on the human operators, we have triedto keep the interactive interface minimalist. At first, the python script is requiredto run. A successful script run will display the DCUI window, shown in Figure3.4. The operator would then select the path to the input of the CCTV videos bypressing the “Browse” button. The user must also specify the path where the out-put will be saved. The DCUI consists of two sliders as Confidence and Threshold.These two sliders allow the operators to control or set the confidence and thresh-old scores of the defects they are interested in. Once all the setup is completed, theoperator needs to click the “OK” button. After that, the file is loaded into DCUI,and it takes around 5-10s to load the video file and trained defect detection .h5 file.Then another window is opened (shown in Figure 3.5), which detects the defectswith a mask (geometrical, shape, and size), bounding box, confidence score andclasses for each frame. All the detected files are then automatically saved to theoutput folder.3.4 Experimental Results and AnalysisThe CCTV video data were collected through the free-swimming robotic plat-form, PipeDiver® (owned by Xylem-Pure Water Pipe, Mississauga, Canada).Once the PipeDiver®is deployed into the water pipelines, it starts to capture theinternal condition of the water pipelines. Once the flight has been completed, therecorded videos are then manually analyzed by the pipe inspectors to identify thepipe defects. However, the manual realization of the pipe defects is exhaustive,time-intensive, and error-prone. Therefore, the advantage of using an automateddefect characterization system is that it can reduce the burden of the inspectorsand expedite the inspection process of the water pipelines.For this experiment, 5,550 video frames (images) were extracted from 3051Figure 3.4: The Defect Characterization User Interface (DCUI) (Images re-produced with the permission from the Pure Technologies-Xylem,Mississauga, ON, Canada).videos, and the defects were labeled by following the definition of NASSCOPACP standard code [177] and annotated via VIA tool [165]. Our dataset con-tains four types of defects such as crack, corrosion, metal-loss, and broken, andtwo kinds of construction features such as outlet and pipe joint. The defects ofeach type are attempted to distribute among the dataset evenly. Table 3.1 in thefollowing presents the details of each kind of defect. Our dataset is then randomlysplit into training (70%), validation (10%), and testing (20%) subsets to evalu-ate our proposed algorithm. The training, validation, and testing sets ratio wasdetermined from the previous deep learning studies [81, 178].3.4.1 Model ImplementationAt first, the annotated images are converted into a .json file, including the pipe de-fect information such as the location, bounding boxes (the ground truth), classes,and mask information. Once the .json file is obtained, it is fed into the algorithmto train and validate the performance of the proposed model. The model is trainedand tested with NVIDIA GeForce GTX 1070 GPU for 15,000 epochs. Each of the52Figure 3.5: The Defect detector window (Images reproduced with thepermission from the Pure Technologies-Xylem, Mississauga, ON,Canada).ResNet structures is updated through the stochastic gradient descent (SGD) withmomentum. The training parameters such as the batch size for the training themodel is set to 1, learning momentum was set to 0.9, with a learning rate of 0.001and weight decay 0.0001. Non-maximum suppression (NMS) was also used tofilter out the proposals below the Intersection Over Union (IoU) of 0.4. Besides,the mini mask was also set to “True” so that the mask pixels are stored inside thebounding boxes, and memory spaces can be saved. After 10 hours of training,the training loss of the model was found to be 0.3202, and the validation loss wasfound to be 0.2910. Fig. 3.6 in the following presents the training and validationloss curves of the proposed model. This downward trend in Fig. 3.6 indicates thatthe deviation of the prediction loss of the model gradually decreases by updatingthe loss function in (8) according to the batch size. After 13,500 epochs, the lossfunction for both the training and validation phase reaches a state of convergence.53Table 3.1: Data set description for each types of defects and featuresTraining set Validation set Test setCorrosion 680 105 201Crack 682 100 180Broken 544 78 172Metal loss 681 87 188Outlet 648 91 182Joint 650 94 187Total 3885 555 1110Percentage 70% 10% 20%Once the training is completed, the model weights are saved in .h5 format.After that, the saved weights are loaded to perform the testing of the model in ourdeveloped DCUI. Table 3.2 in the following provides the confusion matrices forthe testing performances. The numbers in the diagonal present the True Positive(TP) values of the defect. TP indicates the number of correctly identified defects.Similarly, the defects which are not correctly identified are known as True Neg-ative (TN). Besides TP and TN, there are False Positive (FP) and False Negative(FN). The FP defines the instances when the incorrect defects are identified as thecorrect instances. On the other hand, FN is defined as the identification of correctinstances as incorrect instances.The model is evaluated in terms of selected metrics, namely, mean AveragePrecision (mAP), Precision, Recall, and F1-score. Average precision is the aver-age of different recall values for a particular class, whereas the mAP is the meanof all the average precision (AP) of each class. Equation (3.10) and (3.11) in thefollowing defines each of the metrics, respectively. The IoU is defined as the ra-tio of the area of the intersection between ground truth (Gt) and prediction (P)over the area of union between the Gt and P. The Precision values define therate of the correctly identified defects over the total detected objects, whereas theRecall values define the rate of completeness of the identified defects among thetotal labeled defects. The F1-score evaluates the harmonic mean of the obtained54Precision and Recall values to define the classifier’s performance. The F1-score,also known as dice similarity score/coefficient, measures in the range of 0 and 1,where any values close to 0 indicate the bad performance of the classifier, and anyvalues closer to 1 indicate a good classifier. Equation (3.13), (3.14), (5.6) definesthe Precision, Recall, and F1-score, respectively.AP =111 ∑r∈{0,0.1..1}max p(r˜)r˜:r˜≥r(3.10)mAP =1Nclass∑APi (3.11)IoU =Gt ∩PGt ∪P (3.12)Precision =T PT P+FP(3.13)Recall =T PT P+FN(3.14)F1-score = 2× Precision×RecallPrecision+Recall(3.15)Table 3.2: Confusion Matrix (CM) for testing setCorrosion Crack Broken MetallossOutlet Joint Missed TotalCorrosion 186 6 0 0 1 0 8 201Crack 6 158 0 0 0 0 16 180Broken 0 0 153 5 0 0 14 172Metalloss0 0 3 163 0 10 12 188Outlet 0 0 0 0 170 8 4 182Joint 0 0 8 0 5 168 6 18755Figure 3.6: Training and validation loss curve of the proposed model©Rakiba Rayhana3.4.2 Model VisualizationThe deep neural networks have the high capability to learn the hierarchical in-formation from the input images. The intermediate layers of the networks em-ploy themselves to learn the input patterns and obtain the gradients and the edges.As the network becomes deeper, the number of layers increases, and it can re-tain the intricate patterns learned from the prior layers [160]. In this study, wehave used three different backbones of residual networks and added the attentionblocks to each backbone to improve the detection performance. Hence, we visu-alize attention-based residual neural networks for one layer to see if the attentionblocks were effective in retaining the information. For ease of comparison and un-56derstanding, we choose to visualize the last layer of “Conv4” block for Resnet50,Resnet101, and Resnet152 backbone to assess as ResNet50, ResNet 101 has thesame number of layers (12 layers) for “Conv3.” However, ResNet152 has a differ-ent number of layers for “Conv3” (24 layers). Fig. 3.7 in the following show thefeature maps for the last layer “Conv4” block. Fig. (a), (b), and (c) represents thefeature maps of ResNet50, ResNet101, and ResNet152, respectively. Fig. (d), (e),and (f) represent the feature maps after attention modules are added to the con-volutional blocks for ResNet50, ResNet101, and ResNet152, respectively. Fromthe feature maps, it can be visible that adding the attention mechanism has en-hanced the features extracted from the input. The feature maps indicate that at thelast layer of the “Conv4” block, the features are more distinct and localized whenattention blocks are added to it. .3.4.3 Comparative StudiesThe results from our experiments show that our proposed model can achieve up to0.8688 mAP values with a precision value of 0.8693 and a recall value of 0.8723,respectively. Also, the F1-score of the proposed model is 0.8708. Table 3.3 in thefollowing presents the results that are obtained from our experiments. To show theeffectiveness of our proposed model, we have also compared the performance ofthe proposed model with the individual backbone such as ResNet50, ResNet101,and ResNet152 performances. Additionally, we have compared the performanceof our proposed model with a popular segmentation-based surface defection algo-rithm, called “Segmentation/DecisionNet” [179] and two popular state-of-the-artsemantic segmentation techniques, called UNet and Fully Convolutional Network(FCN). The DecisionNet algorithm is proposed by Tabernik et al. [179], and theproposed algorithm is based on a segmentation network, which segment the sur-face, detects, and makes a decision by using a convolutional neural network. De-tails of “DecisionNet” algorithm can be found in [179].UNet [180] is a semantic image segmentation technique that was originally de-veloped for biomedical image segmentation problems. UNet is a U-like structure,57Figure 3.7: Visualization of the last convolutional layer of “Conv4” blockfor ResNet50, ResNet101, and ResNet152. (a) The last layer of“Conv4” block for ResNet50 (b) The last layer of “Conv4” block forResNet101 (c) The last layer of“Conv4” block for ResNet152 (d) Thelast layer “Conv4” block after attention is added for ResNet50 (e) Thelast layer of “Conv4” block after attention is added for ResNet101 (f)The last layer of“Conv4” block after attention is added for ResNet152.(The images are resized for the purpose of illustration) ©Rakiba Ray-hana.and it comprises an encoder path. The encoder includes a stack of convolutionaland max-pooling layers, which helps obtain an image’s contextual information.UNet also has a decoder path, which contains the transposed convolutional blocksto determine an image’s precise localization. UNet was first proposed in 2015 byRonneberger et al. [180]. Since then, it has been used in many studies to performsemantic segmentation.58Similar to UNet, FCN was first proposed in 2015 by Long et al. [181]. FCNis also a semantic segmentation technique that employs a skip network to capturethe deep features and semantic information. When an image is fed to the network,it is downsampled to obtain a single output label. After that, the image is upsam-pled through transposed convolutional layers with the aid of fully convolutionallayers. The spatially localized information is fused with the shallower layers toobtain the segmented output. Like UNet, FCN has also become one of the mostused segmentation techniques. Hence, these two popular algorithms were chosenfor the comparison of this study. The results from Table 3.3 show that UNet hasthe second-best performance after our proposed model. The encoder and the de-coder help to retain more information than the FCN, ResNet152, and ResNet101.ResNet18 has lower performance among all these methods, as it has a shallowerlayer than all the other methods. We have also combined ResNet18-ResNet34-ResNet50-ResNet101-ResNet152. and only ResNet101-ResNet152 and obtaineda mAP value of 0.8598 and 0.8408, respectively. One main reason for the re-duced mAP values for ResNet101-ResNet152 could be the individual mAP values(0.8245 and 0.8333). So, when these two mAP values are combined through theensemble mechanism, the mAP value comes to around 0.8408.The area under the curve (AUC) is also higher for our proposed model. Fig.3.8 shows the receiver operating characteristic (ROC) curve for all the comparedmodels, and Fig. 3.9 shows the precision-recall curve for the compared models.Fig. 3.10 and Fig. 3.11 shows the experimental results obtained from our proposedmodel. From Fig. 3.8, Fig. 3.9, Fig. 3.10, Fig. 3.11, and Table 3.3, it can beobserved that our proposed model has better performance and area under the curve(AUC), i.e. 0.8693 than all the other models. Table 3.4 shows that our proposedmodel performs better in processing a 30 mins video than the other state-of-the-artmethods. Hence, it can be deduced that our proposed method is more accurate indetecting, segmenting the defects, and efficient, which can reduce the burden onthe pipeline inspectors.59Figure 3.8: Receiver operating characteristic (ROC) curve for the proposedmodel and state-of-the-art models ©Rakiba Rayhana3.4.4 Ablation StudiesTo obtain more insights into the improvements attained through our proposedmethod, we have investigated the effectiveness of adding each component thatwas added to the experiment. This study is aimed to improve the defect detectionof water pipelines. Hence, we have added attention blocks after the convolutionalblocks of the residual networks of the MaskC-RCNN and obtained the individ-ual performance of the models. Once the output from each of the models wasobtained, an ensemble approach was employed to improve the model’s perfor-mance.Investigation of Attention Module: At first, we have added the CSA attention60Table 3.3: Comparative results of our proposed model and state-of-the-artmodels.Model Name mAP IoU Precision Recall F1-scoreAUCMask-RCNN-ResNet180.6732 0.4521 0.6872 0.6632 0.6750 0.6882Mask-RCNN-ResNet340.7118 0.4651 0.7002 0.6892 0.6947 0.7261Mask-RCNN-ResNet500.7910 0.5113 0.7832 0.7734 0.7783 0.8019Mask-RCNN-ResNet1010.8245 0.5329 0.8319 0.8385 0.8352 0.8257Mask-RCNN-ResNet1520.8333 0.5334 0.8251 0.8325 0.8287 0.8479Mask-RCNN-ResNet101-ResNet1520.8408 0.5333 0.8351 0.8311 0.8331 0.8372Mask-RCNN-ResNet18-ResNet34-ResNet50-ResNet101-ResNet1520.8598 0.5312 0.8458 0.8451 0.8454 0.8591DecisionNet 0.8391 0.5361 0.8454 0.8468 0.8461 0.8342UNet 0.8598 0.5210 0.8521 0.8412 0.8466 0.8638FCN 0.8486 0.5126 0.8432 0.8348 0.8390 0.8498Ours 0.8689 0.5378 0.8693 0.8723 0.8708 0.8693module [4] individually with the ResNet50, ResNet101, and ResNet152 blocks.The CSA module multiplies input feature map for refined adaptive features. Toimprove the attention mechanism, the max-pooled features are employed with theaverage-pool features. This can aid in the finer attainment of the features for theinput data set.Secondly, we have also investigated the performance of adding squeeze andexcitation (SE) attention block [182] with the convolutional block of ResNet50,61Table 3.4: Performance assessment of the proposed model with state-of-the-artVideolengthManualassessmentDecisionNet UNet FCN Ours30 mins 8 hrs 4 hrs 56 mins 3 hrs 40mins4 hrs 20mins1 hr 10minsFigure 3.9: Precision Recall (PR) curves of the proposed model and state-of-the-art models©Rakiba RayhanaResNet101, and ResNet152 blocks. The SE attention block requires the convolu-tional block as an input. With average pooling, the channels are compressed to anumeric number, and dense layers of ReLU and sigmoid functions are added toreduce and smooth the gating component. The blocks are stacked to accomplish62Figure 3.10: Experimental results from the proposed model. From top tobottom: corrosion, crack, outlet, metal loss and joints defects arepresented. From left to right: (a) original image (b) ground truth(c) masks (d) predictions. Note that the images are resized for thepurpose of this illustrations. ©Rakiba Rayhanadynamic channel-wise feature optimization [182]. However, the problem withthe SE attention block is that it uses the scaling operation, which broadcasts thechannel vector and element-wise multiplies the weighted channel vector values tothe input tensors. This occupies a large amount of memory space and slows thetraining process.63Figure 3.11: Results for different types of defects. From left to right: (a)corrosion (b) crack and corrosion (c) broken pipes (d) corrosion (e)outlet and pipe joint. Note that all the images are resized for thepurpose of this illustration. ©Rakiba RayhanaTable 4.1 in the following presents the effectiveness of using CSA moduleover SE attention blocks to improve the performance of MaskC-RCNN for defectdetection of the water pipelines.2) Investigation of Ensemble Algorithm: We have investigated the effective-ness of ensemble learning. Ensemble learning is a very popular approach in ma-chine learning and the deep learning field. The ensemble learning [183] is knownto improve the accuracy by taking into account several important parameters suchas estimation of confidence, selection of features, error correction, incrementallearning, and so on [183]. For this study, we have adopted the ensemble learn-ing approach to combine the output of the chosen classifiers (ResNet50, 101, and152). Table 3.5 in the following presents the results of the ablation studies.3.5 SummaryThis paper presents an automated defect characterization system for CCTV in-spection videos obtained through the robotic platform, PipeDiver®for water64Table 3.5: Ablation studies for the proposed model.Backbone Name mAP(%)ResNet50 79.10ResNet101 82.45ResNet152 83.33ResNet50+SE (attention) 80.39ResNet101+SE (attention) 82.91ResNet152+SE (attention) 83.89ResNet50+CSA (attention) 81.08ResNet101+CSA (attention) 84.01ResNet152+CSA (attention) 83.98Ensemble+CSA (Ours) 86.89pipelines. An efficient, fast, and reliable algorithm, CSA-MaskC-RCNN, is pro-posed to detect structural defects such as corrosion, crack, broken, and metal loss,as well as outlet and joint features. The proposed framework is employed to de-velop a user-friendly defect characterization unit interface (DCUI) or a tool whichcan help the pipeline operators to identify the defects in a few clicks on the CCTVvideo. Compared to the prior studies, this study’s contributions include improv-ing the defect detection rate, classification accuracy for the structural defects (orrelevant features) of water pipelines, and efficiency of the system. The experimen-tal results from this study show that the proposed framework can achieve highervalues of mAP (0.8689), Precision (0.8693), Recall (0.8723), F1-score (0.8708),and AUC (0.8693) in comparison with the state-of-the-art. Hence, integrating theproposed automated defect characterization system can achieve higher defect de-tection accuracies, reduce the burden on inspection operators, aid the operators tomake timely decisions for repair and rehabilitation, and save time and expensesfor the water pipeline industry.65Chapter 4Valve detection for robotic waterpipeline inspection platformOnce the robotic platforms start to record the internal condition of the waterpipelines with cameras [184] for defect analysis, it often encounters difficultywhile passing through the valves. Valves are an essential component of waterpipelines, as they can control the flow of the water and also can help isolate a sec-tion of the pipe network for maintenance schedules [185]. When the robotic plat-forms (shown in Fig. 4.1) encounter complications in passing through the butterflyvalves, it obstructs the videotaping process for the robotic platforms. This ham-pers the inspection and the condition assessment process and, at times, causes therobotic platform to be at a halt. Therefore, this chapter analyzes the video streamsobtained by the robotic platform; PipeDiver®, and proposes an automatic valvedetection system to assist the robotic platform in activating the control mecha-nisms and passing through the valves.This chapter is a minor modification of R. Rayhana, Y. Jiao, Z. Bahrami, Z.Liu, A. Wu, and X. Kong, “Valve detection for autonomous water pipeline inspec-tion platform,” IEEE/ASME Transactions on Mechatronics, vol. 27, no. 2, pp.1070-1080, 2022, and it is reproduced here with the permission of IEEE and thecopyright is reserved with the authors.66Figure 4.1: The video acquisition platform, PipeDiver®. (Images repro-duced with the permission from the Pure Technologies-Xylem, Mis-sissauga, ON, Canada)4.1 Valve DetectionAs more inter-connectors and new pipelines are added to the water supply system,there is a constant need to develop an intelligent automatic inspection system,which can not only capture the conditions of the pipelines but can also operateand pass through any obstacles, such as valve inside the pipelines. In the pastfew years, numerous studies have been conducted for the development of an au-tomated inspection system to diagnose and classify pipe defects without humaninterventions [186–188]. Most of these automated inspection systems incorporatetraditional machine learning and image processing methods [189–191]. However,no studies related to valve detection for water pipelines are attempted yet.Over the past few years, computer vision and deep neural network based tech-niques have been widely adopted to perform the detection tasks [192, 193] Ourobjective of this study is to detect the valves in the water pipelines, which canbe considered as an object detection task. Therefore, an object detection basedapproach is chosen to solve our problem. The traditional approach for object de-tection usually employs image processing algorithms to extract the features andthen classify the object with a classifier. However, the traditional approach fo-67cuses on the processing speed of frames rather on the accuracy. This makes themrestrictive to be used in the real-time applications. On the other hand, deep neu-ral networks (DNNs) have outstanding ability to capture and learn the features,and process the frames in real-time than the traditional methods [194]. Hence,inspired by the success of the deep neural network in the object detection tasks,we present a deep neural network based automatic valve detection framework forthe robotic platforms of the water pipeline.This chapter advocates that the development of the automatic valve detectionsystem for sparse data set is feasible and integration of the proposed model tothe robotic platforms can aid in the real-time condition assessment process ofthe water pipelines. Therefore, the convolutional neural network (CNNs), whichis widely used in visual recognition [195], detection [196], segmentation [197]and classification [198] architecture, is adopted for this study. Nowadays, deeplearning based models are progressively being used in the condition assessmentand structural health monitoring fields [199, 200] A deep learning model requiresa large amount of well-annotated data to train the model, which increases thecomplexity of the learning process. However, the valve images obtained fromthe video streams of the robotic platforms are very limited in number, which in-creases the challenge to design the automatic valve detection system. Apart fromthe sparse data set, there is also a challenge to develop an efficient algorithm whichcan detect valves with higher precision and can process the videos at higher speedfor real-time analysis. In other words, this chapter attempts to solve the challengeof 1) sparse data 2) good accuracy in detection, and 3) faster video processingspeed (FPS) to process the videos. Therefore, to solve the sparse dataset a gen-erative adversarial network (GAN) [201] is used to generate artificial images toincrease the data set and to understand the future prospect of the proposed model.Once the data set is increased, it is fed to the valve detection model, which com-prises of MobileNet [202] and Feature Pyramid Network (FPN) [203] to detectthe valves at higher FPS in real-time scenarios. A pre-trained model is also usedto perform the transfer learning so that the weight of the parameters of the valve68detection network can be initialized.In this chapter, a deep learning-based automatic valve detection frameworkto facilitate the robot’s navigation and ensure continuous inspection without anyinterruptions. The valve detection model is developed by combining MobileNet-160 and Feature Pyramid Network (FPN) and is named as MFPN. The developedframework also employs a generative adversarial network to solve the sparse dataset issues and improve the generality of the framework. The comparative studyand ablation analyses demonstrate that the proposed framework can achieve ahigher mAP value of 89.11% in comparison with the state-of-the-art. Hence, thislight-weight and efficient solution can be deployed to the robotic platform for real-time valve detection and enable autonomous navigation of the robotic platform forcondition assessment of water pipelines.4.2 Proposed FrameworkFig. 4.2 represents the proposed framework for the automated valve detection sys-tem. At first, the historical video database containing valves are directed as an in-put to the valve detection framework. The valve detection framework consists ofdata pre-processing, a valve detection algorithm (proposed model), and the videoprocessing, which are concatenated in sequential order. The output of the frame-work is the labeled video frames containing the valves from the CCTV videos.The details of each of the sections will be discussed in the following sections.4.2.1 Data Pre-processing ModuleThe video frames/images are extracted from the CCTV video files. The imagesthat contain valves are then separated from the rest of the video frames. How-ever, the raw images containing valves are very few in numbers, which makes itunsuitable to use for any deep learning based model. Hence, synthetic generationof image techniques is acquired to increase our data set. For the purpose of thisstudy, BigGAN [204], a type of generative adversarial network (GAN) [201] is69Figure 4.2: The automated valve detection framework [5]used to generate synthetic images. A GAN model comprises of a generator anda discriminator network, which maps the random noise to the input samples andaids in discriminating the real and the artificial (generated) samples [201]. Re-cently, GAN has spurred many types of deep generative frameworks and reachedto the stage where the generated images are difficult to distinguish from the orig-inal image. Besides GAN, Autoencoders (VAE) [205, 206], and Auto-Regressivemodels (AR) [207] are the popular generative networks. However, their efficiencyin reconstructing the image is lower than the GAN models [208].The main reason to choose BigGAN [204] amongst other popular GAN is thatmost of the GAN network requires a very large amount of data to train and thento generate novel images. However, obtaining huge data set is always not a fea-sible option as sometimes data collection is not an easy process. Therefore, forour sparse data set, BigGAN [204] with the transfer learning knowledge and somestatistical adaption in the generator is chosen to generate synthetic images to in-crease our data set. Scaling and shifting filters are employed in the pre-trainedgenerators. The scale and the shift results in the convolution layer as in equation(1), where W represents the weight of the convolution, Wi shows the ith filter, γ70represents the scaling factor of the activation filter for each convolution, β rep-resents the shifting factor of the activation filter and cout represents the numberof output channels for convolution. For training on the small data set, we haveused the truncation threshold between 0.3-0.4 to generate new images to intro-duce variation in our data set [208]. Once the settings are completed and thehyperparameters are tuned, the BigGAN [204] is fed with the valve images. Themodel is trained for 6 hours to generate synthetic images. Once the images aregenerated they are shuffled with the original data set, resized to 480 × 480, andlabeled. The effectiveness of the synthetic data generation is then established bycomparing the Peak Signal to Noise Ratio (PSNR) in decibles (db) and StructuralSimilarity Index (SSIM). PSNR and SSIM helps to understand the quality of thegenerated images with respect to the original images. For our case, average PSNRvalue was at 36 dB and average SSIM was 0.84. As the average PSNR values arehigher, and the average SSIM values are close to 1, it can be deduced that the gen-erated synthetic images have good qualities with respect to the original images.Fig. 4.3 in the following shows some of the synthetic images generated from themodified BigGAN [204, 208] and this approach successfully solves the sparsedata set issues for our problem.conv(x;W ) · γ+β= conv(x;W · γ+β )= conv(x;γ1W1+β1, ...,γcoutWcout +βcout)(4.1)4.2.2 MFPN ModuleThe proposed MFPN model is inspired by the RetinaNet framework [209], whichis a one-stage object detector algorithm. The main motivation for our proposedframework is to develop a light-weight and an efficient algorithm which can aidthe robotic platforms to activate the control mechanism and pass through thevalves. Therefore, a new light-weight valve detection framework is proposed.71Figure 4.3: Some of the synthetic valve images generated by the modifiedBigGAN [5]Similar to the RetinaNet framework, our valve detection model comprises of abackbone network, which is the classification network on top of the Feature Pyra-mid Network (FPN). In our case, MobileNet-160 is chosen as the backbone net-work to make our system light-weight and efficient for the valve detection andvideo processing of the CCTV files. MobileNet [202] is one of the efficient state-of-the-art image classification networks which is very light-weight and efficientin detection purpose. As the proposed algorithm is developed for the integrationof the pipe inspection robotic platforms in real-time, we have chosen MobileNetas the backbone of our model.MobileNet comprises of depthwise convolution and point wise convolutions(1 × 1 convolutions). The depth wise convolution uses one filter for each of theinput images and the point wise convolution employs 1× 1 convolutions to mergethe output of the depth wise convolutions. The important feature of depth wiseconvolutions is that it has two different layers; one for filtering and the other onefor merging. This helps to reduce the computation cost and the size of the model.The convolutional neural network takes input as DF×DF×M for feature map, Fand generates DF×DF×N feature map of G. The DF denotes the spatial widthand height of the input feature map, M denotes as the number of input channelsand N denotes the number of the output channels. The convolutional kernel K is72defined by DK×DK×M×N, where DK denotes the spatial width and height of theconvolutional kernel. Therefore, assuming the padding and a stride=1, the outputfeature map for mth filter can be expressed as equation (4.2).Gk,l ,n = ∑i, j,mKi, j,m,n ·Fk+i−1,l+ j−1,m (4.2)For the valve detection model, the MobileNet-160 acts as an encoder, which pro-cesses the input images through the convolution kernels and generates the fea-ture maps. On the other hand, the FPN network acts as a decoder which isknown to process multi-scale feature input layers and produce output feature lay-ers at a homogeneous scale of the input. Usually, feature extractors provide achoice to choose the layers which are processed through the merging element ofFPN layers. In our case, we choose “conv pw5 relu”, “conv pw 11 relu” and“conv pw 13 relu” layers to convolute and generate input features at a featuresize=256. These features are then advanced towards the FPN network, where eachof the pyramids has the capability to detect the valves and outputs multi-scale fea-ture representations of {P3,P4,P5,P6,P7}. The P3 to P5 feature representations areevaluated through the aforementioned intermediate layers of MobileNet-160 anduses top to down and lateral interconnections similar to [203]. However, P6 isacquired through the 3× 3 convolution, and with a stride value of 2 on C5 andP7 is acquired by employing ReLU activation, 3×3 convolution and with a stridevalue of 2 on P6. This minor modification step is adopted from the original Reti-naNet [209] paper which could enhance the speed and the detection accuracy ofthe valves. For P3 pyramid level, the anchor area is set to 322 and for P7 pyramidlevel, the anchor area is set to 5122, which improves the average precision (AP)for the valve detection process.For the classification subnet, which can predict the probabilities of the valvesin the frames and from each of the spatial position, four 3×3 convolution layerswith the C number of filters which is accompanied by the ReLU activations, one3× 3 convolutional layer with KA (K for the number of object classes and A73for number of anchors) filters are used. For the purpose of this study, a fewerparameter based class-agnostic box regressor subnet [209] is used. The model isthen trained until the loss minimizes and a frozen inference is saved. Fig. 4.4 inthe following illustrates the proposed valve detection model.Figure 4.4: An illustration of the architecture for the proposed valve detec-tion system [5]4.2.3 Valve Detection ModuleThe main aim of this automatic valve detection algorithm is to help the roboticplatforms to make intelligent and to activate the control mechanism to navigatethrough the valves in the water pipelines. Generally, the CCTV video files col-lected from the robotic platforms are sent to the survey office to post-process the74videos. The expert operators then need to check frame by frame to know the de-fects or any form of obstacles that are present in the videos. In some instances,long hours of motionless videos (when robotic platforms are at a halt) are ob-tained, and the expert operators can not accurately interpret the situation. Apartfrom the valves, the robotic platform can also stop due to sagging, tree root intru-sions, blockage, or even for pipe collapsing. Therefore, it is important for expertoperators to know why condition assessment recordings were interrupted. Withthe automatic valve detection system, the operators can process the videos fasterand identify the reason behind the interrupted recordings. A video processing al-gorithm with the aid of open-cv is separately written in the python scripts andthe frozen inference model is loaded to the process the videos and identify thevalves. The video frames are processed in Frame Per Second (FPS) and labeledwith a confidence score to detect the valves in each frame. An inference time ofloading the images and processing each frame is also calculated to know if theproposed model is performing well or not. The results and performance analysisare presented in the following section.4.3 Experimental Results and Analysis4.3.1 Data Acquisition and DescriptionThe data set that is used for this experiment is collected by an underwater un-tethered inspection robot which is mainly used to inspect the wastewater and waterpipelines. For this study, the videos are obtained from five different pipelines(two 36 inches pre-stressed concrete cylinder pipe (PCCP), two 24 inches andone 48 inches ductile iron (DI) pipelines). The robot captures each frame insidethe pipelines and records the video without causing interruption to the serviceof the pipelines. The valve images are then extracted from the video files andlabeled manually. The images are labeled manually by using LabelImg software.The extracted images are then divided into training, validation, and testing. Thetotal number of images after the synthetic image generation is 2000. So, at first75percentage splitting technique, (70% of the images are used for training, 10% forvalidation and 20%) were employed. After that, 5-fold cross-validation techniqueswere employed to optimize the hyperparameters and to remove the overfittingissues of the model.4.3.2 Model Training and TestingOur models are trained in two different stages. For the synthetic image gener-ation, we have used BigGAN with statistical adaptation and it was trained withan NVIDIA GeForce GTX 1070 GPU for 5,000 epochs. After that, the imagesare manually labeled and fed to the valve detection model. The valve detectionmodel comprises of FPN and the MobileNet-160 layers and it is implementedby using Keras, one of the high-level programming interfaces for deep learning.The models are trained by using the same GPU configuration as mentioned be-fore. The MobileNet160-FPN model was trained with 100,000 iterations and ittook around 9 hours of training to converge and minimize the loss function forthe proposed valve detection model. Fig. 4.5 shows the training and validationloss of our proposed model. The valve detection model is optimized by usingAdam optimizer and with the initial learning rate of 0.0001. Normal (Gaussian)distribution with zero mean and a standard deviation value of 0.01 was used forthe kernel initializer and prior probability was used for the bias initializer to ini-tialize every convolutional (Conv 2D) layer. For this experiment, the batch sizewas set to 1 and Batch Normalization (BN) was used after each of the Conv 2Dlayers to normalize the input to the next layers for the MobileNet-160 layers. TheMobileNet-160 is based on the depth wise convolutional neural network. There-fore, depthwise Conv2D was applied after each of the BN layers. Rectified LinearUnit (ReLU) is used as an activation function for both MobileNet-160 layers andthe FPN layers. Non-Maximum Suppression (NMS) layer is also used in the FPNnetwork to reduce the redundant filter detection results. A pre-trained model onImageNet data set for MobileNet-160 was also used to optimize the weights andto reduce the model training time (converge fast) to increase the model efficiency.76After that, some additional fine-tuning of the model was performed.Once the training of the model is completed, the model is evaluated with thetesting data set (400 images). These test images are new (16 inches, 30 inches, and84 inches PCCP pipelines) to the network and are used for performance evaluationpurposes. Then, the raw videos (12 videos) of the butterfly valve is also tested toobserve the model performance in real-time.Figure 4.5: The training and validation loss curve for the proposed frame-work [5]4.3.3 Comparative Studies with State-of-the-ArtsAs the purpose of this experiment was to design an efficient (in terms of accuracyand video processing speed) valve detection algorithm, we have chosen some ofthe popular object detection techniques that were used in the state-of-the-art. From77the literature studies, we have chosen Histogram of Gradient (HOG) with SupportVector Machine (SVM) [153], Wavelet transform with Back propagation neuralnetwork (BPN), Radial Basis Network (RBN), and SVM [210] to evaluate theperformance for the valve detection. The results are summarized in Table 4.2.From the deep learning based models, we have chosen state-of-the-art objectdetectors, such as VGG-16, ResNet-50, DenseNet-121, and MobileNet-128 asthe backbone and compare the performance with our proposed model. VGG16was first developed by the Simonyan and Zisserman [211] in the year of 2014.VGG basically stacks 3×3 convolutional layers on top of one another. The max-pooling layers are two fully connected layers with a softmax classifier is used forthe detection. The VGG has two popular versions namely VGG-16 and VGG-19.The numbers 16 and 19 represent the layers used for the training of the model[211]. As the FLOPS, which measures the computation performance is higherfor VGG-19 (20 GFLOPs) than VGG-16 (16 GFLOPs) [212], only VGG-16 waschosen to observe the performance on our dataset. Apart from VGG-16, ResNet-50 [213] is also used as a backbone. ResNet is a very deep neural network, madeup of the residual blocks and trained with the normal stochastic gradient descent(SGD) and global pooling layer. ResNet model is very deep compared to theVGG models, which substantially improves the performance of the model [213].However, the size of the model (103 MB) [212] and the FLOPs (4 GFLOPs) [212]are considerably lower than VGG. For our purpose of comparison, ResNet-50 ischosen as it is one of the efficient and lighter versions amongst ResNet-101 andResNet-152.In 2017, DenseNet was proposed by Huang et al. [214], which combined thefeatures of the input through concatenation. One reason to choose DenseNetfor comparison was that it requires a fewer parameter to train compared to theResNet and also to the traditional CNNs. DenseNet has narrow layers (12 filterseach layer) and it requires only 3 GFLOPs [212] to train the model. Also, thedense networks are well known to minimize the overfitting problems for smallerdatasets [214]. Since our dataset is quite lower than the traditional machine learn-78ing, DenseNet-121 is chosen to be in our comparison studies. As our proposed,MFPN model combines MobileNet-160 and FPN together, we wanted to observeand compare the performance for MobileNet-128 and also to prove the effective-ness of our own model. Table 4.3 in the following summarizes the results ob-tained from the models. The corresponding precision-recall graphs are presentedin Fig. 4.10. Additionally, the receiver operating graph for threshold is presentedin Fig. 4.9 to show that the model has good performance as the area under thecurve (AUC) represents an area of 0.8688. This graph shows that the proposeddetection model is capable to distinguish the true positives and true negative sam-ples. Besides the detection, the proposed framework can process the videos fasterthan manual speculation. For this study, The videos obtained from the roboticplatforms were in 3FPS as the camera fastened with robotic platforms need tocapture the internal condition of the water pipelines as accurately as possible. Ta-ble 4.4 in the following presents the False Positives (FP), False Negatives (FN),and a comparison between the manual assessment technique and the proposedassessment technique.FP present the instances when the model falsely predicts the positive classand FN present the instances when the model falsely predicts the negative class.From Table 4.4, it can be observed that the FP and FN values were 149 and 167,respectively. This indicates that our model has less number of incorrectly labeledpositive classes and negative classes.The results from Table 4.2 and 4.3 depict that the proposed framework cansave a good amount of time for the inspectors and can enhance the operating effi-ciency. Fig. 4.6 presents some testing images to prove that our model is efficientto detect valves in the water pipelines. Besides, our proposed algorithm can beeasily employed to detect the large defects such as root intrusion, blockage, orany obstacles type defects inside the pipelines. The requirements of this algo-rithm would be annotated data of the defects, (root intrusion, blockage, sagging,etc.) modification of the number of classes according to the number of defects,and re-training the model with the new defect data set.79Figure 4.6: Some of the valve testing sample results for our proposed model[5].4.3.4 Model Visualization and Ablation StudiesDeep neural networks are very popular to learn the hierarchical relationship be-tween the input and output of the images of a network [215]. The peripherallayers learn the gradients and the edges of the images (basic patterns). As thedepth of the layers increases for the network, the complex patterns are accumu-lated through the peripheral layers. Fig. 4.7 represents the visualization of thefeature maps to apprehend the network architecture of our proposed model. Thefeature maps in the following are from the Conv pointwise layer of 5, 11, and 13ReLu activation layers which are passed to the FPN network shown in Fig. 4.4 forfurther processing. The corresponding filters are also presented in Fig. 4.8 to showthe effectiveness of the models. The initial layers of the feature maps representthe background, texture, and edges whereas the deeper layer presents the sparseand data specific features that the model needs to learn during the training process.This feature map helps to understand that the transfer learning with a pre-trainedweight should be trained with the deeper layers to make the model learn fasterand minimize the total loss function. The filters of the networks also representthe learning process of the patterns. The initial filters (conv-pw-5-ReLu) indicatesthe basic patterns. On the other hand, the layers from 11 to 13 represent complexpatterns, which indicates the learning is based more on abstract knowledge. Thesefilters aid in the model generalization for the classes, which is a valve in our case.80If a filter remains empty, it indicates the information of the image was not presentor adequate for the particular filter to activate in the earlier layers. Table 4.3 in thefollowing presents the result for our experiments.We have also conducted an ablation study for the proposed model to detect thevalves in the water pipelines. The images (video frames) extracted from the videoswere very few in number (500 images). This is because not every frame in thevideos contained the valve. Therefore it was difficult to design a detection algo-rithm that would prove to be an effective one for valve detection. Hence, syntheticimages were generated and added to see the model performance. At first, the gen-eral image augmentation technique, which included random zoom, random fliphorizontal, random flip vertical, random rotation, and scaling were performed toincrease our sparse data set. This increased the value of 1,000 images in total.However, 1000 images were still not sufficient to prove that the proposed algo-rithm is good enough to prove the detection performance. Hence, BigGAN withstatistical modification of shifting and scaling were adapted to increase out datasetfrom 500 to 2,000 images. Table 4.1 on the following shows the effectiveness ofadopting the BigGAN with our MFPN model.Figure 4.7: Visualization of the feature maps for the DCNN architecture foran image containing valve (Images are resized for this illustration) [5].81Figure 4.8: The filters for the corresponding DCNN layers (Images are re-sized for this illustration)[5]Table 4.1: Ablation studies for the proposed networkApproach mAP valuesMFPN 0.7532Image Augmentation+MFPN 0.8454BigGAN+MFPN (Ours) 0.89114.4 SummaryThe advances in automated robotic platform are enabling the water pipe inspec-tion without interrupting the service for water supply. One challenge for the robotto navigate in the water pipe is the negotiation with installed valves. The robotneeds to recognize and activate the control mechanism to pass through the valve.Thus, this paper proposes a deep learning based automatic valve detection frame-work, by combining MobileNet-160 and Feature Pyramid Networks (MFPN). Ad-ditionally, a customized generative adversarial network is employed to mitigatethe sparse data issues, which can also improve the generality of the network. Thesolution can achieve mAP value 89.11% and can be implemented efficiently forreal-time operation. The integration of this light-weight and efficient techniquewith the robotic inspection platform will ensure a smooth operation of inspectionfor water pipe condition assessment. The contribution of this paper is threefold: 1)82Figure 4.9: Comparison of the Receiver Operating Characteristics (ROC)graph for the proposed model [5].an automatic valve detection algorithm is proposed by combining MobileNet-160and Feature Pyramid Network (FPN) to detect the valves; 2) automatic generationof synthetic images for sparse data set with customized BigGAN is presented fordata augmentation, and lastly 3) the proposed framework can be implemented forreal-time operation and it can achieve better performance in comparison with thestate-of-the-arts. With the edge computing device, the proposed solution can bedeployed to field inspection robots for online operation. From our experimentalresults and analysis, it is expected that the performance of online operations wouldremain same and the framework would assist the robotic platforms to efficientlydetect the installed valves inside the water pipelines.83Figure 4.10: Comparison of the Precision Recall (PR) curve for the proposedmodel [5].For the future work, the proposed framework can be further optimized forimproved efficiency and robustness. The performance can be boosted with a largermAP value and faster processing speed so that the robotic platforms can get moretime to respond and activate the control mechanism to pass through the valves. Asolution to tackle the data imbalanced problem for the sparse data set will also beconsidered. Additionally, it is possible to automatically classify the valve type,such as, ball, and gate, etc., identify the status, e.g., open/ closed or partiallyopen/closed, and even assess the valves’ orientation/shape. These remain topicsfor our future research and development work.84Table 4.2: Performance evaluation with the state-of-the-art modelsModel Evaluation ParametersName mAP Precision Recall F1-score AUC IoU FPS Inferencetime (s)/imageHOG+SVM 0.7446 0.7417 0.7282 0.7297 0.7891 0.6395 3.97 1.3066Wavelet+BPN 0.5717 0.5438 0.5216 0.5324 0.6312 0.4314 3.85 1.4241Wavelet+RBN 0.5231 0.5318 0.5267 0.5292 0.6714 0.4732 3.90 1.3423Wavelet+SVM 0.7712 0.7845 0.7351 0.7590 0.7342 0.5214 4.76 1.0201Table 4.3: Performance evaluation for the valve detection modelsBackbone Evaluation ParametersName mAP Precision Recall F1-score AUC IoU FPS Inferencetime (s)/imageVGG-16 0.6647 0.6231 0.6723 0.6467 0.5210 0.5385 6.90 0.9611ResNet-50 0.8898 0.8912 0.8698 0.8813 0.8497 0.6333 10.10 0.1572DenseNet-121 0.8631 0.8713 0.8499 0.8604 0.7524 0.6247 11.40 0.1391MobileNet-128 0.7941 0.8010 0.7891 0.7950 0.6732 0.63 12.38 0.0983Ours 0.8911 0.8995 0.8890 0.8942 0.8588 0.6392 12.40 0.0811Table 4.4: False Positives, False Negative and Performance on the videosobtained from robotic platformsFP FN Manual VideoAssessmentTimeAutomated VideoAssessment TimePerformanceassessment149 167 1.07 hr 14 mins 43 s85Chapter 5Real-time Valve Detection Systemwith Edge DeviceOver the past few years, several free-swimming robotic platforms have developedand have claimed to perform the monitoring, inspection, and evaluation of theburied pipelines [216]. One of the most common challenges of these robotic plat-forms is the inefficient navigation system, which disrupts the inspection processof the water pipelines. The robotic platforms are not intelligent enough to acti-vate the control mechanism in time so that it can pass through any obstacle, suchas butterfly valves, in real-time. Thus, this delay in the activation of the controlmechanism of the robotic platform will disrupt the inspection process or causea problem with the recorded inspection data. This phenomenon delay the over-all condition assessment process [6]. Therefore, to overcome the aforementionedissues, this chapter proposes to develop a real-time valve detection system withan edge device (Jetson TX2™) so that valves can be detected in real-time, andthe robotic platforms can get ready to activate their control mechanism in time.Integrating the edge valve detection solution with the in-pipe inspection systemwill enable uninterrupted video recording, improve the quality of inspection andfacilitate the analysis process for the pipeline inspectors [6].This chapter is a modification of R. Rayhana, Y. Jiao, Z. Liu, A. Wu, and X.86Kong, “Real-time embedded system for valve detection in water pipelines,” Jour-nal of Real-Time Image Processing, pp. 1–13, Springer, 2021, it is reproducedhere with the permission of Springer and the copyright is reserved with the au-thors.5.1 Real-time Valve Detection System with EdgeDeviceTo address the current challenge of robotic platforms to navigate through thevalves inside water pipeline, an efficient real-time lightweight valve detection al-gorithm with an edge device is proposed. This will ensure that that valves canbe detected in real time, and the robotic platforms can get ready to activate theircontrol mechanism in time. The contributions of this chapter are summarized asfollows:• This paper combines the technological novelties of the object detection al-gorithm, “You Only Look Once” (YOLOv4), with an light-weight valve de-tector (MobileNetv3) for the edge computing system so that the in-pipe in-spection tool can activate the control mechanism to pass through the valvesin real time without interrupting the in-pipe inspection.• This work unleashes a new concept on the real-time video processing forautomated water pipeline inspection and ensures a continuous data acquisi-tion during the inspection.5.2 Proposed FrameworkFigure 5.1 an overview of the real-time valve detection system for water pipelineswith the edge device. The system comprises three main sections; image pre-processing, the valve detection algorithm, and the edge device, Jetson TX2™.The details of each section are provided in the following.87Figure 5.1: Overview of the real-time valve detection system with the edgedevice ©Rakiba Rayhana5.2.1 Image pre-processingFor this study, the saved (offline) videos are used to design this real-time valve de-tection system. The video frames (images) were then extracted from the videos.Around 378 images (1920 × 1080 pixels) containing valves were used in thesestudies. However, as the number of the images was not sufficient enough forDCNN architectures, image augmentation methods were used to increase thedataset to around 900 images. In this study, a conventional image augmentationmethod is employed. The image augmentation that was used are: random horizon-88tal and vertical flip, scaling, random rotation, and random zoom. After the imageaugmentation, the new images were labeled using open-source software named“LabelImg.” Both the augmented images and original images were fed into thenetwork. From the new dataset (the augmented plus original ones), 70 % of theannotated images were used for training, 10% of images were used in validationand 20% were used for testing.5.2.2 Real-time valve detection algorithmThe valves are round in shape and come in different colors and outlooks. The de-tection of the valves are unique from other problem as it can be partially openedor closed, or fully closed. Therefore, the detection algorithm needs to identifyclosed or partially opened/closed conditions. Figure 5.2 shows the proposed archi-tecture, YOMO for the valve detection in water pipeline. The original backboneof YOLOv4 proposed by Bochkovskiy et al. [217] is CSPDarkNet. However, inthis study CSPDarkNet is replaced by the light-weight backbone, MobileNetv3.The structure of MobileNetv3 involves depth-wise separable convolution, inverseresidual block, light-weight attention mechanism, and h-swipe instead of swipefunction, which can significantly reduce the parameters and computation of themodel [218, 219]. Hence, the blocks are named as BN in Fig. 5.2 The depthwise separable convolution is mainly developed for mobile or light-weight appli-cations. The details of each of the convolutional layers are discussed in chapter4 of this thesis. At first, the residual blocks reduce the dimensions of the imagesand then convolute to obtain the features of the images. However, the reductionin dimension can cause a large number of detailed information to be lost. There-fore, depth-wise separable convolution are used to expand the dimension, withoutincreasing the number of parameters. Afterwards, the dimensions of the extractedfeatures are increased through 1×1 convolution and added with the input imagesto obtain the output. This helps to reduce the issue of gradient explosion or thevanishing gradient, while the model is training.To efficiently detect the conditions of the valve, we have employed an up-89Figure 5.2: The proposed architecture (YOMO) for real-time valve detectionwith edge device ©Rakiba Rayhanagraded version of You Only Look Once (YOLO) [141], which is YOLOv4 [217].YOLOv4 is a subsequent upgrade of the YOLO [141] . YOLOv2 (YOLO9000)[143], and YOLOv3 [220]. YOLOv4 was proposed by Bochkovskiy et al. [217] asan improvement to YOLOv3. The main objective of YOLOv4 was to obtain a fastoperating speed of the object detector along with optimized parallel computationso that real-time analysis with higher accuracy can be achieved. The YOLOv4comprises of a backbone for feature extraction, a neck for semantic representa-tion of extracted features, and head to obtain the predictions [221]. In the originalYOLOv4 structure, the backbone CSPDarkNet53 was integrated with SPP block.However, in this study, we employ MobileNetv3 instead of CSPDarkNet53 sothat the fast and accurate valve detection can take place in the edge device. The90SPP block is used to combine feature information from different feature layersthrough four different max-pooling layers. The max-pooled information obtainedfrom SPP is concatenated and convoluted to enhance the receptive ability of thedetection network. Afterwards, the obtained information from the MobileNet (thebackbone) for the three layers ( 64× 64, 32× 32, and 16× 16) and SPP blockare convoluted and up-sampled. To obtain the semantic features, the features ob-tained through the backbone are convoluted, concatenated, upsampled and thendownsampled to improve the feature fusion approach. In this study, we employedbi-cubic linear interpolation method to improve the upsampling approach of thealgorithm. Bi-cubic linear interpolation method takes 16 coordinates from the in-put images and attempts to locate the pixel values of the valves more accurately, sothat maximum features can be fused together and used for upsampling approach.The neck of YOLOv4 acquires the rich (feature-fused) method for accuratepredictions. Afterwards, the YOLOv4 model predicts the bounding boxes by us-ing the YOLO head at three different levels. The prediction process is presentedin Fig. 5.3. In the first level, the input image feature is discretized into N×Nevenly spaced grids. If the target or the object belongs to the grid cells, the modelgenerates a confidence score and a predictive bounding boxes. The confidencescore can be defined as equation 5.1. The bounding boxes are determined throughthe cluster of anchor boxes defined in equation 5.2, 5.3, 5.4, and 5.5.Con f idence = P(ob j)× IoU |gpred,P(ob j) ∈ {0,1} (5.1)bx = σ(tx)+ cx (5.2)by = σ(ty)+ cy (5.3)bw = pwetw (5.4)bh = pheth (5.5)Here, σx defines the horizontal aspect ratio of the predicted boxes comparedto the input images, bx,by and bw,bh represents the centroid values and the size91of the predicted bounding boxes respectively. The cx,cy and pw, ph represents thecentroid values and size of the bounding boxes for the feature maps. Finally, thecentre offset value of the bounding box prediction from the entire network and thesubsequent scaling size is defined with tx, ty and tw, th [221].The best prediction of the bounding boxes is then filtered out by using non-maximum suppression (NMS) [80] technique. YOLOv4 also employs drop blockregularization [222] technique to learn spatially distinct features and class labelsmoothing [217] to improve the generalization ability of the model [221]. Thishas drastically improved the detection speed of all other state-of-the-art objectdetection algorithms. Hence, this algorithm was thought to be the most suitableto be integrated with the edge device and perform the real-time valve detection inwater pipelines.Figure 5.3: An illustration of the prediction process for valve detection withYOMO ©Rakiba Rayhana and [6]925.2.3 Edge DeviceOur computing device is Jetson TX2™, which is an edge device manufacturedby NVIDIA. Fig. 5.4 shows a typical Jetson TX2™ module. It is the secondgeneration of Jetson TX1™, which has been used in many computing vision ap-plications [223–228]. It is developed around the 256-core NVIDIA Pascal archi-tectures with 128-bit CPU memory. It also has a 4K video encode and decodecapability with 1400 Megapixels/sec. NVIDIA Jetson TX2™ is considered to bean ideal computing platform to operate end-to-end pipelines for automated opera-tion. It can assimilate data from multiple sensors such as camera, ultrasonic, radiofrequency (RF), LIDAR sensors and operate the decoding/encoding, detecting/-classifying with the aid of the GPU computing ability. It can simultaneously storethe processed data in cloud computing services and also can output the detectionresults through a display monitor. An illustration of this pipeline is presented inFig. 5.5. In addition to that, the specifications of the Jetson TX2™ [7] are pre-sented in Table 5.1. NVIDIA Jetson TX2™ can perform the GPU computation byusing NVIDIA’s Compute Unified Device Architecture (CUDA) computing plat-form. It is one of the powerful computer vision edge computing platforms, whichoperates with 7.5 Watt (very low) power consumption [7].Table 5.1: Specifications for NVIDIA Jetson TX2™Parts SpecificationsCPU ARM Cortex-A57 (quad-core) + NVIDIA Denver-2GPU 256-core Pascal @ 1300MHzMemory 8GB 128-bit LPDDR4 (59.7 GB/s)Operating system Ubuntu Linux 14.04 LTSVideo Encode 4K 60Hz, 4K 30HzVideo Decode 4K 60HzDisplay eDP 1.2 (2x MIPI DSI, 2x HDMI 2.0 / DP 1.2)Storage 32GB eMMC 5.1Connectivity Connects to 802.11a Wi-Fi and Bluetooth 4.1Networking 1 Gigabit Ethernet93Figure 5.4: NVIDIA Jetson TX2 module (Image Courtesy: NVIDIA [7])[ USB: Universal Serial Bus, M.2 E: Next Generation Form Factor,HDMI: High Definition Multimedia Interface, SD: Secure Digital,PCIE: Peripheral Component Interconnect Express, STATA: Statisti-cal Package, JTAG: Joint Test Action Group, UART: Universal Asyn-chronous Receiver-TransmitterThe operating system (OS) for the edge device, Jetson TX2™ is Linux-14.04LTS version. However, as the Linux OS has limited abilities to process the bigdata or for signal processing, some additional software was required to install forJetson TX2™. A development package, provided by NVIDIA Developer is in-stalled. The development package is known as “JetPack.” For this experimentJetPack 4.3 is installed as it is compatible with the NVIDIA Jetson TX2™. TheJetPack 4.3 comprises of development tools for the host (the Ubuntu CPU) and94Figure 5.5: End-to-end pipeline of NVIDIA Jetson TX2 [7] [LIDAR: LightDetection and Ranging, RF: Radio Frequency, USB: Universal Se-rial Bus, PCIE: Peripheral Component Interconnect Express, MIPICSI: Mobile Industry Processor Interface and Camera Serial Interface,ISP: Internet Service Provider, H.265: High Efficiency Video Cod-ing, CUDA: Compute Unified Device Architecture, cuDNN: ComputeUnified Device Architecture Deep Neural Network, Open GL: OpenGraphics Library, CANbus: Controller Area Network bus].the Jetson TX2™, CUDA, Image processing tools, OpenCV, APIs (samples, doc-umentation, and middlewares) and with a 16 nm NVIDIA Tegra “Parker” system.As multiple models were tested to observe the performance of the valve detectionmodels, an SD card of size 256 GB is used to increase the storage memory of theJetson TX2™. The SD card is flashed with the JetPack 4.3 tool kit and configured.The configuration process requires an active internet connection and interaction ofthe host and the target (Jetson TX2™) to set up via a USB 3.0 port [229].Once the setup is completed, Jetson TX2™ is disconnected from the host as itcan operate as a stand-alone device. The proposed light-weight YOMO algorithmis then fed to the edge device to make it an integrated platform.5.3 Experimental Results and AnalysisThe algorithm is written in python 3.6 which requires TensorFlow, Keras, Numpy,and open-cv and CUDA tool kit (CUDA-9.00 and CUDNN 7.41) as its depen-dencies. A local environment with the aforementioned dependencies should beinstalled in the operating system.95Once the aforementioned dependencies and environment are set, the detectionmodels are trained and tested on NVIDIA GeForce GTX 1070. The hyperparam-eters are then tuned according to the dataset of valve detection. Table 5.2 showsthe initialization of the hyperparameters that are used for this experiment. TheYOMO resizes the images from 1920× 1080 to 416× 416. Once all the stepsabove are ensured, the training for the model is started. The training step for themodel is set to 5,000, with a batch size of 32, momentum of 0.9, learning rateto 0.001, and a decay rate of 0.0005. The loss curve of the proposed model ispresented in Fig 5.6. Once the training is completed, the models are tested on theNVIDIA GeForce GTX 1070 and on the NVIDIA Jetson TX2™ to evaluate theperformance. The evaluation indices that are used for YOMO are briefly explainedin the following section.Table 5.2: Initialization parameters of YOMO for valve detectionInput size Batch size Momentum Learning rate Decay Training steps1920×1080 16 0.9 0.001 0.0005 5,0005.3.1 Evaluation MetricsThe problem that we are trying to address in this paper is a binary classificationproblem. In other words, the detector should detect whether there is a valve (posi-tive(1)) or no valve (negative(0)) in the image/video frames. However, the sampleimages can be categorized into four different types, such as True Positive (TP),False Positive (FP), True Negative (TN), and False Negative (FN). When the la-beled image and predicted image are positive, the images are defined as TP. Ifthe labeled image is positive but the predicted image is negative then the imagesare defined as FN. Similarly, when the labeled image is negative but the predictedimage is positive, the images are defined as FP. When both the labeled image andpredicted image are negative, the images are defined as TN. A table with TP, FP,TN, FN values is called a confusion matrix. Table 5.3 summarizes the definitionof the confusion matrix.96Figure 5.6: The training and the validation loss curves of the proposed model©Rakiba RayhanaThe important metrics that are used for the evaluation process are, Precision,Recall, F1-score and mean Average Precision (mAP) values. However, these eval-uation metrics are dependent upon certain parameters such as Intersection overUnion (IoU). A brief description of each of the metrics and parameters is pro-vided in the following for a better understanding of the evaluation process.5.3.1.1 PrecisionA precision is defined by the TP values over the sum of TP and FP values [230].Precision =T PT P+FP(5.6)97Table 5.3: Confusion Matrix (CM) for typical classification taskLabeled image Predicted image Confusion MatrixNegative (0) Negative (0) TNNegative (0) Positive (1) FPNegative (0) Positive (1) FPPositive (1) Positive (1) TP5.3.1.2 RecallA recall value is defined as the TP values over the sum of TP and FN values [230].Recall =T PT P+FN(5.7)5.3.1.3 F1-ScoreF1-score is defined by two times the product of precision and recall values over thesum of the precision and recall values. The F1-score indicates the performancesof the evaluated models [230].F1-Score =2×Precision×RecallPrecision+Recall(5.8)5.3.1.4 Intersection over Union (IoU)Intersection over Union (IoU), which is also known as a Jaccard index is an im-portant evaluation metric for object detection algorithm. It is mostly used forcomparing the similarity between the locations, heights, and widths of the twobounding boxes and converts them into regional attributes and areas [231]. There-fore, mathematically IoU is defined as the ratio of the overlapping region betweenthe ground truth boxes (G) and the predicted boxes (P) on the images over thetotal region of ground truth (G) and predicted boxes (P). IoU helps to understandthe accuracy of the objects, which in our case is the valve.98IoU =G∩PG∪P (5.9)5.3.2 Results and Comparative Analysis for State-of-the-artAlgorithmsIn this section, the results of YOMO are compared with the results achieved by thestate-of-the-art real-time algorithms, such as YOLOv2, YOLOv3-tiny, and SSD(inception, VGG (300, 500), mobilenetv1) backbone. To make the comparisoninsightful, a brief description of the algorithms (YOLOv2 and SSD) is provided.YOLOv2 is the previous version of YOLOv3, and the description can be foundin [143]. The description of YOLOv3 can also be found in [144]. YOLOv3-tiny is a modified version of the YOLOv3 and it can be found in [6]. However,the SSD [142] algorithm processes image and video by using multi-scale featuremaps to extract features from the images with the aid of Visual Geometry Group(VGG)-16 and divides them into cells. Each cell can make four predictions andapplies convolutional layers for object detection. For boundary box regression,the multi-box technique is used. The loss function of multi-box detectors com-prises confidence (confidence value of the objectness inside the bounding box)and location loss (deviation values from the ground truth values to the predictedvalues inside the bounding box. SSD employs Categorical Cross-Entropy (CCE)for the confidence loss L-2 norm for the location loss.Once the algorithms are selected, the models were trained with the images andtested with the videos of water pipelines containing the valves. The performanceevaluation of YOMO for real-time valve detection is presented in Table 5.4.The detection speed in frame per second (FPS) in Table 5.4 is for NVIDIAGeForce GTX 1070 GPU. The performance evaluation for valve detection inNVIDIA Jetson TX2 ™ is presented in Table 5.5. The precision-recall valuesare obtained when the IoU values were set to 0.75. The precision-recall curvefor all the models (SSD-VGG (300), SSD-VGG (500), SSD-MobileNetv1, SSD-Inception, YOLOv2, YOLOv3-tiny, and YOMO) that are compared in this section99is shown in Fig. 5.7.Figure 5.7: The precision recall (PR) curves of the models ©Rakiba Ray-hana.5.3.3 DiscussionsFrom Table 5.4, it can be observed that YOLOv2 has the lowest mAP, precision,and recall values. However, our proposed model seems to outperform all the otherreal-time object detectors interms of the evaluation metrics. Therefore, our pro-posed algorithm seems to be suitable for the real-time valve detection task. Thus,YOMO was tested in NVIDIA Jetson TX2™ to observe the real-time detectionon the videos.As the model is transferred to Jetson TX2™, the Tensorflow algorithm is con-100(a) Test sample image-1 (b) Test sample image-2Figure 5.8: Some testing samples for the real-time valve detection with edgedevice, Jetson TX2 ©Rakiba Rayhanaverted to TensorRT to make it lightweight and the performance level was set tothe maximum level. After the completion of the required settings, the model wastested to observe the performance. It was observed there was a slight decreasein terms of mAP, precision, recall, and F1-score. However, the detection speed(FPS) was greatly reduced from 67.0 to 12.4 FPS (shown in Table 5.5). This isdue to the difference in the specification variations between the two GPUs. Fromthe perspective of real-time application, this speed is acceptable as the in-pipe in-spection platform normally moves along the pipeline at a relatively slow speed(0.15m/s [232]) and grabs the frame at 3 FPS. Table 5.6 in the following presentsthe time for the manual detection approach and the automated approach. Here,the manual approach indicates a rough estimate of the time that is spent by the op-erators to check individual frames. However, the automated approach shows howfast the total video is processed with a detection time. The frame saved indicatesthat how many frames skipped with our proposed framework. Therefore, the in-tegration of the edge device with this algorithm on the in-pipe inspection tool canhelp the tool to prepare itself and activate the control mechanism to pass throughthe valves. Fig. 5.8 shows some of the testing results for the proposed model onthe edge device, NVIDIA Jetson TX2™101Table 5.4: Performance evaluation for the proposed real-time valve detectionalgorithmReal-timeobject dee-tectorsmAP(%) Precision Recall F1-score Detectionspeed(FPS)SizeSSD-VGG(300)66.92 0.6009 0.7719 0.6613 8.0 392MBSSD-VGG(500)67.02 0.6012 0.7899 0.6732 8.1 330MBSSD-MobileNetv167.16 0.6000 0.7889 0.6921 20.0 15MBSSD-Inception67.17 0.6066 0.7912 0.6862 9.8 50MBYOLOv2 61.94 0.6140 0.6346 0.6241 10.2 48.91MBYOLOv3-tiny67.17 0.6157 0.7924 0.6929 65.0 16MBYOMO(ours)76.21 0.7345 0.8321 0.7802 67.0 15MBTable 5.5: Performance evaluation for valve detection in NVIDIA JetsonTX2™Real-time objectdetectorsmAP(%) Precision Recall F1-score Detectionspeed(FPS)YOMO 70.32 0.7971 0.7342 0.7643 12.4Table 5.6: Performance assessment (with respect to time) between the man-ual and automated detection systemVideo length Manual approach Automated approach Frame saved1 hr 15 mins 5 hrs 45 mins 15,700 frames5.4 SummaryThis chapter proposes a real-time valve detection system as a portable solution forthe water pipeline inspection to ensure an uninterrupted process and enhance thecondition assessment technique. The experiments with the edge device (YOMOwith Jetson TX2™) can achieve higher mAP values and detection speed (in FPS)102for the real-time video processing. The proposed solution was also comparedwith the state-of-the-art algorithms. The comparative study shows that the pro-posed model outperforms the state-of-the-art algorithms. Therefore, the integra-tion of the proposed edge device into the in-pipe inspection platform can detectthe valves in real-time and assist the platform to negotiate through valves. The fu-ture extension of this study will optimize and improve the mAP values so that thereliability of the edge device can be further improved. Additionally, the integra-tion of the subroi reduction techniques presented by De Vitis et al. [233], can alsobe employed with the proposed YOMO network to observe the detection speed ofthe model with respect to mAP values.103Chapter 6ConclusionsThe water transmission system that serves our cities and towns is one of the mostexpensive and complex networks and remains unnoticed until a failure occurs[234]. Therefore, periodic inspection of the water pipelines to assess the condi-tion of the water pipeline is crucial to avoid any unforeseen disasters (e.g., flood-ing, etc.). Many municipalities use in-pipe inspection robotic platforms to con-duct routine inspections. The in-pipe inspection robotic platforms are equippedwith cameras, which can record the internal condition of the pipelines. However,post-processing of the video recordings obtained by the video platforms is one ofthe primary challenges that the municipalities face. This is because the humanoperators manually assess the recordings to identify the defects inside the waterpipelines, which is quite error-prone, laborious, and time-consuming. In additionto the manual assessment challenges, there are some challenges regarding the au-tonomous navigation of the in-pipe inspection robotic platforms, as the roboticplatforms operate in a very constrained environment with limited knowledge.Therefore, in this thesis, the main objectives were to develop an automateddefect characterization system so that the post-assessment process can be fast, ef-ficient, accurate, and reliable. This will reduce the burden on the pipeline inspec-tors and contribute to the overall condition assessment process of the pipelines.In addition, an automated valve detection algorithm is developed to ensure the104valves are detected beforehand and smooth autonomous navigation for the in-pipeinspection robotic platform occurs.Chapter 3 presented a robust, efficient, and accurate defect characterizationalgorithm for the inspection videos obtained through the inspection robotic plat-form. The algorithm comprises of channel-spatial attention mechanism, threeresidual networks as backbone, and MaskC-RCNN to identify, locate, and ob-tain the geometrical shape of the defects inside the water pipelines. An ensemblemechanism is also employed to ensure the accuracy of the defect detection is im-proved. Based on the algorithm, a simple user-friendly tool is developed, whichcan effectively process the videos and detect the defects efficiently. The experi-mental results show that the proposed model is effective, efficient and robust tocharacterize the defects inside the water pipelines compared to the state-of-the-art.Chapter 4 proposed a deep-learning based automated valve detection frame-work to detect the valves inside the water pipeline. The framework is developed byusing MobileNet-160 as a lightweight backbone and feature pyramid layer (FPN)as the valve detector. The framework is named as MFPN. The experimental re-sults of the evaluation metrics show that the proposed model can detect the valvesefficiently and can outperform the traditional machine learning algorithms withhigher video processing speed (in frame per second).Chapter 5 proposed the system implementation of valve detection with a pow-erful edge device, NVIDIA Jetson TX2, to conduct real-time valve detection inwater pipelines. The main objective of this chapter is to propose a solution whichcan be integrated easily with the robotic platform to perform real-time valve de-tection so that the autonomous navigation system of the in-pipe inspection roboticplatforms can be enhanced. Hence, a new lightweight valve detector, YOMO, isproposed for the real-time valve detection of the water pipeline. The proposedframework is then integrated with Jetson TX2 to observe the detection accuracyand processing speed. The results from this experiment show that this system canbe used in the robotic platforms to enhance the autonomous navigation system ofthe in-pipe inspection robotic platforms.1056.1 ContributionsThe main contributions of this study are summarized in the following.• A comprehensive review of the state-of-the-art robotic platforms and theautomated/computational frameworks to automate the inspection process ofthe water pipelines is published. The study highlights the pressing issues,and trends and provides a potential solution for future research on the waterpipelines’ inspection process and condition assessment.• An automated defect characterization algorithm and a user-friendly tool isproposed to detect the defects, types, and locations of defects effectivelyand efficiently. Integration of this algorithm will allow obtaining reliableand accurate defect analysis of the water pipeline. It can reduce the burdenof the manual defect assessment process of the water pipeline.• The valves inside the pipelines, which control the water flow, are auto-matically detected from the inspection videos. The study proposes a deeplearning-based framework to effectively defect the valves inside the waterpipelines to improve the videos’ accuracy and processing/detection speedfor the sparse data set. This is the first kind of study to detect the valvesfrom the inspection videos of water pipelines. Integration of the frame-works to the robotic platforms can improve the navigation system and en-sure a smooth videotaping process for the water pipeline.• A real-time valve detection system with the edge device is proposed sothat it can be easily integrated to the in-pipe inspection robotic platforms.The study suggests a lightweight framework for real-time valve detectionwith NVIDIA Jetson TX2. The results from the system integration showedpromising performance in terms of detection accuracy and video process-ing. Therefore, integrating this edge device with the robotic platforms canimprove the real-time detection and navigation system for the in-pipe in-spection robotic platforms.106The contributions of this research aims to will facilitate both the inspectionand condition assessment of water pipelines, which will improve the overall man-agement system of water municipalities. Besides water municipalities or waterpipeline management industry, the developed algorithm for defect detection in wa-ter pipelines can be used to detect any small/minute structures such as cancerouscells in human body or disease in plants. The algorithms developed for the valvedetection can be employed to solve any large objects such as ships, pedestrian andso on. Lastly, the developed real-time detection algorithm can be employed forreal-time vehicle detection and real-time manufacturing applications.6.2 Future WorkThe proposed models in this thesis have proved their effectiveness in the inspec-tion process of water pipelines. The limitations and the future works of this re-search are presented in the following.• The proposed defect characterization model rendered wrong predictions onseveral defects. One potential reason could be the data set’s class imbalanceissues, which resulted in low predictive accuracy for the minority classes.Also, some of the defects, such as corrosion and metal loss defects, lookedvery similar to each other. This could also reduce the accuracy of the de-fect detection models. Therefore, the model performance can be improvedby using a larger data set with various types of defects. Re-sampling tech-niques and synthetic data generation techniques can be employed to increasethe minority class data so that the overall performance of the defect charac-terization model is improved.• The training data and computational speed of the training device (wherethe training takes place) plays an important role in inference time as well.Therefore, some advanced deep learning based frameworks should be de-veloped to factor in the issue of number of training data and computationalspeed of the device.107• The current proposed valve detection model is developed on offline videos,primarily by using still images of valves. However, if the location infor-mation, such as the mapping information, could be included in the model,the model could improve the valve detection accuracy and also provide thelocation information of the valves. Hence, including the geographic infor-mation system (GIS) information to locate the valves can be one way toenhance the performance of the proposed model.• The proposed real-time valve detection system is currently developed onthe edge device NVIDIA Jetson TX2. However, over the course of time,upgraded versions of the edge systems will arrive in the market and mightreplace the current edge device. One potential solution to avoid such kind ofproblem will be to use transfer learning of the trained models and integrateit with the updated edge devices. Also, the proposed framework shouldbe compatible with the new device by using hyper-parameter tuning of theessential parameters.Besides the topics covered in this research, environment prediction using the in-spection video and GIS information for the in-pipe inspection platform can beanother interesting topic for future research. If the in-pipe inspection robotic plat-form can predict the environment, it can maintain its position in the center andrecord the pipeline’s conditions adequately. Also, it can help the robotic platformsto pass through the elbows and T-zone of the pipelines. The future development ofthe robotic platforms can also include the multi-sensor inspection system to detectas much as pipe defects. This will also improve the autonomous navigation of thein-pipe inspection platform and contribute to the enhanced condition assessmentof the water pipeline.108Bibliography[1] R. Rayhana, Y. Jiao, A. Zaji, and Z. Liu, “Automated vision systems forcondition assessment of sewer and water pipelines,” IEEE Transactions onAutomation Science and Engineering, 2020. → pagesxiii, xv, 10, 11, 12, 17, 20, 21, 38[2] R. Harris and C. Dobson, “Sewer pipe infiltration assessment: Comparisonof electro-scan, joint pressure testing and cctv inspection,” in Pipelines2006: Service to the OwnerAmerican Society of Civil Engineers, 2006. →pages xv, 19, 20[3] R. Harris and J. Tasello, “Sewer leak detection-electro-scan adds a newdimension, case study: City of redding, california,” in Pipelines 2004:What’s on the Horizon? American Society of Civil EngineersPrestressedConcrete Cylinder Pipe Users ForumNational Association of CorrosionEngineers, 2004. → pages xv, 19, 20[4] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional blockattention module,” in Proceedings of the European conference oncomputer vision (ECCV), 2018, pp. 3–19. → pages xv, 45, 46, 49, 61[5] R. Rayhana, Y. Jiao, Z. Bahrami, Z. T. Liu, A. Wu, and X. Kong, “Valvedetection for autonomous water pipeline inspection platform,”IEEE/ASME Transactions on Mechatronics, 2021. → pagesxvi, xvii, 70, 72, 74, 77, 80, 81, 82, 83, 84[6] R. Rayhana, Y. Jiao, Z. Liu, A. Wu, and X. Kong, “Real-time embeddedsystem for valve detection in water pipelines,” Journal of Real-Time ImageProcessing, pp. 1–13, 2021. → pages xvii, 86, 92, 99[7] D. Franklin, “Nvidia jetson tx2 delivers twice the intelligence to the edge,”NVIDIA Developer Blog, 2017. → pages xvii, xviii, 93, 94, 95109[8] S. Folkman, “Water main break rates in the usa and canada: Acomprehensive study,” 2018. → page 1[9] M. Stephens, M. Lambert, A. Simpson, J. Nixon, and J. Vitkovsky, “Waterpipeline condition assessment using transient response analysis,” 2005. →pages 1, 2[10] T. Sangster, “Pipeline condition assessment. the essential engineering stepin the asset management process,” PureTechltd. → page 2[11] Z. Liu and Y. Kleiner, “State of the art review of inspection technologiesfor condition assessment of water pipes,” Measurement, vol. 46, no. 1, pp.1–15, 2013. → pages 2, 14, 15, 16, 17[12] J. M. M. Tur and W. Garthwaite, “Robotic devices for water main in-pipeinspection: A survey,” Journal of Field Robotics, vol. 4, no. 27, pp.491–508, 2010. → pages 2, 10, 11, 12, 13, 14[13] S. K. Dwivedi, M. Vishwakarma, and A. Soni, “Advances and researcheson non destructive testing: A review,” Materials Today: Proceedings,vol. 5, no. 2, pp. 3690–3698, 2018. → page 2[14] “Visual testing 101: A look at ndtaˆC™s oldest and most commontechnique,” 20/20 NDT, 2016, https://2020ndt.com. → page 2[15] B. Rajani and Y. Kleiner, “Non-destructive inspection techniques todetermine structural distress indicators in water mains,” Evaluation andControl of Water Loss in Urban Water Networks, pp. 21–25, 2004. →page 2[16] J. Moraleda, A. Ollero, and M. Orte, “A robotic system for internalinspection of water pipelines,” IEEE Robotics & Automation Magazine,vol. 6, no. 3, pp. 30–41, 1999. → page 2[17] O. Duran, K. Althoefer, and L. D. Seneviratne, “A sensor for pipeinspection: model, analysis and image extraction,” in Proceedings 2003International Conference on Image Processing (Cat. No. 03CH37429),vol. 3. IEEE, 2003, pp. III–597. → page 2[18] A. N. Tafuri and A. Selvakumar, “Wastewater collection systeminfrastructure research needs in the usa,” Urban water, vol. 4, no. 1, pp.21–29, 2002. → page 4110[19] W. Guo, L. Soibelman, and J. Garrett, Jr, “Automated defect detection inurban wastewater pipes using invariant features found in video images,” inConstruction Research Congress 2009: Building A Sustainable Future,2009, pp. 1194–1203. → pages 4, 28[20] P. Water and S. Authority, “Monitoring & sampling plan.” → page 4[21] B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg, “A survey of research oncloud robotics and automation,” IEEE Transactions on AutomationScience and Engineering, vol. 12, no. 2, pp. 398–409, 2015. → page 10[22] H. Schempf, E. Mutschler, V. Goltsberg, G. Skoptsov, A. Gavaert, andG. Vradis, “Explorer: Untethered real-time gas main assessment robotsystem,” in Proceedings of International Workshop on Advances inService Robotics, ASER, vol. 3, 2003. → page 10[23] D. I. Stein and R. I. Stein, “Pirat system,” Trenchless Technology Centre,2004, https://www.unitracc.com. → page 10[24] A. Ratliff, “An overview of current an developing technologies for pipecondition assessment,” in Pipeline Engineering and ConstructionInternational Conference 2003American Society of Civil Engineers, no.Volume 1, 2003. → pages 11, 12[25] M. Kolesnik and H. Streich, “Visual orientation and motion control ofmakro-adaptation to the sewer environment,” in Proc. Int. Conf.Simulation of Adaptive Behavior. Citeseer, 2002, pp. 62–29. → page 11[26] “Pipe crawler—innovation & research,” Inspector Systems, 2018,Available:https://inspector-systems.com, Accessed on: July 14, 2019. →page 11[27] “Rapid view,” PANORAMO 3d Optical Scanner, 2007, Available:http://www.rapidview.com,Accessed on: July 16, 2019. → page 12[28] “Redzone robotics,” Available: https://www.redzone.com,Accessed on:July 16, 2019. → page 12[29] A. A. Nassiraei, Y. Kawamura, A. Ahrary, Y. Mikuriya, and K. Ishii,“Concept and design of a fully autonomous sewer pipe inspection mobilerobot” kantaro”,” in Proceedings 2007 IEEE International Conference onRobotics and Automation. IEEE, 2007, pp. 136–143. → page 13111[30] A. Ghadei, A. Nassiraei, and K. Ishii, “Pipe inspection robot system forunderground infrastructure with an emergency release mechanism,” in2008 34th Annual Conference of IEEE Industrial Electronics. IEEE,2008, pp. 1566–1571. → page 13[31] I. Clarke, “Trunk main leak location–development of a new locatorsystem,” No-Dig International, vol. 11, no. 12, 2000. → pages 13, 14[32] “Sahara-pipeline inspection system,pure technologies-a xylem brand,”xprt environmental, Available: https://www.environmental-expert.com,Accessed on: December 15, 2019. → page 14[33] “Xylem, lets solve water,” walledlakel, Available: https://walledlake.us,Accessed on: December 16, 2019. → page 14[34] S. Costello, D. Chapman, C. Rogers, and N. Metje, “Underground assetlocation and condition assessment technologies,” Tunnelling andUnderground Space Technology, vol. 22, no. 5-6, pp. 524–542, 2007. →pages 15, 16[35] T. Hao, C. Rogers, N. Metje, D. Chapman, J. Muggleton, K. Foo, P. Wang,S. R. Pennock, P. Atkins, S. Swingler et al., “Condition assessment of theburied utility service infrastructure,” Tunnelling and Underground SpaceTechnology, vol. 28, pp. 331–344, 2012. → page 16[36] Z. Liu, H. Ukida, P. Ramuhalli, and K. Niel, Integrated Imaging andVision Techniques for Industrial Inspection. Springer, 2015. → page 16[37] M. G. Wade and D. Buonadonna, “Large-diameter rcp conditionassessment program—a progressive approach,” in Pipelines 2010:Climbing New Peaks to Infrastructure Reliability: Renew, Rehab, andReinvest, 2010, pp. 979–988. → page 17[38] A. Selvakumar, M. E. Tuccillo, K. D. Martel, J. C. Matthews, andC. Feeney, “Demonstration and evaluation of state-of-the-art wastewatercollection systems condition assessment technologies,” Journal ofPipeline Systems Engineering and Practice, vol. 5, no. 2, p. 04013018,2013. → pages 17, 18, 19112[39] K. Bainbridge and H. Krinas, “Efficient condition management of sewersystems using both zoom camera technology and “traditional cctv.”,” inProc. Of the NASTT No-Dig Conference, 2008. → page 17[40] J. Makar, “Diagnostic techniques for sewer systems,” Journal ofInfrastructure Systems, vol. 5, no. 2, pp. 69–78, 1999. → page 18[41] T. Iseley and A. Ratliff, “Digital imaging for characterizing pipelinedefects,” Proc., NASTT No-Dig, 2002. → page 18[42] M. Knight, R. Younis, B. Barrall, J. Russin, and C. Manners, “Advances inpipeline asset management using digital side scanning evaluationtechnology,” in Proc. Of the NASTT No-Dig Conference, 2009. → page 18[43] Y. J. Jung and S. K. Sinha, “Evaluation of trenchless technology methodsfor municipal infrastructure system,” Journal of infrastructure systems,vol. 13, no. 2, pp. 144–156, 2007. → page 18[44] R. Wirahadikusumah, D. M. Abraham, T. Iseley, and R. K. Prasanth,“Assessment technologies for sewer system rehabilitation,” Automation inConstruction, vol. 7, no. 4, pp. 259–270, 1998. → page 18[45] M. E. Tuccillo, C. Wilmut, C. Feeney, K. Martel, and A. Selvakumar,“Field demonstration of electro-scan defect location technology forcondition assessment of wastewater collection systems,” Proceedings ofthe Water Environment Federation, vol. 2011, no. 5, pp. 265–281, 2011.→ page 19[46] C. Hansen and A. O’Keefe, “Update from christchurch on conditionassessment of drains,” Proceedings materials NO-DIG, p. 10, 2013. →page 19[47] S. Gokhale and J. A. Graham, “A new development in locating leaks insanitary sewers,” Tunnelling and underground space technology, vol. 19,no. 1, pp. 85–96, 2004. → page 19[48] M. Wang, H. Luo, and J. C. Cheng, “Towards an automated conditionassessment framework of underground sewer pipes based on closed-circuittelevision (cctv) images,” Tunnelling and Underground Space Technology,vol. 110, p. 103840, 2021. → page 19113[49] S. S. Kumar, D. M. Abraham, M. R. Jahanshahi, T. Iseley, and J. Starr,“Automated defect classification in sewer closed circuit televisioninspections using deep convolutional neural networks,” Automation inConstruction, vol. 91, pp. 273–283, 2018. → pages 19, 25, 40[50] H. Fares and T. Zayed, “Hierarchical fuzzy expert system for risk offailure of water mains,” Journal of Pipeline Systems Engineering andPractice, vol. 1, no. 1, pp. 53–62, 2010. → page 20[51] Y. Yu, A. Safari, X. Niu, B. Drinkwater, and K. V. Horoshenkov, “Acousticand ultrasonic techniques for defect detection and condition monitoring inwater and sewerage pipes: A review,” Applied Acoustics, vol. 183, p.108282, 2021. → page 21[52] M. Sonka, V. Hlavac, and R. Boyle, Image processing, analysis, andmachine vision. Cengage Learning, 2014. → page 22[53] W. Guo, L. Soibelman, and J. H. Garrett, “Imagery enhancement andinterpretation for remote visual inspection of aging civil infrastructure,”Tsinghua Science & Technology, vol. 13, pp. 375–380, 2008. → page 22[54] W. Jiang, M. L. Baker, Q. Wu, C. Bajaj, and W. Chiu, “Applications of abilateral denoising filter in biological electron microscopy,” Journal ofstructural biology, vol. 144, no. 1-2, pp. 114–122, 2003. → page 22[55] Y. Amit, 2D object detection and recognition: models, algorithms, andnetworks. MIT Press, 2002. → page 22[56] I. T. Young and L. J. Van Vliet, “Recursive implementation of the gaussianfilter,” Signal processing, vol. 44, no. 2, pp. 139–151, 1995. → page 22[57] I. Pitas, Digital image processing algorithms and applications. JohnWiley & Sons, 2000. → page 22[58] D. Ziou, S. Tabbone et al., “Edge detection techniques-an overview,”Pattern Recognition and Image Analysis C/C of Raspoznavaniye ObrazovI Analiz Izobrazhenii, vol. 8, pp. 537–559, 1998. → page 22[59] O. Moselhi and T. Shehab-Eldeen, “Automated detection of surfacedefects in water and sewer pipes,” Automation in Construction, vol. 8,no. 5, pp. 581–588, 1999. → page 22114[60] J. Kittler, “On the accuracy of the sobel edge detector,” Image and VisionComputing, vol. 1, no. 1, pp. 37–42, 1983. → page 23[61] L. Ding and A. Goshtasby, “On the canny edge detector,” Patternrecognition, vol. 34, no. 3, pp. 721–725, 2001. → page 23[62] R. S. Stephens, “Probabilistic approach to the hough transform,” Imageand vision computing, vol. 9, no. 1, pp. 66–71, 1991. → page 23[63] E. Nadernejad, S. Sharifzadeh, and H. Hassanpour, “Edge detectiontechniques: Evaluations and comparisons,” Applied MathematicalSciences, vol. 2, no. 31, pp. 1507–1520, 2008. → page 23[64] S. Datta and S. Sarkar, “A review on different pipeline fault detectionmethods,” Journal of Loss Prevention in the Process Industries, vol. 41,pp. 97–106, 2016. → page 23[65] M. R. Halfawy and J. Hengmeechai, “Efficient algorithm for crackdetection in sewer images from closed-circuit television inspections,”Journal of Infrastructure Systems, vol. 20, no. 2, p. 04013014, 2014. →page 23[66] Y.-T. Hsiao, C.-L. Chuang, Y.-L. Lu, and J.-A. Jiang, “Robust multipleobjects tracking using image segmentation and trajectory estimationscheme in video frames,” Image and Vision Computing, vol. 24, no. 10,pp. 1123–1136, 2006. → page 23[67] F. Zana and J.-C. Klein, “Segmentation of vessel-like patterns usingmathematical morphology and curvature evaluation,” IEEE transactionson image processing, vol. 10, no. 7, pp. 1010–1019, 2001. → page 23[68] S. Iyer and S. K. Sinha, “Automated condition assessment of buried sewerpipes based on digital imaging techniques,” Journal of the Indian Instituteof Science, vol. 85, no. 5, p. 235, 2005. → page 23[69] S. K. Sinha and P. W. Fieguth, “Segmentation of buried concrete pipeimages,” Automation in Construction, vol. 15, no. 1, pp. 47–57, 2006. →page 23[70] M. D. Yang and T. C. Su, “Segmenting ideal morphologies of sewer pipedefects on cctv images for automated diagnosis,” Expert Systems withApplications, vol. 36, no. 2, pp. 3562–3573, 2009. → page 23115[71] T.-C. Su, M.-D. Yang, T.-C. Wu, and J.-Y. Lin, “Morphologicalsegmentation based on edge detection for sewer pipe defects on cctvimages,” Expert Systems with Applications, vol. 38, no. 10, pp.13 094–13 114, 2011. → page 23[72] T. Chen, Q. Wu, R. Rahmani-Torkaman, and J. Hughes, “A pseudo top-hatmathematical morphological approach to edge detection in dark regions,”Pattern Recognition, vol. 35, no. 1, pp. 199–210, 2002. → page 23[73] L. Shao, Y. Pan, J. Li, H. Liu, X. Chen, and X. Yu, “Steam piping infraredimage segmentation with trend coefficients algorithm,” InternationalJournal of Pressure Vessels and Piping, vol. 170, pp. 49–58, 2019. →page 23[74] J. Mashford, M. Rahilly, P. Davis, and S. Burn, “A morphologicalapproach to pipe image interpretation based on segmentation by supportvector machine,” Automation in Construction, vol. 19, no. 7, pp. 875–883,2010. → page 23[75] Y.-J. Cha, W. Choi, and O. Bu¨yu¨ko¨ztu¨rk, “Deep learning-based crackdamage detection using convolutional neural networks,” Computer-AidedCivil and Infrastructure Engineering, vol. 32, no. 5, pp. 361–378, 2017.→ page 24[76] D. Meijer, L. Scholten, F. Clemens, and A. Knobbe, “A defectclassification methodology for sewer image sets with convolutional neuralnetworks,” Automation in Construction, vol. 104, pp. 281–298, 2019. →page 24[77] O. Duran, K. Althoefer, and L. D. Seneviratne, “Automated pipe defectdetection and categorization using camera/laser-based profiler andartificial neural network,” IEEE Transactions on Automation Science andEngineering, vol. 4, no. 1, pp. 118–126, 2007. → page 24[78] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchiesfor accurate object detection and semantic segmentation,” in Proceedingsof the IEEE conference on computer vision and pattern recognition, 2014,pp. 580–587. → pages 25, 31[79] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE internationalconference on computer vision, 2015, pp. 1440–1448. → pages 25, 32116[80] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-timeobject detection with region proposal networks,” arXiv preprintarXiv:1506.01497, 2015. → pages 25, 32, 34, 40, 46, 92[81] J. C. Cheng and M. Wang, “Automated detection of sewer pipe defects inclosed-circuit television images using deep learning techniques,”Automation in Construction, vol. 95, pp. 155–171, 2018. → pages25, 39, 40, 52[82] S. I. Hassan, L. M. Dang, I. Mehmood, S. Im, C. Choi, J. Kang, Y.-S.Park, and H. Moon, “Underground sewer pipe condition assessment basedon convolutional neural networks,” Automation in Construction, vol. 106,p. 102849, 2019. → pages 26, 40[83] N. Botteghi, L. Grefte, M. Poel, B. Sirmacek, C. Brune, E. Dertien, andS. Stramigioli, “Towards autonomous pipeline inspection with hierarchicalreinforcement learning,” in International Conference on Robot IntelligenceTechnology and Applications. Springer, 2022, pp. 259–271. → page 26[84] Z. Zou, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: Asurvey,” arXiv preprint arXiv:1905.05055, 2019. → page 27[85] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang,A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visualrecognition challenge,” International journal of computer vision, vol. 115,no. 3, pp. 211–252, 2015. → page 27[86] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,P. Dolla´r, and C. L. Zitnick, “Microsoft coco: Common objects incontext,” in European conference on computer vision. Springer, 2014,pp. 740–755. → page 27[87] W. Zhiqiang and L. Jun, “A review of object detection based onconvolutional neural network,” in 2017 36th Chinese Control Conference(CCC). IEEE, 2017, pp. 11 104–11 109. → pages 28, 29[88] Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deeplearning: A review,” IEEE transactions on neural networks and learningsystems, vol. 30, no. 11, pp. 3212–3232, 2019. → pages 28, 29, 30, 33117[89] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,”International journal of computer vision, vol. 60, no. 2, pp. 91–110, 2004.→ page 28[90] T. Ahonen, A. Hadid, and M. Pietika¨inen, “Face recognition with localbinary patterns,” in European conference on computer vision. Springer,2004, pp. 469–481. → page 28[91] N. Dalal and B. Triggs, “Histograms of oriented gradients for humandetection,” in 2005 IEEE computer society conference on computer visionand pattern recognition (CVPR’05), vol. 1. IEEE, 2005, pp. 886–893. →page 28[92] R. Lienhart and J. Maydt, “An extended set of haar-like features for rapidobject detection,” in Proceedings. international conference on imageprocessing, vol. 1. IEEE, 2002, pp. I–I. → page 28[93] R. S. Stankovic´ and B. J. Falkowski, “The haar wavelet transform: itsstatus and achievements,” Computers & Electrical Engineering, vol. 29,no. 1, pp. 25–44, 2003. → page 28[94] T.-C. Su and M.-D. Yang, “Application of morphological segmentation toleaking defect detection in sewer pipelines,” Sensors, vol. 14, no. 5, pp.8686–8704, 2014. → pages 28, 39[95] T. Surasak, I. Takahiro, C.-h. Cheng, C.-e. Wang, and P.-y. Sheng,“Histogram of oriented gradients for human detection in video,” in 20185th International Conference on Business and Industrial Research(ICBIR). IEEE, 2018, pp. 172–176. → page 28[96] D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen, “Local binarypatterns and its application to facial image analysis: a survey,” IEEETransactions on Systems, Man, and Cybernetics, Part C (Applications andReviews), vol. 41, no. 6, pp. 765–781, 2011.[97] J. Barreto, P. Menezes, and J. Dias, “Human-robot interaction based onhaar-like features and eigenfaces,” in IEEE International Conference onRobotics and Automation, 2004. Proceedings. ICRA’04. 2004, vol. 2.IEEE, 2004, pp. 1888–1893.118[98] J. Cruz-Mota, I. Bogdanova, B. Paquier, M. Bierlaire, and J.-P. Thiran,“Scale invariant feature transform on the sphere: Theory andapplications,” International journal of computer vision, vol. 98, no. 2, pp.217–241, 2012.[99] G. Zhao, T. Ahonen, J. Matas, and M. Pietikainen, “Rotation-invariantimage and video description with local binary pattern features,” IEEEtransactions on image processing, vol. 21, no. 4, pp. 1465–1477, 2011. →page 28[100] W. Ma, L. Huang, and C. Liu, “Advanced local binary pattern descriptorsfor crowd estimation,” in 2008 IEEE Pacific-Asia Workshop onComputational Intelligence and Industrial Application, vol. 2. IEEE,2008, pp. 958–962.[101] P. Carcagnı`, M. Del Coco, M. Leo, and C. Distante, “Facial expressionrecognition and histograms of oriented gradients: a comprehensive study,”SpringerPlus, vol. 4, no. 1, pp. 1–25, 2015. → page 28[102] Y. Freund and R. E. Schapire, “A decision-theoretic generalization ofon-line learning and an application to boosting,” Journal of computer andsystem sciences, vol. 55, no. 1, pp. 119–139, 1997. → page 28[103] C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning,vol. 20, no. 3, pp. 273–297, 1995. → page 28[104] L. Breiman, “Random forests,” UC Berkeley TR567, 1999. → page 28[105] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan,“Object detection with discriminatively trained part-based models,” IEEEtransactions on pattern analysis and machine intelligence, vol. 32, no. 9,pp. 1627–1645, 2009. → page 28[106] J. Myrans, Z. Kapelan, and R. Everson, “Automated detection of faults inwastewater pipes from cctv footage by using random forests,” Procediaengineering, vol. 154, pp. 36–41, 2016. → pages 28, 29[107] J. Mashford, P. Davis, and M. Rahilly, “Pixel-based colour imagesegmentation using support vector machine for automatic pipe inspection,”in Australasian Joint Conference on Artificial Intelligence. Springer,2007, pp. 739–743.119[108] J. Mashford, M. Rahilly, P. Davis, and S. Burn, “A morphologicalapproach to pipe image interpretation based on segmentation by supportvector machine,” Automation in Construction, vol. 19, no. 7, pp. 875–883,2010.[109] J. Mashford, M. Rahilly, and D. Marney, “Processing by svm of haarwavelet transforms for discontinuity detection,” in Proceedings of theInternational Conference on Image Processing, Computer Vision, andPattern Recognition (IPCV). The Steering Committee of The WorldCongress in Computer Science, Computer Science, 2011, p. 1.[110] J. Mashford, M. Rahilly, B. Lane, D. Marney, and S. Burn, “Edgedetection in pipe images using classification of haar wavelet transforms,”Applied artificial intelligence, vol. 28, no. 7, pp. 675–689, 2014. → page28[111] J. Myrans, R. Everson, and Z. Kapelan, “Automated detection of faults insewers using cctv image sequences,” Automation in Construction, vol. 95,pp. 64–71, 2018. → page 29[112] J. Myrans, Z. Kapelan, and R. Everson, “Automatic detection of sewerfaults using continuous cctv footage,” in Computing & Control for theWater Industry, 2017.[113] ——, “Combining classifiers to detect faults in wastewater networks,”Water Science and Technology, vol. 77, no. 9, pp. 2184–2189, 2018. →page 29[114] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp. 5–32,2001. → page 29[115] W. Pitts and W. S. McCulloch, “How we know universals the perceptionof auditory and visual forms,” The Bulletin of mathematical biophysics,vol. 9, no. 3, pp. 127–147, 1947. → page 29[116] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learningrepresentations by back-propagating errors,” nature, vol. 323, no. 6088,pp. 533–536, 1986. → page 29120[117] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality ofdata with neural networks,” science, vol. 313, no. 5786, pp. 504–507,2006. → page 29[118] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521,no. 7553, pp. 436–444, 2015. → page 30[119] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: Alarge-scale hierarchical image database,” in 2009 IEEE conference oncomputer vision and pattern recognition. Ieee, 2009, pp. 248–255. →page 30[120] M. Oquab, L. Bottou, I. Laptev, J. Sivic et al., “Weakly supervised objectrecognition with convolutional neural networks,” in Proc. of NIPS.Citeseer, 2014, pp. 1545–5963. → page 30[121] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferringmid-level image representations using convolutional neural networks,” inProceedings of the IEEE conference on computer vision and patternrecognition, 2014, pp. 1717–1724. → page 30[122] K. Kavukcuoglu, P. Sermanet, Y.-L. Boureau, K. Gregor, M. Mathieu,Y. Cun et al., “Learning convolutional feature hierarchies for visualrecognition,” Advances in neural information processing systems, vol. 23,pp. 1090–1098, 2010. → page 30[123] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fastfeature embedding,” in Proceedings of the 22nd ACM internationalconference on Multimedia, 2014, pp. 675–678. → page 30[124] L. Perez and J. Wang, “The effectiveness of data augmentation in imageclassification using deep learning,” arXiv preprint arXiv:1712.04621,2017.[125] S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson,“Deep learning for hyperspectral image classification: An overview,”IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp.6690–6709, 2019. → page 30121[126] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodaldeep learning,” in ICML, 2011. → page 30[127] E. Nishani and B. C¸ic¸o, “Computer vision approaches based on deeplearning and neural networks: Deep neural networks for video analysis ofhuman pose estimation,” in 2017 6th Mediterranean Conference onEmbedded Computing (MECO). IEEE, 2017, pp. 1–4. → page 30[128] A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky, “Neural codesfor image retrieval,” in European conference on computer vision.Springer, 2014, pp. 584–599. → page 30[129] K. Lin, H.-F. Yang, J.-H. Hsiao, and C.-S. Chen, “Deep learning of binaryhash codes for fast image retrieval,” in Proceedings of the IEEEconference on computer vision and pattern recognition workshops, 2015,pp. 27–35. → page 30[130] Z. Yang and R. Nevatia, “A multi-scale cascade fully convolutionalnetwork face detector,” in 2016 23rd International Conference on PatternRecognition (ICPR). IEEE, 2016, pp. 633–638. → page 30[131] Y. Wang, X. Ji, Z. Zhou, H. Wang, and Z. Li, “Detecting faces usingregion-based fully convolutional networks,” arXiv preprintarXiv:1709.05256, 2017. → page 30[132] D. Tome`, F. Monti, L. Baroffio, L. Bondi, M. Tagliasacchi, and S. Tubaro,“Deep convolutional neural networks for pedestrian detection,” Signalprocessing: image communication, vol. 47, pp. 482–489, 2016. → page 30[133] J. Liu, S. Zhang, S. Wang, and D. N. Metaxas, “Multispectral deep neuralnetworks for pedestrian detection,” arXiv preprint arXiv:1611.02644,2016. → page 30[134] J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders,“Selective search for object recognition,” International journal ofcomputer vision, vol. 104, no. 2, pp. 154–171, 2013. → page 31[135] I. Endres and D. Hoiem, “Category independent object proposals,” inEuropean Conference on Computer Vision. Springer, 2010, pp. 575–588.→ page 31122[136] J. Carreira and C. Sminchisescu, “Cpmc: Automatic object segmentationusing constrained parametric min-cuts,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 34, no. 7, pp. 1312–1328, 2011.→ page 31[137] C. L. Zitnick and P. Dolla´r, “Edge boxes: Locating object proposals fromedges,” in European conference on computer vision. Springer, 2014, pp.391–405. → page 31[138] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classificationwith deep convolutional neural networks,” Advances in neural informationprocessing systems, vol. 25, pp. 1097–1105, 2012. → page 31[139] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deepconvolutional networks for visual recognition,” IEEE transactions onpattern analysis and machine intelligence, vol. 37, no. 9, pp. 1904–1916,2015. → pages 31, 32[140] S. S. Kumar, M. Wang, D. M. Abraham, M. R. Jahanshahi, T. Iseley, andJ. C. Cheng, “Deep learning–based automated detection of sewer defectsin cctv videos,” Journal of Computing in Civil Engineering, vol. 34, no. 1,p. 04019047, 2020. → pages 32, 33, 34, 38, 40[141] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once:Unified, real-time object detection,” in Proceedings of the IEEEconference on computer vision and pattern recognition, 2016, pp.779–788. → pages 33, 34, 90[142] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.Berg, “Ssd: Single shot multibox detector,” in European conference oncomputer vision. Springer, 2016, pp. 21–37. → pages 33, 34, 40, 99[143] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” inProceedings of the IEEE conference on computer vision and patternrecognition, 2017, pp. 7263–7271. → pages 33, 90, 99[144] ——, “Yolov3: An incremental improvement,” arXiv preprintarXiv:1804.02767, 2018. → pages 33, 40, 99[145] D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov, “Scalable objectdetection using deep neural networks,” in Proceedings of the IEEE123conference on computer vision and pattern recognition, 2014, pp.2147–2154. → page 34[146] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick, “Inside-outside net:Detecting objects in context with skip pooling and recurrent neuralnetworks,” in Proceedings of the IEEE conference on computer vision andpattern recognition, 2016, pp. 2874–2883. → page 34[147] S. S. Kumar, D. M. Abraham, M. R. Jahanshahi, T. Iseley, and J. Starr,“Automated defect classification in sewer closed circuit televisioninspections using deep convolutional neural networks,” Automation inConstruction, vol. 91, pp. 273–283, 2018. → page 35[148] A.-W. Taha, S. Sharma, R. Lupoja, A.-N. Fadhl, M. Haidera, andM. Kennedy, “Assessment of water losses in distribution networks:Methods, applications, uncertainties, and implications in intermittentsupply,” Resources, Conservation and Recycling, vol. 152, p. 104515,2020. → page 38[149] M. E. Fontana and D. C. Morais, “Decision model to control water lossesin distribution networks,” Production, vol. 26, no. 4, pp. 688–697, 2016.→ page 38[150] W. Maize, “Unacceptable loss,” Water Finance & Management, Feb. 2018,Available:https://waterfm.com/unacceptable-loss, Accessed on: July 12,2019. → page 38[151] Z. Liu and Y. Kleiner, “State of the art review of inspection technologiesfor condition assessment of water pipes,” Measurement, vol. 46, no. 1, pp.1–15, 2013. → page 38[152] R. R. Harvey and E. A. McBean, “Predicting the structural condition ofindividual sanitary sewer pipes with random forests,” Canadian Journal ofCivil Engineering, vol. 41, no. 4, pp. 294–303, 2014. → page 38[153] M. R. Halfawy and J. Hengmeechai, “Automated defect detection in sewerclosed circuit television images using histograms of oriented gradients andsupport vector machine,” Automation in Construction, vol. 38, pp. 1–13,2014. → pages 38, 39, 78124[154] H. Tran, B. Perera, and A. Ng, “Predicting structural deteriorationcondition of individual storm-water pipes using probabilistic neuralnetworks and multiple logistic regression models,” Journal of WaterResources Planning and Management, vol. 135, no. 6, pp. 553–557, 2009.→ page 38[155] M. R. Halfawy, L. Dridi, and S. Baker, “Integrated decision support systemfor optimal renewal planning of sewer networks,” Journal of Computing inCivil Engineering, vol. 22, no. 6, pp. 360–372, 2008. → page 38[156] K. Xu, A. Luxmoore, and T. Davies, “Sewer pipe deformation assessmentby image analysis of video surveys,” Pattern Recognition, vol. 31, no. 2,pp. 169–180, 1998. → page 39[157] O. Moselhi and T. Shehab-Eldeen, “Classification of defects in sewerpipes using neural networks,” Journal of infrastructure systems, vol. 6,no. 3, pp. 97–104, 2000. → page 39[158] W. Guo, L. Soibelman, and J. Garrett Jr, “Visual pattern recognitionsupporting defect reporting and condition assessment of wastewatercollection systems,” Journal of computing in civil engineering, vol. 23,no. 3, pp. 160–169, 2009. → page 39[159] S. Moradi and T. Zayed, “Real-time defect detection in sewer closedcircuit television inspection videos,” in Pipelines 2017, 2017, pp.295–307. → page 39[160] Q. Xie, D. Li, J. Xu, Z. Yu, and J. Wang, “Automatic detection andclassification of sewer defects via hierarchical deep learning,” IEEETransactions on Automation Science and Engineering, vol. 16, no. 4, pp.1836–1847, 2019. → pages 40, 56[161] X. Yin, Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, andL. Kurach, “A deep learning-based framework for an automated defectdetection system for sewer pipes,” Automation in Construction, vol. 109,p. 102967, 2020. → page 40[162] A. F. Colombo and B. W. Karney, “Energy and costs of leaky pipes:Toward comprehensive picture,” Journal of Water Resources Planning andManagement, vol. 128, no. 6, pp. 441–450, 2002. → page 41125[163] K. Xin, F. Li, T. Tao, N. Xiang, and Z. Yin, “Water losses investigationand evaluation in water distribution system–the case of sa city in china,”Urban Water Journal, vol. 12, no. 5, pp. 430–439, 2015. → page 41[164] Q. Zhou, Z. Situ, S. Teng, and G. Chen, “Convolutional neuralnetworks–based model for automated sewer defects detection andclassification,” Journal of Water Resources Planning and Management,vol. 147, no. 7, p. 04021036, 2021. → page 42[165] A. Dutta, A. Gupta, and A. Zissermann, “Vgg image annotator (via),”URL: http://www. robots. ox. ac. uk/vgg/software/via, vol. 2, 2016. →pages 43, 52[166] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al.,“An image is worth 16x16 words: Transformers for image recognition atscale,” arXiv preprint arXiv:2010.11929, 2020. → page 44[167] P. Gao, J. Lu, H. Li, R. Mottaghi, and A. Kembhavi, “Container: Contextaggregation network,” arXiv preprint arXiv:2106.01401, 2021. → page 44[168] Q. Zhang and Y.-B. Yang, “Rest: An efficient transformer for visualrecognition,” Advances in Neural Information Processing Systems, vol. 34,pp. 15 475–15 485, 2021. → page 44[169] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attentionfor rapid scene analysis,” IEEE Transactions on pattern analysis andmachine intelligence, vol. 20, no. 11, pp. 1254–1259, 1998. → page 44[170] R. A. Rensink, “The dynamic representation of scenes,” Visual cognition,vol. 7, no. 1-3, pp. 17–42, 2000.[171] M. Corbetta and G. L. Shulman, “Control of goal-directed andstimulus-driven attention in the brain,” Nature reviews neuroscience,vol. 3, no. 3, pp. 201–215, 2002. → page 44[172] K. He, G. Gkioxari, P. Dolla´r, and R. Girshick, “Mask r-cnn,” inProceedings of the IEEE international conference on computer vision,2017, pp. 2961–2969. → page 46126[173] A. Sanakoyeu, “You don’t need efficientnets. simple tricks make resnetsbetter and faster than efficientnets,” Gradient Dude, 2021,https://gdude.de. → page 47[174] J. Canny, “A computational approach to edge detection,” IEEETransactions on pattern analysis and machine intelligence, no. 6, pp.679–698, 1986. → page 47[175] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778. → page 48[176] H. A. Qadir, Y. Shin, J. Solhusvik, J. Bergsland, L. Aabakken, andI. Balasingham, “Polyp detection and segmentation using mask r-cnn:Does a deeper feature extractor cnn always perform better?” in 2019 13thInternational Symposium on Medical Information and CommunicationTechnology (ISMICT). IEEE, 2019, pp. 1–6. → pages 48, 49[177] “Nassco (national association of sewer service companies),” NASSCODallas, 2001. → page 52[178] X. Yin, Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, andL. Kurach, “A deep learning-based framework for an automated defectdetection system for sewer pipes,” Automation in Construction, vol. 109,p. 102967, 2020. → page 52[179] D. Tabernik, S. Sˇela, J. Skvarcˇ, and D. Skocˇaj, “Segmentation-baseddeep-learning approach for surface-defect detection,” Journal ofIntelligent Manufacturing, vol. 31, no. 3, pp. 759–776, 2020. → page 57[180] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networksfor biomedical image segmentation,” in International Conference onMedical image computing and computer-assisted intervention. Springer,2015, pp. 234–241. → pages 57, 58[181] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks forsemantic segmentation,” in Proceedings of the IEEE conference oncomputer vision and pattern recognition, 2015, pp. 3431–3440. → page59127[182] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” inProceedings of the IEEE conference on computer vision and patternrecognition, 2018, pp. 7132–7141. → pages 61, 63[183] R. Polikar, “Ensemble learning,” in Ensemble machine learning.Springer, 2012, pp. 1–34. → page 64[184] M. Abdel-Aleem, C. C. Chibelushi, and M. Moniri, “Multisensor datafusion for the simultaneous location and condition assessment ofunderground water pipelines,” in 2011 International Conference onNetworking, Sensing and Control. IEEE, 2011, pp. 416–421. → page 66[185] A. Campisano, E. Creaco, and C. Modica, “Rtc of valves for leakagereduction in water supply networks,” Journal of Water Resources Planningand Management, vol. 136, no. 1, pp. 138–141, 2010. → page 66[186] Z. Liu and Y. Kleiner, “Computational intelligence for urban infrastructurecondition assessment: Water transmission and distribution systems,” IEEESensors Journal, vol. 14, no. 12, pp. 4122–4133, 2014. → page 67[187] M. Stephens, M. Lambert, A. Simpson, J. Nixon, and J. Vitkovsky, “Waterpipeline condition assessment using transient response analysis,” 2005.[188] S. K. Sinha, P. W. Fieguth, and M. A. Polak, “Computer vision techniquesfor automatic structural assessment of underground pipes,”Computer-Aided Civil and Infrastructure Engineering, vol. 18, no. 2, pp.95–112, 2003. → page 67[189] O. Duran, K. Althoefer, and L. D. Seneviratne, “Automated pipe defectdetection and categorization using camera/laser-based profiler andartificial neural network,” IEEE Transactions on Automation Science andEngineering, vol. 4, no. 1, pp. 118–126, 2007. → page 67[190] M. R. Halfawy and J. Hengmeechai, “Efficient algorithm for crackdetection in sewer images from closed-circuit television inspections,”Journal of Infrastructure Systems, vol. 20, no. 2, p. 04013014, 2014.[191] ——, “Integrated vision-based system for automated defect detection insewer closed circuit television inspection videos,” Journal of Computingin Civil Engineering, vol. 29, no. 1, p. 04014024, 2015. → page 67128[192] C. Koch, K. Georgieva, V. Kasireddy, B. Akinci, and P. Fieguth, “A reviewon computer vision based defect detection and condition assessment ofconcrete and asphalt civil infrastructure,” Advanced EngineeringInformatics, vol. 29, no. 2, pp. 196–210, 2015. → page 67[193] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, andM. Pietika¨inen, “Deep learning for generic object detection: A survey,”International journal of computer vision, vol. 128, no. 2, pp. 261–318,2020. → page 67[194] V. Kastrinaki, M. Zervakis, and K. Kalaitzakis, “A survey of videoprocessing techniques for traffic applications,” Image and visioncomputing, vol. 21, no. 4, pp. 359–381, 2003. → page 68[195] N. K. Logothetis and D. L. Sheinberg, “Visual object recognition,” Annualreview of neuroscience, vol. 19, no. 1, pp. 577–621, 1996. → page 68[196] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-timeobject detection with region proposal networks,” in Advances in neuralinformation processing systems, 2015, pp. 91–99. → page 68[197] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchiesfor accurate object detection and semantic segmentation,” in Proceedingsof the IEEE conference on computer vision and pattern recognition, 2014,pp. 580–587. → page 68[198] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778. → page 68[199] D. Kang and Y.-J. Cha, “Autonomous uavs for structural health monitoringusing deep learning and an ultrasonic beacon system with geo-tagging,”Computer-Aided Civil and Infrastructure Engineering, vol. 33, no. 10, pp.885–902, 2018. → page 68[200] Q. Xie, D. Li, J. Xu, Z. Yu, and J. Wang, “Automatic detection andclassification of sewer defects via hierarchical deep learning,” IEEETransactions on Automation Science and Engineering, vol. 16, no. 4, pp.1836–1847, 2019. → page 68129[201] I. Goodfellow, “Nips 2016 tutorial: Generative adversarial networks,”arXiv preprint arXiv:1701.00160, 2016. → pages 68, 69, 70[202] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand,M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neuralnetworks for mobile vision applications,” arXiv preprintarXiv:1704.04861, 2017. → pages 68, 72[203] T.-Y. Lin, P. Dolla´r, R. Girshick, K. He, B. Hariharan, and S. Belongie,“Feature pyramid networks for object detection,” in Proceedings of theIEEE conference on computer vision and pattern recognition, 2017, pp.2117–2125. → pages 68, 73[204] A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training forhigh fidelity natural image synthesis,” arXiv preprint arXiv:1809.11096,2018. → pages 69, 70, 71[205] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXivpreprint arXiv:1312.6114, 2013. → page 70[206] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou, “Revisiting batch normalizationfor practical domain adaptation,” arXiv preprint arXiv:1603.04779, 2016.→ page 70[207] A. v. d. Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrentneural networks,” arXiv preprint arXiv:1601.06759, 2016. → page 70[208] A. Noguchi and T. Harada, “Image generation from small datasets viabatch statistics adaptation,” in Proceedings of the IEEE InternationalConference on Computer Vision, 2019, pp. 2750–2758. → pages 70, 71[209] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dolla´r, “Focal loss fordense object detection,” in Proceedings of the IEEE internationalconference on computer vision, 2017, pp. 2980–2988. → pages 71, 73, 74[210] M.-D. Yang and T.-C. Su, “Automated diagnosis of sewer pipe defectsbased on machine learning approaches,” Expert Systems with Applications,vol. 35, no. 3, pp. 1327–1337, 2008. → page 78[211] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014. →page 78130[212] “Estimates of memory consumption and flop counts for variousconvolutional neural networks,” Github,https://github.com/albanie/convnet-burden [Online], Accessed on 03April, 2020. → page 78[213] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778. → page 78[214] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Denselyconnected convolutional networks,” in Proceedings of the IEEEconference on computer vision and pattern recognition, 2017, pp.4700–4708. → page 78[215] L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling, “Visualizing deepneural network decisions: Prediction difference analysis,” arXiv preprintarXiv:1702.04595, 2017. → page 80[216] M. J. Chae and D. M. Abraham, “Neuro-fuzzy approaches for sanitarysewer pipeline condition assessment,” Journal of Computing in Civilengineering, vol. 15, no. 1, pp. 4–14, 2001. → page 86[217] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speedand accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.→ pages 89, 90, 92[218] Z. Zhang, Q. Liang, Z. Feng, W. Ji, H. Wang, and J. Hu, “Application ofimproved yolov4 in intelligent driving scenarios,” in 2021 IEEEInternational Conference on Progress in Informatics and Computing(PIC). IEEE, 2021, pp. 144–148. → page 89[219] D. Li, X. Sun, H. Elkhouchlaa, Y. Jia, Z. Yao, P. Lin, J. Li, and H. Lu,“Fast detection and location of longan fruits using uav images,” Computersand Electronics in Agriculture, vol. 190, p. 106465, 2021. → page 89[220] A. Farhadi and J. Redmon, “Yolov3: An incremental improvement,” inComputer Vision and Pattern Recognition, 2018, pp. 1804–02 767. →page 90131[221] A. M. Roy, R. Bose, and J. Bhaduri, “A fast accurate fine-grain objectdetection model based on yolov4 deep neural network,” NeuralComputing and Applications, pp. 1–27, 2022. → pages 90, 92[222] G. Ghiasi, T.-Y. Lin, and Q. V. Le, “Dropblock: A regularization methodfor convolutional networks,” Advances in neural information processingsystems, vol. 31, 2018. → page 92[223] A. Mhalla, S. Gazzah, N. E. Ben Amara et al., “A faster r-cnn multi-objectdetector on a nvidia jetson tx1 embedded system,” in Proceedings of the10th International Conference on Distributed Smart Camera. ACM,2016, pp. 208–209. → page 93[224] Y. Han and E. Oruklu, “Traffic sign recognition based on the nvidia jetsontx1 embedded system using convolutional neural networks,” in 2017 IEEE60th International Midwest Symposium on Circuits and Systems(MWSCAS). IEEE, 2017, pp. 184–187.[225] H. Mao, S. Yao, T. Tang, B. Li, J. Yao, and Y. Wang, “Towards real-timeobject detection on embedded systems,” IEEE Transactions on EmergingTopics in Computing, vol. 6, no. 3, pp. 417–431, 2016.[226] R. Rayhana, G. Xiao, and Z. Liu, “Rfid sensing technologies for smartagriculture,” IEEE Instrumentation & Measurement Magazine, vol. 24,no. 3, pp. 50–60, 2021.[227] ——, “Internet of things empowered smart greenhouse farming,” IEEEJournal of Radio Frequency Identification, vol. 4, no. 3, pp. 195–211,2020.[228] ——, “Printed sensor technologies for monitoring applications in smartfarming: A review,” IEEE Transactions on Instrumentation andMeasurement, 2021. → page 93[229] G. Li, P. Ren, X. Lyu, and H. Zhang, “Real-time top-view people countingbased on a kinect and nvidia jetson tk1 integrated platform,” in 2016 IEEE16th international conference on data mining workshops (ICDMW).IEEE, 2016, pp. 468–473. → page 95[230] C. Goutte and E. Gaussier, “A probabilistic interpretation of precision,recall and f-score, with implication for evaluation,” in European132conference on information retrieval. Springer, 2005, pp. 345–359. →pages 97, 98[231] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese,“Generalized intersection over union: A metric and a loss for boundingbox regression,” in Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, 2019, pp. 658–666. → page 98[232] S.-g. Roh and H. R. Choi, “Differential-drive in-pipe robot for movinginside urban gas pipelines,” IEEE transactions on robotics, vol. 21, no. 1,pp. 1–17, 2005. → page 101[233] G. A. De Vitis, P. Foglia, and C. A. Prete, “A pre-processing technique todecrease inspection time in glass tube production lines,” IET ImageProcessing, vol. 15, no. 10, p. 2179–2191, 2021. → page 103[234] T. Hao, C. Rogers, N. Metje, D. Chapman, J. Muggleton, K. Foo, P. Wang,S. R. Pennock, P. Atkins, S. Swingler et al., “Condition assessment of theburied utility service infrastructure,” Tunnelling and Underground SpaceTechnology, vol. 28, pp. 331–344, 2012. → page 104133
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Liu, Zheng (Professor of engineering)
Video data analytics for the automation of water pipeline inspection
Text