Applied Science, Faculty of
Civil Engineering, Department of
DSpace
UBCV
Seng, Vannary
Master of Applied Science - MASc
University of British Columbia
Closed-circuit television (CCTV) is widely employed for assessing defects in sewers. This thesis: i) develops a number of deep convolutional neural network (CNN)-based image classification models for identifying sewer defects in CCTV videos, ii) develops a number of Machine Learning (ML)-based models for predicting the likely occurrence of specific defects in sewer networks, as well as identifying the factors that contribute to these defects, and iii) analyses the model performances with data from multiple utilities. The image classification models are trained and tested using images extracted from CCTV inspection videos for three utilities. They are analysed for serving two purposes: conducting a quality assurance and quality control (QA/QC) assessment of manual inspection reports of previously inspected videos, and identifying defects in images from videos that have not yet been manually inspected. Models are developed based on three pre-trained CNN backbone architectures: MobileNet V3 Large, ResNet50, and ResNet101; two model configurations: one- and two-stage models; and grouped classes. Two-stage models result in higher performance compared with alternative approaches. For QA/QC, on average, the ResNet models outperform MobileNet V3 Large where model accuracies for Utilities A, B, and C, are 0.95, 0.70, and 0.98, respectively. For recognizing defects in videos not previously viewed, the optimal model for each of the three utilities does not perform well. However, all models possess the ability to distinguish between non-defective and defective images with accuracies higher than 0.50, where the Utility A model achieves an accuracy of 0.80. Multiple decision tree-based ML models that predict the likely occurrence of infiltration and structural defects are developed using data from CCTV inspections coupled with additional pipe information and Geographical Information System files for two utilities. The performance of the models is assessed using the area under the Receiver Operating Characteristics (AUC-ROC) and Precision Recall (AUC-PR) curves. LightGBM-based models with cost-sensitive learning show the best performance overall when predicting infiltration and structural defects. The best performing model achieves an AUC-ROC of 0.79 and an AUC-PR of 0.62. For these utilities, an application of SHapley Additive exPlanations (SHAP) shows that the most important factors are “pipe location” and “pipe age”.
ENHANCING SEWER ASSET MANAGEMENT USING MACHINE LEARNING ALGORITHMS by Vannary Seng B.Eng., McGill University, 2019 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Civil Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) April 2024 © Vannary Seng, 2024 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled: Enhancing Sewer Asset Management Using Machine Learning Algorithms submitted by Vannary Seng in partial fulfilment of the requirements for the degree of Master of Applied Science in Civil Engineering Examining Committee: Dr. Barbara J. Lence, Professor, Department of Civil Engineering, UBC Supervisor Dr. Steven Weijs, Associate Professor, Department of Civil Engineering, UBC Supervisory Committee Member iii Abstract Closed-circuit television (CCTV) is widely employed for assessing defects in sewers. This thesis: i) develops a number of deep convolutional neural network (CNN)-based image classification models for identifying sewer defects in CCTV videos, ii) develops a number of Machine Learning (ML)-based models for predicting the likely occurrence of specific defects in sewer networks, as well as identifying the factors that contribute to these defects, and iii) analyses the model performances with data from multiple utilities. The image classification models are trained and tested using images extracted from CCTV inspection videos for three utilities. They are analysed for serving two purposes: conducting a quality assurance and quality control (QA/QC) assessment of manual inspection reports of previously inspected videos, and identifying defects in images from videos that have not yet been manually inspected. Models are developed based on three pre-trained CNN backbone architectures: MobileNet V3 Large, ResNet50, and ResNet101; two model configurations: one- and two-stage models; and grouped classes. Two-stage models result in higher performance compared with alternative approaches. For QA/QC, on average, the ResNet models outperform MobileNet V3 Large where model accuracies for Utilities A, B, and C, are 0.95, 0.70, and 0.98, respectively. For recognizing defects in videos not previously viewed, the optimal model for each of the three utilities does not perform well. However, all models possess the ability to distinguish between non-defective and defective images with accuracies higher than 0.50, where the Utility A model achieves an accuracy of 0.80. Multiple decision tree-based ML models that predict the likely occurrence of infiltration and structural defects are developed using data from CCTV inspections coupled with additional pipe information and Geographical Information System files for two utilities. The performance of the models is assessed using the area under the Receiver Operating iv Characteristics (AUC-ROC) and Precision Recall (AUC-PR) curves. LightGBM-based models with cost-sensitive learning show the best performance overall when predicting infiltration and structural defects. The best performing model achieves an AUC-ROC of 0.79 and an AUC-PR of 0.62. For these utilities, an application of SHapley Additive exPlanations (SHAP) shows that the most important factors are “pipe location” and “pipe age”. v Lay Summary Sewer infrastructures are aging and require a regular inspection. Closed-circuit television (CCTV) inspection videos are the standard procedure for examining the interior of the pipe. This process can be time consuming and subjective. Researches have applied machine learning (ML) models to automate and assist with the process. The majority of these models are evaluated on a single data set and are developed to detect limited types of sewer defects. Firstly, this research proposes a framework to examine the application of ML models for their application to multiple data sets and for their ability to identify more types of defects. Secondly, this research develops ML models to predict the likelihood of specific defects being present in a pipe using historical data of inspected pipes. These models can be utilized to assess the condition of uninspected pipes, which is beneficial in cases where budgets and pipe accessibility are limited. vi Preface This research was conducted by Vannary Seng, under the supervision of Dr. Barbara J. Lence, Professor of Civil Engineering at UBC, and Dr. Sudhir Kshirsagar, President of Global Quality Corp. (GQC), based in Covington, Kentucky, USA. The general topic of the research was identified during a Mitacs Accelerate Industrial Internship with Dr. Kshirsagar and in collaboration with Dr. Lence. Through Dr. Lence and Dr. Kshirsagar, we were fortunate to secure data from three utilities based in Western Canada. Dr. Lence and I collaborated with GQC throughout the Mitacs Accelerate Program to process data, develop models to assist the CCTV inspection process and predict pipes with high likelihood of experiencing structural defects and infiltration. The expertise and software tools of GQC were valuable throughout the overall extent of this research, particularly during the development process. Deven Mistry and Pavan Saranguhewa helped setting up the training of the models on multi-GPUs, resulting in a significant reduction in training time. I conducted the analysis of the model output with the assistance of Dr. Lence. Figure 1 in Chapter 1 was created by (Kley et al. 2013), and I got the permission from Nicolas Caradot to include it in this work. All other figures and tables were created by me. Chapters 3 and 4 of the thesis are written to be stand alone and will be submitted as journal publications and will also be presented at two conferences. A version of Chapter 3 will be presented as Seng, V., Lence, B.J., Kshirsagar, S., Mistry, D., and Saranguhewa, P. (2024, June 12). Deep Learning-Based Automated Detection of Sewer Defects in CCTV Videos: Comparison of Models for Three Characteristic Datasets. Paper will be presented at ACE24, Anaheim, CA. vii A version of Chapter 4 will be presented as Seng, V., Lence, B.J., Kshirsagar, S., Rangapuram, S., and Saranguhewa, P. (2024, May 20). A Novel Sewer Condition Assessment Approach that Leverages Advanced Machine Learning (ML) Models. Paper will be presented at World Environmental & Water Resources Congress 2024, Milwaukee, WI. viii Table of Contents Abstract ......................................................................................................................................... iii Lay Summary ................................................................................................................................ v Preface ........................................................................................................................................... vi Table of Contents ....................................................................................................................... viii List of Tables ................................................................................................................................. x List of Figures .............................................................................................................................. xii List of Abbreviations ................................................................................................................. xiv Acknowledgements .................................................................................................................. xviii Dedication .................................................................................................................................... xx Chapter 1: Introduction ............................................................................................................... 1 1.1 Sewer Systems ................................................................................................................ 3 1.2 Sewer Pipe Failure .......................................................................................................... 3 1.3 Sewer Inspection Technologies and Standards ............................................................... 4 1.3.1 CCTV inspection .................................................................................................... 5 1.3.2 Coding standards ..................................................................................................... 6 1.4 Asset Management and Rehabilitation ........................................................................... 9 Chapter 2: Literature Reviews .................................................................................................. 11 2.1 Classification Models.................................................................................................... 11 2.2 Automating the CCTV Inspection Process ................................................................... 12 2.2.1 Feature Extraction Approaches ............................................................................. 12 2.2.2 Deep Learning (DL)-Based Models ...................................................................... 13 2.2.3 Transfer Learning.................................................................................................. 14 2.3 Sewer Deterioration Models ......................................................................................... 15 2.3.1 Three Types of Sewer Deterioration Model ......................................................... 15 2.3.2 Techniques to Manage Class Imbalance ............................................................... 17 Chapter 3: Deep Learning-Based Sewer Defects Detection Model: Comparison of Models for Multiple Utilities ................................................................................................................... 19 3.1 Introduction ................................................................................................................... 19 3.2 Methodology ................................................................................................................. 22 3.2.1 Data Preparation.................................................................................................... 25 3.2.2 Training and Testing ............................................................................................. 27 ix 3.2.2.1 Deep Convolutional Neural Networks (CNNs) and Transfer Learning .......... 29 3.2.2.2 One-Stage and Two-Stage classification approaches ...................................... 31 3.2.3 Evaluation Metrics ................................................................................................ 32 3.3 Results and Discussion ................................................................................................. 33 3.3.1 Evaluation of One- and Two-Stage Classification Models................................... 34 3.3.2 Evaluation of Pre-trained Models ......................................................................... 36 3.3.3 Sensitivity Analyses .............................................................................................. 40 3.4 Conclusions ................................................................................................................... 44 Chapter 4: A Novel Sewer Condition Assessment Approach that Leverages Advanced Machine Learning (ML) Models ............................................................................................... 47 4.1 Introduction ................................................................................................................... 47 4.2 Methodology ................................................................................................................. 50 4.2.1 Model Predictor Variables .................................................................................... 50 4.2.2 Model Target Variables ........................................................................................ 53 4.2.3 Model Development.............................................................................................. 53 4.2.3.1 Ensemble Learning Decision Tree Algorithms ............................................... 54 4.2.3.2 Methods for Addressing Imbalanced Data ...................................................... 57 4.2.3.3 Model Training and Tuning ............................................................................. 58 4.2.4 Model Evaluation Metrics..................................................................................... 59 4.2.5 Shapley Additive exPlanations (SHAP) Analysis and Partial Dependence Plots 61 4.3 Results and Discussion ................................................................................................. 62 4.3.1 Classification Model Comparison......................................................................... 62 4.3.2 Feature Importance and Partial Dependence Plot of the Best Models .................. 66 4.4 Conclusions and Recommendations ............................................................................. 74 Chapter 5: Conclusions .............................................................................................................. 76 References .................................................................................................................................... 77 Appendices ................................................................................................................................... 83 Appendix A ............................................................................................................................... 83 A.1 Computational Time ................................................................................................... 83 Appendix B ............................................................................................................................... 84 B.1 Hyperparameter Tunning ............................................................................................ 84 B.2 Evaluate the performance of different models using Mann-Whitney U tests ............ 84 B.3 Results from all models .............................................................................................. 89 x List of Tables Table 1. Defect score ranges for pipe condition grades (Rahman and Vanier 2004) .................... 7 Table 2. CCTV inspection videos and reports information ......................................................... 23 Table 3. Sewer defect grouped classes ......................................................................................... 24 Table 4. Pre-trained CNN models information ............................................................................ 30 Table 5. Macro-averaged F1&F2 scores of the defective grouped classes for two-stage ResNet50 model pre-trained with data for a given utility (columns) and further fine-tuned with sewer images from other utilities (rows) ................................................................................................. 40 Table 6. CCTV and Sample Pipe Information ............................................................................. 51 Table 7. Model predictor variables .............................................................................................. 51 Table 8. Number of training and testing data ............................................................................... 54 Table 9. List of hyperparameters.................................................................................................. 59 Table 10. Models with the Highest Mean AUC-ROC and AUC-PR for each algorithm ............ 64 Table 11. Average training and prediction time of different models ........................................... 83 Table 12. Range of hyperparameter values used during cross-validation grid search ................. 84 Table 13. The area under the average ROC curves of each classification model and its U statistics, p-value, and rank when comparing its performance with model with other imbalanced data managing techniques, as determined by the Mann-Whitney U test at significance level equals to 0.05 ................................................................................................................................ 85 Table 14. The area under the average PR curves of each classification model and its U statistics, p-value, and rank when comparing its performance with model with other imbalanced data managing techniques, as determined by the Mann-Whitney U test at significance level equals to 0.05................................................................................................................................................ 87 xi Table 15. Rank ordered model predictor variables from first to last place of each classification model............................................................................................................................................. 89 xii List of Figures Figure 1. Overview of main sewer inspection technologies (Kley et al. 2013) ............................. 5 Figure 2. The setup of CCTV inspection ....................................................................................... 6 Figure 3. Overall development process........................................................................................ 25 Figure 4. Sample images from CCTV inspection videos with distance and: a) no defect annotation, b) showing perspective view of defect and defect annotation, c) showing zoom-in of defect and defect annotation, and d) showing multiple defect annotations at the same distance . 26 Figure 5. Grouped classes distribution of each utility data for a) FB and b) VB models ............ 29 Figure 6. Macro-averaged metric scores of a) defective grouped classes and b) the ND class ... 35 Figure 7. Percent difference in TP and FN images of two-stage relative to one-stage models for a) FB ResNet50 models and b) VB ResNet50 models ................................................................. 36 Figure 8. Portion of true positive, misclassified, and unclassified images of each grouped class for a) FB two-stage ResNet50 and b) VB two-stage ResNet50 for each utility ........................... 39 Figure 9. Metric values of the grouped classes and individual classes a) FB two-stage ResNet50 and b) VB two-stage ResNet50 models ........................................................................................ 41 Figure 10. Model performance relative to two-stage ResNet101 Utility A model for different hyperparameter values and enhancement methods for a) FB model and b) VB model ............... 43 Figure 11. a) ROC and b) PR curves for models with the highest means for classifying structural defects for Utility B ...................................................................................................................... 64 Figure 12. a) ROC and b) PR curves of the four classification models using LightGBM with the CSL approach................................................................................................................................ 66 Figure 13. Model predictor variable importance for a) infiltration and b) structural defects classification models for Utility A ................................................................................................ 68 xiii Figure 14. Model predictor variable importance for a) infiltration and b) structural defects classification models for Utility B ................................................................................................ 68 Figure 15. SHAP dependence plots for Utility A infiltration classification model for: a) pipe age; b) material; c) size; and d) length ................................................................................................. 70 Figure 16. SHAP dependence plots for Utility A structural defects classification model for: a) pipe age; b) material; c) size; and d) length .................................................................................. 70 Figure 17. SHAP dependence plots for Utility B infiltration classification model for: a) pipe age; b) material; c) size; and d) length ................................................................................................. 72 Figure 18. SHAP dependence plots for Utility B structural defects classification model for: a) pipe age; b) material; c) size; and d) length .................................................................................. 72 Figure 19. Partial dependence plot between x- and y- coordinates, likelihood of a) infiltration and b) structural defects in Utility A where lighter colors represent a larger likelihood while darker colors suggest a lower likelihood ...................................................................................... 73 Figure 20. Partial dependence plot for x- and y- coordinates, likelihood of a) infiltration and b) structural defects for Utility B where lighter colors represent a larger likelihood while darker colors suggest a lower likelihood .................................................................................................. 74 Figure 21. a) ROC and b) PR curves for models with the highest means for classifying infiltration for Utility A ................................................................................................................ 91 Figure 22. a) ROC and b) PR curves for models with the highest means for classifying structural defects for Utility A ...................................................................................................................... 92 Figure 23. a) ROC and b) PR curves for models with the highest means for classifying infiltration for Utility B ................................................................................................................. 93 xiv List of Abbreviations A Access Points AC Asbestos Cement ANN Artificial Neural Network AP Arched Pipe ASCE American Society of Civil Engineers AUC Area Under the Curve AUC-PR Area Under the Precision and Recall Curve AUC-ROC Area Under the Receiver Operating Characteristics Curve BH Breaks or Holes BW Defective Brick Pipes C Circular Pipe CatBoost Category Boosting CCTV Closed-Circuit Television CF Cracks or Fractures CNN Deep Convolutional Neural Network COF Consequence of Failure CP Concrete Pipe CSL Cost-Sensitive Learning CSOs Combined Sewer Overflows CV Computer Vision D Attached, Settled, or Ingress Deposits DF Deformed Flexible Pipes DL Deep Learning DT Deformed Brick Pipes E Egg-Shaped Pipe EFB Exclusive Feature Bundling xv FB Frame-Based FN False Negative FP False Positive FPR False Positive Rate GIS Geographic Information System GIST Global Image Structure Tensor GOSS Gradient-based One-Side Sampling GQC Global Quality Corp. H Horseshoe Pipe HOG Histograms of Oriented Gradients I Infiltration I&I Inflow and Infiltration ID Defective Taps ISS Intruding Sealing Materials J Offset, separated, or angular joints L Line of Sewer Change LF Defective Lining Features LightGBM Light Gradient Boosting Machine LOF Likelihood of Failure MCU Camera Underwater ML Machine Learning MMC Point Repair or Change in Pipe Material MSA Survey Abandoned MSC Shape/Size Change MSCC Manual of Sewer Condition Classification MWLS Water Level Sag NASSCO National Association of Sewer Service Companies ND Non-Defective xvi No. Number O&M Operation and Maintenance OB Obstacle Obstructions OCR Optical Character Recognition OR Overall Pipe Rating PACP Pipeline Assessment and Certification Program PDP Partial Dependence Plots PE Polyethylene PR Precision Recall PRD Defective Point Repair PVC Polyvinyl Chloride QA/QC Quality Assurance and Quality Control QOR Overall Quick Rating R Roots R-CNN Region-based Convolutional Neural Network RCP Reinforced Concrete Pipe RDII Rainfall Dependent Infiltration and Inflow ResNet Residual Network RF Random Forest RI Pipe Rating Index ROC Receiver Operating Characteristics RP Rectangular Pipe RUS Random Undersampling S Surface Damge SHAP Shapley Additive exPlanations SMOTE Synthetic Minority Oversampling Technique SMOTE-NC Synthetic Minority Oversampling Technique for Nominal and Continuous Features xvii SP Steel Pipe SRM Sewerage Rehabilitation Management SURF Speeded-Up Robust Features SVM Support Vector Machine T Taps TD Defective Taps TI Intruding Taps TN True Negative TP True Positive TPR True Positive Rate VB Video-Based VCP Vitrified Clay Pipe WF Weld Failure WRc Water Research Centre X Collapses or Deformed Rigid Pipes XGBoost eXtreme Gradient Boosting XXX Not Known pipe material YOLO You Only Look Once ZZZ Other pipe material xviii Acknowledgements I would like to express my appreciation to my supervisor, Dr. Barbara Lence, for her guidance and support throughout my master’s thesis. I am grateful for her availability and for her generosity in funding my certification for Pipeline Assessment Certification Program (PACP) by National Association of Sewer Service Companies (NASSCO) through the NSERC Grant (GR001517). This program is a significant part in the successful completion of this research. I would like to thank Dr. Sudhir Kshirsagar, President of GQC, for giving me the internship opportunity and for initiating the idea for this research. His expertise has been invaluable in steering the research toward the right direction, and his conscientious guidance during my internship was outstanding. I gained and developed many new skills during my time at GQC. I would like to thank Deven Mistry, Pavan Saranguhewa, and Srujana Rangapuram, who, during their time at GQC, assisted in developing models for this research, introducing me to valuable resources, such as the Fastai library and Microsoft Azure Optical Character Recognition (OCR), and providing data related insights, which have been helpful for this research. I would like to extend my appreciation to all participating utilities for their cooperation and willingness to share their data with our research team. This research was made possible by their collaboration. This research is funded by Mitacs and GQC through a Mitacs Accelerate Grant (No. IT28084) and NSERC (GR001517). In addition, Michael Tang from UBC ARC supported this research by granting us Microsoft Azure Credit through UBC ARC Azure Pilot 2023 program. I thank the Digital Research Alliance of Canada for their technical assistance when using the Narval Cluster. xix Finally, I am grateful to have my family and friends who have served as my emotional support and encouraged me to complete my study. xx Dedication “To the future generation” 1 Chapter 1: Introduction Sewer systems are responsible for collecting and transporting wastewater and surface runoff to wastewater treatment plants where its quality is improved before ultimately releasing it to the environment. These systems are crucial for maintaining public health, particularly in urban areas, and demand a significant amount of capital and routine inspection due to the harsh conditions under which they operate and the substantial costs associated with system failure. Statistics Canada (2022) reported that in 2020, 12% of publicly owned sewer pipes across the country with a diameter of 450 mm or less had unknown physical condition, 8.7% were in poor condition, and 5.1% were in very poor condition. Moreover, 13.9% of sewer pipes with diameters ranging from 450 mm to 1500 mm and 27.6% of those with diameters of 1500 mm or higher, had unknown physical conditions. In 2021, sewage infrastructure had the highest average age and shortest average remaining useful life in comparison with other infrastructures, such as electric power and highway networks. In the 2021 Infrastructure Report Card, the American Society of Civil Engineers (ASCE) (2021) gave the United States wastewater infrastructure a mark of D+. Both countries have made significant investments in capital to improve sewage infrastructure. In 2020, sewage infrastructure accounted for 15.4% of total capital investment on infrastructure by municipal, local, and regional governments in Canada (Statistics Canada 2022), while ASCE estimated that utilities allocated more than $3 billion towards replacement of around 4,700 miles of sewer pipeline nationwide in 2019 (ASCE 2021). Moreover, increasing population growth and the impact of climate change place additional strains on sewer systems, thereby accelerating the rate of pipe deterioration (Wang et al. 2012; Li et al. 2023). These issues, along with the limited budgets of utilities, and the harsh environments in which sewers operate highlight the need for efficient asset management of these underground systems. 2 Periodic condition assessment of sewers is necessary to assure that the system is operating properly, identify any defects and repair and replace the defective elements in time before they lead to system failure and cause environmental hazards. Sewer inspections are commonly undertaken using a vision-based inspection, such as closed-circuit television (CCTV) technology (Rahman and Vanier 2004; Kumar et al. 2018). However, this inspection method can be time consuming, subjective, and inconsistent as it relies on the individual inspector’s expertise and interpretation of CCTV video footage (Hassan et al. 2019). In addition, not all pipes can be inspected due to limitations in funds or access points (A) (Harvey and McBean 2014b; Tscheikner-Gratl et al. 2020). These concerns have led researchers to develop statistical and machine learning (ML)-based models to automate the CCTV inspection process and predict the pipe condition of uninspected pipes. The objectives of this thesis are to investigate the performance of various ML models for automating the CCTV inspection process, predicting the likelihood of different types of defects, and estimating the pipe characteristics that influence the occurrence of the defects in pipes for different datasets. The remainder of this chapter provides general background on sewer systems, and methods for assessing their condition. Chapter 2 provides the conceptual grounding and development of ML models advanced in this thesis for automating the CCTV inspection process and predicting the probability of various types of defects. Chapter 3 presents and analyses ML models applied for conducting a quality assurance and quality control (QA/QC) assessment of manual inspection reports of previously inspected videos, and for identifying defects in images from videos that have not yet been manually inspected. In Chapter 4, methods are developed for identifying defects that would lead to pipe collapse such as cracks, fractures, and breaks, and the presence of infiltration that can potentially lead to hydraulic failures. Chapters 3 and 4 have been 3 written as stand-alone journal articles. The findings of the research are summarized and discussed in Chapter 5. This work is a collaborative endeavour with Global Quality Corp. (GQC). 1.1 Sewer Systems There are three types sewer systems, namely sanitary, storm, and combined sewers. Sanitary sewers collect and transport domestic and industrial wastewater. Storm sewers are designed for transporting surface runoff without treatment to the nearby water bodies. Combined sewers are designed to collect both wastewater and surface runoff and to transfer these to wastewater treatment plants. Combined sewers are frequently found in older sewer systems and are susceptible to combined sewer overflows (CSOs) which typically occur during precipitation events, when excess untreated wastewater and surface runoff are released because the capacity of wastewater treatment plants is exceeded. This work focuses on sanitary and combined sewers. 1.2 Sewer Pipe Failure Unlike pressurized pipes in water distribution systems, in open flow sewer pipes, failures are typically not immediate, but are rather the result of accumulated issues over time (Delleur et al. 1998). For example, structural failures, such as pipe collapses, may result from cracks propagating to fractures, and eventually leading to the collapses. Pipe condition assessments are conducted to identify these defects in the pipes before they can lead to such failures. Sewer failures can be categorized as structural, hydraulic, and environmental. A structural failure occurs when sewers collapse or become damaged as a result of structural defects, e.g., cracks and fractures, corrosion, loss of support from the surrounding soil, improper installation, interference from other structures and excavation. Hydraulic failure refers to flooding, i.e., street and 4 basement flooding, surcharge, inflow and infiltration (I&I), water hammer, or flow instabilities. Hydraulic failures occur when there is a reduction in sewer capacity due to increased hydraulic roughness caused by aging pipe material and the accumulation of debris or deposits. An environmental failure occurs when the receiving water bodies are polluted due to the CSOs or sanitary sewer overflows. (Delleur et al. 1998) 1.3 Sewer Inspection Technologies and Standards The objective of condition assessment of a sewer system is to evaluate the current operational state of the system, identify the pattern of deterioration, and provide support for the maintenance and rehabilitation plans (Rahman and Vanier 2004). It is recommended that sewer inspections be conducted at least once a year, ideally at the end of the wet season (Marsalek and Schilling 1998). Figure 1 shows the overview of such technologies and categorizes them into two types, visual and physical techniques. While the industry-standard technology for condition assessment is the visual inspection via CCTV, physical technologies are also used to evaluate the structural integrity of the pipe wall and the condition of the bedding. These may obtain information regarding the pipe deterioration rate and the surrounding soils, and are not considered a replacement for CCTV inspections (Kley et al. 2013; Wang et al. 2022). In-line physical technologies refer to any method that requires access to the inside of the sewer and on-line technologies refer to methods that require access to the inside of the sewer as well as making contact with sewer material. The procedure of CCTV inspection is discussed in Section 1.3.1 and the two common coding standards used to record sewer defects are discussed in Section 1.3.2. For more detailed discussion of other inspection technologies, see Kley et al. (2013) and Wang et al. (2022). 5 Figure 1. Overview of main sewer inspection technologies (Kley et al. 2013) 1.3.1 CCTV inspection CCTV is used to provide the visual of the pipe interior without requiring human-entry, making it possible to inspect sewers that are not accessible by humans. The three main components of CCTV inspection are a CCTV unit truck, a flusher truck, and a video camera mounted on a tractor or crawler attached to a cable. Sludge and water can affect the CCTV performance, therefore the inspected pipe is cleaned and flushed before each inspection (Wang et al. 2022). During the inspection, a tractor or crawler with mounted camera is deployed down the access point, e.g., a manhole. The slack is pulled from the cable to ensure the meterage counter is set to zero distance at the beginning of inspection. The operator records the water level at this point and controls the tractor to move in the direction of inspection from the CCTV unit truck. The operator stops the tractor when defects, construction features, and other observations are detected and records them according to the coding standard specified by their utility client. The inspection is finished when another access point is reached or when there is an obstacle that prevents the tractor from moving forward. Figure 2 shows the typical setup at the beginning of CCTV inspection. 6 Figure 2. The setup of CCTV inspection 1.3.2 Coding standards Consistent coding standards provide the benchmarks for detecting changes in pipe condition over time (NASSCO 2022). While Canada has no national standard for sewer condition assessment, utilities typically follow standards that are widely used worldwide though some (e.g., the City of Winnipeg, MB) have developed their own set of standards (Rahman and Vanier 2004). Based on a survey conducted by Rahman and Vanier (2004) for National Research Council Canada, about 68% of participated utilities have used the UK standard Manual of Sewer Condition Classification (MSCC) by the Water Research Centre (WRc). In 2010, Canadian utilities began using the Pipeline Assessment and Certification Program (PACP) standard developed by the National Association of Sewer Service Companies (NASSCO), which is widely used in the United States. These two standards are used by the utilities participating in this study, and are briefly described here. For more details regarding these, see WRc (2004) and NASSCO (2022). MSCC by WRc is the first coding standard and was released in 1980. It provided the foundation for other condition assessment standards, including the PACP standard, which was developed in 7 2002 (Kley et al. 2013; NASSCO 2022). Both of these standards, comprise four categories of codes: i) structural, ii) service, for WRc, or operation and maintenance (O&M), for PACP, iii) construction, and iv) miscellaneous. For both standards, the pipe condition grade is calculated for structural and service or O&M defects. Structural defects, e.g., cracks, fractures, and offset joints, are used to describe the physical defects of the pipe wall and service. O&M defects, e.g., root, deposit, and infiltration, are used to describe types of foreign objects or other indications found in pipes that may interfere with the operation of the system. For both standards, scores for structural defects are based on the presence and severity of the defect, and scores for service or O&M defects may be based either on the severity of the defect or the percentage loss of cross-sectional area of the pipe. In the WRc standard procedure, MSCC is used to code the defect and the pipe condition grade is calculated according to the Sewer Rehabilitation Management (SRM) document (WRc 2001). The score of an individual structural defect ranges from 1 to 165 and an individual service defect ranges from 1 to 20. The structural defect score may be applied for different pipe materials. The overall pipe condition grades for pipes experiencing structural and service defects range from 1 to 5. For a structural or service defect, the corresponding pipe condition grade is based on whether the defect level falls within the specified ranges provided in in Table 1. For multiple defects, the mean of defect score or the worst defect score among the defects is selected. WRc suggests using the worst defect score among the defects to determine the pipe condition grade. Table 1. Defect score ranges for pipe condition grades (Rahman and Vanier 2004) Condition Grade 1 2 3 4 5 Structural <10 10 – 39 40 – 79 80 – 164 165 and > Service <1 1 – 1.9 2 – 4.9 5 – 9.9 10 and > 8 For the PACP standard, the scores of individual structural and O&M defects range from 1 to 5. The scoring procedure for any defect may be applied for different types of sewer systems, i.e., sanitary, storm, or combined systems. The coding standard employs multiple methods to calculate the pipe condition grade. The Overall Pipe Rating (OR) method is based on the sum of all the defect scores for defects exhibited within a pipe. The Pipe Rating Index (RI) is based on the average defect severity among all types of defects exhibited within a pipe, and is calculated by dividing OR by the total number of defects. Lastly, the Quick Rating is a four-character score used to express the number of occurrences of the two highest severity grades in the inspected pipe segment. For example, suppose that there are six structural defects with condition grade of five, five structural defects with condition grade of four, and five O&M defects with condition grade of five, and seven O&M defects with condition grade of three. The Quick Ratings for the structural and O&M defects are 5645 and 5537, respectively. For the Overall Quick Rating (QOR), which combines both defect types, there will be 11 defects with condition grade of five, five defects with condition grade of four, and seven defects with condition grade of three. Since the QOR only includes the two highest severity grades, the QOR of this pipe segment is 5A45 where “A” is used to represent values from 10 to 14 (in this case 11). The letter A and subsequent letters are used to signify number greater or equal to two digits. The QOR provides information regarding the highest condition grade, which is not always shown when using the first two rating methods. NASSCO suggests using the QOR to determine the pipe likelihood of failure (LOF) for risk-based asset management. Section 1.4 discusses the LOF and risk-based asset management. 9 1.4 Asset Management and Rehabilitation Effective maintenance and rehabilitation plans are crucial for maintaining sewer system performance at acceptable levels of service and preventing unexpected failures in a cost-efficient manner (Malek Mohammadi et al. 2019). Sewer system maintenance can be classified as: i) corrective or short-term maintenance to address immediate problems that affect the system operation and ii) preventive or long-term maintenance, including system rehabilitation and renovations, to prevent unforeseen problems in the system (Marsalek and Schilling 1998). This thesis focuses on preventive or long-term maintenance by aiding and advancing the condition assessment process and predicting potential defects and their causes in sewer pipes so that they can be inspected and repaired. Utilities use risk-based asset management to identify and prioritize critical assets (Vladeanu and Matthews 2019). To determine a pipe’s risk of failure, it’s LOF and consequence of failure (COF) are needed. The LOF of sewer pipes are typically estimated based on pipe condition grades obtained from a condition assessment. For WRc, the LOF ranges from 1 to 5 based on the ranges of defect scores provided in Table 1. For PACP, the standard suggests using QOR to determine LOF. For example, if QOR of the pipe segment is 4432, the LOF is 4.4. In general, the higher the condition grade is the higher probability failure. The COF is estimated based on the direct and indirect impact of pipe failure. The process of determining the COF of sewer pipes is more complex compared with the LOF of sewer pipes as it is important include both social and environment costs and not just economical cost. The estimation of these costs can be subjective, involve a great deal of uncertainty, and cannot easily be expressed in monetary terms. Hence, rather than expressing the COF in the monetary terms, alternative metrics have been used to measure the severity of a potential pipe failure. These metrics may include the proximity of the 10 pipe to waterways and important buildings such as hospitals, the type of road under which the pipe is buried, the diameter of sewer, and other pipe characteristics (Vladeanu and Matthews 2019). This work does not focus on COF and risk of failure of a given pipe. Refer to NASSCO (2022) and Vladeanu and Matthews (2019) for more details on COF and risk-based asset management. 11 Chapter 2: Literature Reviews In the era of constantly generated information, with unprecedentedly high temporal and spatial resolutions, so called big “data,” ML has emerged as a prominent area of research for addressing real-world problems, such as those in the medical, automobile, and infrastructure fields. ML is a data-driven approach in which a machine is trained to make accurate predictions by analyzing input data. ML is generally categorized into two main categories: supervised and unsupervised learning. Supervised learning involves training a model using labeled data sets to learn the relationship between the input and the desired output. Unsupervised learning involves using unlabeled data to make predictions based on the patterns in the input data. For this work, models are developed using supervised learning ML algorithms, for which the model is trained using labeled data sets. Supervised learning algorithms can be categorized by two types of output generated: classification, i.e., discrete variables, and regression, i.e., continuous variables. This study focuses on developing models to classify different types of defects found in the sewer system. 2.1 Classification Models These models classify input data into discrete categories, i.e., classes, based on their attributes, or model predictor variables, i.e., features. A binary classification model indicates only two possible classes, i.e., defect or no defect. A multi-class classification model indicates more than two classes for which each sample is assigned to only one class. A multi-label classification model indicates more than two classes for which each sample can be assigned to multiple classes. Chapter 3 applies both binary and multi-label classification models to classify different 12 types of sewer defects and Chapter 4 applies a binary classification model to predict pipes with high likelihood of having structural and infiltration defects. 2.2 Automating the CCTV Inspection Process In order to automate the CCTV inspection process, researchers have used computer vision (CV), a technology where a machine is trained to recognize and identify meaningful information in images and videos (Hawari et al. 2018). The application of CV can be used for image classification, object detection, and image segmentation. In the sewer defect classification and localization domain, the application of image classification determines the types of defects in images, object detection determines and locates the types of defects in images, and image segmentation delineates the shape of defects in images. Chapter 3 focuses on the application of image classification for determining the types of defects within sewer images. 2.2.1 Feature Extraction Approaches Image processing techniques such as edge detection techniques and morphology analysis have been be used to extract features such as edges, gradients, and textures of images to identify defects within the image, along with mathematical formulae to characterize the mechanics of interest (Xu et al. 1998; Yang and Su 2009; Su et al. 2011; Hawari et al. 2018). For example, Li et al. (2023) applies image process techniques to extract the shape of a pipe joint in images and uses these images to compute the deformation of the joint. The process is often complex requiring a high level of expertise. Other researchers combine the use of feature extraction techniques, such as Global Image Structure Tensor (GIST) descriptor (Oliva and Torralba 2001), histograms of oriented gradients (HOG) (Dalal and Triggs 2005), and edge detection, as well as traditional ML algorithms, such as Random Forest (RF) (Breiman 2001) and Support Vector 13 Machine (SVM), to detect and classify defects in sewer pipe images (Yang and Su 2009; Halfawy and Hengmeechai 2014; Myrans et al. 2019). However, these methods require one to pre-specify the input features, often exhibit poor robustness, and have limited capacity for generalization (Kumar et al 2018; Li et al., 2021). Generalization refers to the ability to classify images that possess substantial variations, e.g., color or brightness of the images, from images used in the training process Therefore, deep learning (DL)-based algorithms are more appealing in comparison because they have the ability to automatically learn the image features without the need for pre-specifying them (Cheng and Wang 2018). 2.2.2 Deep Learning (DL)-Based Models DL is a subset of ML that is based on artificial neural networks (ANNs), which is inspired from human neural networks, to process data. In contrast to simple ANNs with only a single hidden layer, the term “deep” in DL refers to the multiple layers in the networks that gradually extract more complex features from the input data as the depth of the layers increases. DL-based algorithms, specifically deep convolutional neural networks (CNNs), have become popular in the field of CV. CNNs are a type of DL that are designed specifically to process images as input. The input images undergo a series of convolutional layers, where both linear and non-linear operations are applied, to transform and compress them aiming to only extract critical features of images that are related to their labeled classes. In general, adding more layers to a DL-based model can improve its ability to extract features and its capacity for generalization (Ma et al. 2021: from sun et al). Kumar et al. (2018) is among the initial studies that apply CNN to automate sewer defect classification. The study developed three binary classification models to classify images of root intrusions, deposits, and cracks. Subsequently, other multi-class and multi-label classification 14 models have been developed to classify images of other sewer defects, such as cracks, deformations, deposits, holes, infiltration, joints, and roots (Hassan et al. 2019; Li et al. 2019; Meijer et al. 2019; Moradi et al. 2019). Some of these studies applied a hierarchical classification approach to distinguish between defective and non-defective (ND) images prior to classifying the defective images into multiple sewer defect classes (Chen et al. 2018; Li et al. 2019; Xie et al. 2019; Haurum and Moeslund 2021). This approach enhances the performance of the models, especially in data sets where there is an imbalance in the distribution of sewer defect classes. CNN-based algorithms, including region-based CNN (R-CNN), fast R-CNN, faster R-CNN, You Only Look Once (YOLO), have also been applied to perform object detection and image segmentation of sewer images and have shown potential in this endeavor (Wang and Cheng 2019; Kumar et al. 2020; Li et al. 2021; Tan et al. 2021). However, these tasks require one to manually draw a box around the defect or delineate the shape of the defect in order to create training data for the model. This study only focuses on CNN-based algorithms as image classification models, because, for asset management purposes, utilities are more interested in identifying the defects in the video than their position within a frame. 2.2.3 Transfer Learning One of the challenges encountered when using real-world data for developing CNN models for conducting CV identification of sewer defects for any one utility is the insufficient amount of data, which is typically inadequate to train the model so as to achieve a high level of accuracy. Transfer learning is used to overcome this challenge, particularly in the field of CV, which is computationally intensive and data demanding. The object of this approach is to transfer the knowledge acquired by a given model, which has been trained on extensive data sets for a 15 particular task, to apply it to a new model task (Gurjar and Voditel 2023). In the context of CNNs, the extracted image features and optimized parameters from a previously developed CNN-based model are used and modified to create a new CNN-based model that is suitable for a new task (Sun et al. 2023). Several researchers have utilized and adapted CNN-based models, including AlexNet (Krizhevsky et al. 2012) and Residual Network (ResNet) (He et al. 2016), which were originally trained on general images from an open-source visual database containing millions of images, such as ImageNet, to develop image classification models specifically for sewer defects images (Hassan et al. 2019; Li et al. 2019; Haurum and Moeslund 2021). Chapter 3 investigates and compares the performance of three well-known CNN-based models when applied to classify different types of sewer defects. 2.3 Sewer Deterioration Models The key factor for effective sewer maintenance and rehabilitation is a good inventory of the current state of the system which is created through the use of CCTV inspection (Marsalek and Schilling 1998). Nevertheless, not all pipes can be inspected due to budgetary constraints or inaccessibility. Hence, sewer deterioration modelling has been developed to estimate the sewer condition of uninspected sewers. This section gives the summary of three types of sewer deterioration models and describes different techniques that can be used to improve classification model performance when dealing with imbalanced data sets, i.e., exhibiting a disproportionate number of samples in different classes, which is common when working with real-world data. 2.3.1 Three Types of Sewer Deterioration Model Sewer deterioration models can be categorized into three types: i) deterministic (mechanistic relationship), ii) statistical (probabilistic relationship), and iii) ML-based (data-driven) models 16 (Kley and Caradot 2013 ). The objective of deterministic models is to use mathematical equations to determine the mechanistic relationships between pipe condition and various factors that contribute to pipe deterioration, such as pipe characteristics (e.g., pipe material and age), operational, and environmental factors. Such models are by design incomplete, frequently relying on simplified assumptions, and are inadequate for capturing the complexity of the deterioration process (Kleiner and Rajani 2001). Statistical models improve the ability to describe complexity by using probability-based equations to establish the relationship between pipe deterioration factors and historical data regarding condition of inspected pipes. The popular statistical methods used to analyze sewer deterioration are survival analysis, Markov-chains, logistic regression, and discriminant analysis (Le Gat 2008; Salman and Salem 2012; Egger et al. 2013; Rokstad and Ugarelli 2015; Caradot et al. 2017; Elmasry et al. 2017). Each of these models rely on underlying assumptions related to distributions and interdependencies of the model predictive variables. In contrast, ML-based models are data-driven models which can be based solely on historical data on the condition of inspected pipes and the factors that contribute to their deterioration without relying on other assumptions. However, they require extensive data sets comprising the condition of inspected pipes and their deterioration factors, and are computationally intensive. The popular ML-based algorithms used to develop sewer deterioration models are RF, SVM, and ANNs (Najafi and Kulandaivel 2005; Tran et al. 2006; Sousa et al. 2014; Harvey and McBean 2014b; Rokstad and Ugarelli 2015; Hernández et al. 2018; Laakso et al. 2018). Sewer Deterioration modelling is useful for predicting pipe condition of uninspected pipes. Nevertheless, the condition grade of a pipe does not provide information about the types of defects present in it. For example, two pipes with identical condition grades may have 17 experienced different failure types. Instead of modelling the condition grade of a pipe, Chapter 4 presents ML-based models that can relate different deterioration factors to specific types of defects, such as structural defects and infiltration. These models are suitable for utilities that want to target on a specific type of defect. 2.3.2 Techniques to Manage Class Imbalance Class imbalance can hinder the performance of a classification model because the model can correctly achieve high accuracy by prioritizing prediction of the majority class and incorrectly overlook a minority class. This issue poses a limitation, especially in fault detection problems where fault cases are considered rare, thus belonging to an imbalanced minority class. In the context of sewer defects, for example, there are more samples of pipes without structural defects than pipes with structural defects. Techniques that are used to address class imbalance fall into three categories: i) data-level approaches, ii) algorithm-level approaches, and iii) cost-sensitive learning (CSL) (Fernández et al. 2018). Data-level approaches are often considered as a preprocessing step in which resampling techniques are used to balance the class distribution of the training and validation data. Resampling techniques include: i) undersampling methods, in which samples of the majority class are removed from training data, ii) oversampling methods, in which new examples of the minority class are created and added to training data, and iii) hybrid methods are a combination of under and oversampling. The algorithm-level approaches involve using an algorithm that is capable of addressing the class imbalance problem. An ensemble algorithm, such as RF, is an example of and algorithm-level approach. The algorithm combines results from multiple classifiers into a single output, which 18 improves the model accuracy in comparison with algorithms that rely on output from a single classifier. Ensemble algorithms often incorporate additional techniques from data-level approaches and CSL to address the issue of class imbalance. Within the framework of class imbalance, CSL can be considered as either as a data-level or an algorithm-level approach. CSL manages the class imbalance by imposing a stronger penalty during the training process on cases when the minority class is misclassified than cases when the majority class is. This implementation can improve the model focus on the minority class. In Chapter 4, the performance of four well-known ensemble algorithms is assessed, along with three additional techniques for managing class imbalance. The objective is to identify the best combination of ensemble algorithm and class imbalance management techniques for predicting defects within sewer systems. 19 Chapter 3: Deep Learning-Based Sewer Defects Detection Model: Comparison of Models for Multiple Utilities 3.1 Introduction The deterioration of sewer infrastructure is a concern for utilities across Canada, where, according to Statistics Canada (2023), the percentage of the remaining useful life of sewers across the country was 54.8% in 2022. The inability to detect critical conditions in such structures can result in sudden collapse, causing significant economic repercussions and impediments to overall operations. Regular inspections of the sewer pipes are necessary in order to detect the defects before such events. Sewer inspections are commonly assessed using CCTV, mostly due to the ease of operating CCTV equipment and the existence of standardized condition protocols, e.g., PACP by NASSCO (Rahman and Vanier 2004; Kumar et al. 2018). CCTV inspection can provide visual information regarding the type and location of pipe defects found in the sewer interior. However, it can be time consuming, subjective, and inconsistent due to the fact that defects are detected manually and classified based on inspector interpretation (Hassan et al. 2019). In order to improve the CCTV inspection process, researchers have proposed methods that automatically detect defects and classify these using CV, where ML-based model is trained to replicate human vision, enabling it to extract and interpret meaningful information from various forms of visual data, such images and videos (Hawari et al. 2018). This study focuses on one CV application, image classification, wherein one or more classes are assigned to an image. In the recent years, CNNs have become popular algorithms in the field of image classification. Krizhevsky et al. (2012) created AlexNet, a CNN model that won the ImageNet contest to 20 classify over one million images. This achievement shows that CNNs are better at classifying images than other ML-based models, e.g., RF or SVM which require feature extraction, such as HOG, and Speeded-Up Robust Features (SURF) (Bay et al. 2008) to extract image features. Unlike the other traditional ML-based models, the CNN-based classification models can automatically learn optimal image features, such as edges, corners, or gradients, for image classification tasks without manually specifying them through a back-propagation algorithm (Cheng and Wang 2018). One of disadvantages of CNNs is that multiple hyperparameters must be tuned increasing computational time compared with other ML models. This may be addressed by using a pre-trained CNN model that has been trained on different data sets as a model architecture and further fine-tuning the model with new data sets. Kumar et al. (2018) develop binary CNNs to identify whether sewer images contain roots, deposits, or cracks. Their approach requires creating n binary models to predict n defects, i.e., three binary models to predict the presence of the three defects. To reduce the number of prediction models, multi-class classification models may be applied, where each image is assigned to one of three or more classes based on its probability of falling within a given class. Hassan et al. (2019) and Li et al. (2019), modify general image classification models, AlexNet and ResNet, respectively, as the model architectures to create sewer defect multi-class classification models. To address the class imbalance in data as the number of defective images are smaller than the number of ND images in nature, Chen et al. (2018), Li et al. (2019), and Xie et al. (2019) propose a hierarchical classification approach, referred to as the two-stage classification approach in Haurum and Moeslund (2021), and henceforth in this study. Here, defective images 21 are separated first from the ND images with a binary classification model (i.e., the first stage model), before classifying the defect type in the defective images with the second stage model. While the aforementioned models are multi-class classification models, Meijer et al. (2019) develop a sewer defect multi-label classification model, that identifies multiple classes within the images (i.e., where multiple classes can be assigned to one image). Haurum and Moeslund (2021) evaluate the performance of six state-of-the-art algorithms for general image classification: ML-GCN by Chen et al. (2019) ; ResNet101 by Wu et al. (2019) ; KSSNet by Wang et al. (2020); and TResNet-M, TResNet-L, and TResNet-XL by Ridnik et al. (2021), and six algorithms for sewer defect classification: Chen et al. (2018); Kumar et al. (2018); Hassan et al. (2019); Meijer et al. (2019); Myrans et al. (2019); and Xie et al. (2019). They evaluate these for classifying multiple classes within 1.3 million images from CCTV sewer inspection with and without hierarchical classification approach. The study found that the hierarchical classification approach using the Xie et al. (2019) model in the first stage model and TResNet-L in the second stage performs the best. In addition to image classification, object detection and image segmentation techniques are applied to locate defects and delineate the shape of defects within images (Cheng and Wang 2018; Wang and Cheng 2019; Kumar et al. 2020). However, these techniques require one to manually draw a box around the defect or delineate the shape of the defect in order to generate the training data for the model. In general, utilities are more interested in identifying the defects in the video than its position within the frame. Moreover, it is more important to know the location of the defect along the pipe which is typically displayed on the frame and can easily be extracted using a text recognition model. This study aims to evaluate the performance of three pre-trained CNN-based models, with a range of complexity, with and without hierarchical 22 classification approaches, for classifying multiple defects within sewer interior images from three different characteristic data sets. The model performances are evaluated based on the following objectives, to: i) conduct a QA/QC assessment of previously inspected videos, to evaluate consistency of inspection and ii) classify defects for videos that have not yet been inspected, in support of future visual inspections. 3.2 Methodology CCTV inspection videos and reports of sanitary and combined sewers that adhere to the PACP standard from three utilities in Western Canada are used to develop CNN-based multi-label classification models to identify defects within images extracted from the CCTV inspection videos. The information regarding the three data sets, referred to as Utility A-C is summarized in Table 2. In this study, defective images are referred to as images exhibiting defects or construction features, such as connection points, access manholes, and changes in pipe material or size. ND images are referred to as images that do not exhibit any of the aforementioned classes. There are 221 possible PACP codes that can be used as the model classes (NASSCO 2022). Any codes related to recording special observations, such as water level or the results of tests conducted during the inspection, such as dye tests, are not considered herein. Grouped classes are created to reduce the number of model classes to a more manageable size and to reduce class imbalance, in which some classes are found in more images than others 23 Table 2. CCTV inspection videos and reports information Information Utility A Utility B Utility C No. of CCTV inspection videos 791 1334 74 Average video resolutiona (no. of pixel width X height) 700 X 470 900 X 580 710 X 480 No. of imagesb 426,499 1,047,655 61,721 Inspection years 2018 – 2022 2017 – 2022 2018 – 2023 No. of unique PACP codes 88 140 73 Total survey length (km) 42 133 5 Types of pipe materialc (% of pipes of this type in the CCTV inspection videos) AC (12%) CP (56%) PVC (6%) RCP (1%) VCP (19%) XXX (6%) AC (4%); CP (57%); PE (<1%); PVC (13%); RCP (30%); SP (<1%); VCP (1%); ZZZ (2%) CP (4%); PVC (16%); RCP (8%); VCP (72%) Types of pipe shapec C AP, C, E, H, RP C Pipe height/diameter (mm) 150 – 600 200 - 4050 150 – 750 Note: No. = Number; AC = Asbestos Cement; CP = Concrete Pipe; PE = Polyethylene; PVC = Polyvinyl Chloride; RCP = Reinforced Concrete Pipe; SP = Steel Pipe; VCP = Vitrified Clay Pipe; XXX = Not Known pipe material; ZZZ = Other pipe material; AP = Arched Pipe; C = Circular; E = Egg-Shaped; H = Horseshoe; RP = Rectangular Pipe. aAverage width X average length pixel size. bTotal number of images used to develop the models. cBased on PACP condition assessment standard, except for AP and RP which are so noted to distinguish arched pipe from access point (A), and rectangular pipe from roots (R), respectively, see Table 3 (NASSCO 2022). PACP codes are sorted into grouped classes, as shown in Table 3, based on i) whether the non-zoomed-in images of the two defects are similar, such as in the case of cracks and fractures; ii) whether the PACP codes are under the same PACP family code, such as in the case of attached, settled, and ingress deposits; and iii) point repairs and changes in pipe material are considered to be within the same group. The identification of individual PACP classes are explored in the sensitivity analysis. Figure 3 shows the overall steps to develop the multi-label sewer defect classification models. First, images of sewer interiors are extracted and their corresponding defect classes are identified, based on defect annotations in the CCTV inspection videos and reports. These 24 comprise the ground truth images and their corresponding ground truth classes. The average frame rate of the obtained CCTV inspection videos is 30 frames per second, thus, every 30th frame of each video was extracted to create the model input. These data are used to train, validate, and test the one- and two-stage multi-label defect classification models and the efficacy of the models are assessed with evaluation metrics. Table 3. Sewer defect grouped classes Categories of Defects Grouped Classesa Description Structural Defect Ib CF Cracks or fractures BH Breaks or holes Xc Collapses or deformed rigid pipes DF Deformed flexible pipes DTd Deformed brick pipes Structural Defect IIb J Offset, separated, or angular joints S Surface damage LF Defective lining features WFd Weld failure BW Defective brick pipes Material/Size Change MMC Point repair or change in pipe material PRD Defective point repair MSC Shape/Size change (e.g., change in pipe shape or size) Operation and Maintenance D Attached, settled, or ingress deposits R Roots I Infiltration OB Obstacle Obstructions Construction Features T Taps (e.g., Types of taps that connect to the pipes) ISS Intruding sealing materials L Change in line of sewer A Access points Defective/Intruding Tap TI Intruding taps TD Defective taps Other MWLS Water Level Sag (e.g., Presence of a sag, dip, or low spot in the pipe) Non-Defective ND Non-defective aName of the grouped classes used in this study (Developed for this work). bStructural defects are split into two defect categories. c This grouped class does not have sufficient number of images to be split into training, validation, and testing data. dThis grouped class is not found in the three data sets. 25 Figure 3. Overall development process 3.2.1 Data Preparation Figure 4 shows examples of four examples of images observed in this study. Distance relative to the start of survey and PACP code annotations, if entered by an inspector, are located in various places depending on the type of annotation software used. In this study, ML-based Optical Character Recognition (OCR) from Microsoft Azure is used to extract these annotations from images. The OCR output, along with CCTV inspection reports, are used to assign ground truth classes to each image. All annotations are blurred in the image to prevent the classification model from associating words with certain defect classes during training. Should the PACP code identified in the CCTV inspection report and PACP code annotated on the video be inconsistent, the inspection report PACP code supersedes the code annotated in the video, given that the former are typically reviewed by a second inspector. In addition, the CCTV inspection report is 26 also used identify any potential PACP codes that may have been overlooked during the filming of the inspection. Figure 4. Sample images from CCTV inspection videos with distance and: a) no defect annotation, b) showing perspective view of defect and defect annotation, c) showing zoom-in of defect and defect annotation, and d) showing multiple defect annotations at the same distance There are three types of defects in video images to consider when creating the ground truth images and classes. These are point defects, continuous defects (i.e., defects that occur over a length of the pipe or recur at regular intervals) (NASSCO 2022), and non-defects. Their locations and PACP codes are determined in sequence. First, for each point defect, the location of the defect is identified based on the CCTV inspection reports. The first image at this location with a PACP code annotation as identified in the OCR output is selected as a ground truth image. To increase the number of defective images and recognising that frames taken near to the location of a point defect may have also captured it, if frames taken within a specified distance before the defect contain images that are considered similar to that of the defect, these images are assigned PACP codes corresponding to the given point defect. The similarity of images is determined by 27 applying a matchTemplate function from an OpenCV module in Python. The threshold likelihood of similarity and the distance from the point defect are determined by trial and error based on visual observations of a subset of the data. They vary with the type of annotation software used and the filming practices of the inspector. Additionally, images taken when the camera is underwater, denoted by the MCU PACP code or images at distances greater than or equal to the distance where the survey is abandoned, denoted by MSA, are excluded from the training data set. To reduce the occurrence of false negatives, images located within a specified distance of the location of the point defect that have a likelihood of similarity lower than the predetermined threshold are omitted from the training data set. Second, the continuous defects and the starting and finishing distances of them are identified as noted in the CCTV inspection reports, and any image that has a location within these distances is assigned the PACP code corresponding to the continuous defect. Finally, the remaining images with no PACP codes are assigned as ND images. This semi-automated image labelling process does not identify all errors in the video annotations, camera observations, and inspection reports. Rather, it provides an efficient approach for approximating ground truth data, which is beneficial when developing models based on large amounts of images. The ultimate purpose of these image classification models is to process large amounts of data and identify patterns that indicate a type of defect. Thus, the effect of subjective errors in data point on model performance is minimal. 3.2.2 Training and Testing The PACP codes for all images are sorted into grouped classes, as shown in Table 2. The data are then split into training, validation, and testing data. To evaluate the model performance for QA/QC purposes, the total data set is stratified split into the three data sets at a ratio of 3(training): 1(validation): 1(testing), to preserve the distributions of the class groups in 28 these data sets, see distributions for the training and validation data, as well as the testing data, shown in Figure 5a. Models trained and tested with these data are referred to as frame-based (FB) models. To evaluate the model performance for classifying defects in new inspection videos, videos are randomly selected to be reserved as testing data, until the number of test images reserved is approximately 20% of the total images. Images from the remaining videos are stratified split into 75% training and 25% validation data. For this split into the three data sets, also at a ratio of 3:1:1, the distribution of the class groups are the same for the training and validation data set, but are different for the testing data, as shown in Figure 5b. Models trained and tested with these data sets are referred to as video-based (VB) models. ND is the most frequent class for Utility A and C while deposit (D) is the most frequent class for Utility B. All images are resized to a fixed resolution, with the number of pixels of 700 (width) x 470 (height), and normalized because these dimensions are the smallest among the three data sets and there is no significant difference in the model performance when using a larger number of pixels. Other vision augmentation techniques, such as flipping, warping, rotating, and adjusting brightness, are explored in the sensitivity analysis and are not used in the general training process. Grayscale images do not improve the model performance and thus are not used in this work. Vision augmentation techniques, and model training are implemented using Fastai Python package (Howard and others 2018). 29 Figure 5. Grouped classes distribution of each utility data for a) FB and b) VB models 3.2.2.1 Deep Convolutional Neural Networks (CNNs) and Transfer Learning The architecture of CNNs is designed to focus on images, represented by 2- (height and width) or 3- (height, width, and channel) dimensional matrices of the pixels. A basic CNN model consists of three types of layers, i.e., convolutional, pooling, and fully-connected layers (O’Shea and Nash 2015). The convolutional layer evaluates the scalar product of a 2- or 3- (i.e., one 2-dimensional matrix for each channel) dimensional matrix of weights and biases, known as kernels, and sub-regions of the input. The dimensions (height and width) of the kernel are generally smaller than the input matrix. The kernels systematically move along the entirety of the input, mapping the original input onto a smaller matrix known as a feature map, which represents specific features of the input. The pooling layer is used to further reduce the dimensions of the feature map. The fully-connected layer is added as the last layer to calculate the class probabilities. During the training process, the weights and biases of kernels are updated after processing one batch size of data through a process known as a backpropagation. The batch size is a hyperparameter provided by a user to define the number of samples the algorithm considers a) b) 30 before updating the model parameters. The training process is undertaken for number of epochs, a hyperparameter that defines the number of times the algorithm has processed all samples in the training data set, as set by the user. If there are 100 samples in the training data set and a batch size is set to 20, the weights and biases of kernels will be updated five times in one epoch. Stacking convolutional layers in deep networks, i.e., by applying a number of kernels, allows more complex image features to be learned as the depth of the layers increase. This study investigates three pre-trained CNN models, MobileNet V3 Large (Howard et al. 2019) and two common versions of ResNets: ResNet50 and ResNet101, to determine whether model accuracy increases with the model complexity, i.e., with increases in the number of layers. Table 4 shows the information regarding the three pre-trained models. The approach of using pre-trained models to be further fine-tuned on different data sets or for different tasks is known as transfer learning (Sun et al. 2023). The advantage of transfer learning is that the weights and biases resulting from the pre-trained model can be used or modified to create a new CNN that is more suitable for the new data set, thus saving computational time. When applying the pre-trained model in transfer learning, some model weights and biases will not be updated during the fine-tuning process, these are considered to be frozen or non-trainable parameters. Weights and biases that are updated during the fine-tuning process are known as trainable parameters. Table 4. Pre-trained CNN models information Pre-trained Model Training Data Year No. of non- trainable parametersa No. of trainable parametersa Total parameters No. of layers Mobilenet V3 Large ImageNet 2019 2,947,552 1,022,032 3,969,584 28 ResNet50 ImageNet 2015 23,454,912 2,160,512 25,615,424 50 ResNet101 ImageNet 2015 42,394,816 2,221,440 44,616,256 101 aReported when the pre-trained model is first loaded. 31 ResNet introduces a new layer known as a residual block, also known as a skip connection, to resolve the problem of model degradation in CNNs, which causes a decline in the performance of the traditional CNNs as the depth of the network increases. Here, a non-parameterized short-cut connection is applied to bypass the convolutional layers that are not useful which prevents the performance of the deeper network model from degrading. ResNet models have been proven to outperform other well-known CNNs. MobileNet V3 Large is a generation of MobileNets (Howard et al. 2017), which are designed to have fewer parameters than the traditional CNNs and thus may be efficiently deployed on mobile and embedded devices which may result in a trade-off between the model accuracy and computational time. In this study, all models are trained using the Alliance Narval cluster with 4 x NVIDIA A100 GPUs. First, as recommended by Fastai, each model is fine-tuned for four epochs for a set of parameters that are partially trainable, that is, both non-trainable and trainable model weights and biases are fine-tuned. See Table 3 for the total number of these parameters. Then the non-trainable weights and biases are unfrozen, i.e., allowed to be updated, and the model is executed for more ten epochs. Based on the validation data set, application of a range of epochs, showed that the validation loss does not further decrease beyond ten epochs. The batch size is limited by computational resources to be 16 for Utility B with ResNet101 model (the utility with the largest number of images, and the model with the greatest complexity) and 32 for all other utility-model pairs. 3.2.2.2 One-Stage and Two-Stage classification approaches The one-stage classification modelling approach develops a multi-label classification model trained on the full training data set to classify the types of defects present in the defective images and to identify ND images. The two-stage classification modelling approach develops two 32 classification models. The first is a binary classification model trained on the full training data set to distinguish defective images from ND images. The second is a multi-label classification model trained only on the data for defective images to classify the types of defects present in the images. The two models may be trained in parallel. However, to execute the two-stage model for prediction purposes, the models must be run in series, where the results of the first model are concatenated with the second model to provide the final output. The two-stage classification approach is used to address the data imbalance problem where the majority of data set mainly consists of ND images. However, it is more computationally intensive than the one-stage classification model. 3.2.3 Evaluation Metrics The prediction of each multi-label classification model is calculated at the probability threshold set to 0.5, which means that an image is predicted to have a given class with a probability above 0.5. The performance of each model is evaluated using the precision [Eq. (1)], recall [Eq. (2)], and Fβ score [Eq. (3)] (Rijsbergen 1979). Precision is the portion of images that are correctly classified as belonging to a given class, among the total number of images that are predicted to have that class. Recall is the portion of images that are correctly classified as belonging to a given class, among the total number of images that actually belong to that class. The Fβ score represents a compromise between precision and recall. The equations of these metrics for each possible class are defined as: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑐𝑙𝑎𝑠𝑠 = 𝑇𝑃𝑇𝑃+𝐹𝑃 (1) 𝑅𝑒𝑐𝑎𝑙𝑙𝑐𝑙𝑎𝑠𝑠 =𝑇𝑃𝑇𝑃+𝐹𝑁 (2) 33 𝐹𝛽,𝑐𝑙𝑎𝑠𝑠 =(1+𝛽2)∙𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛∙𝑅𝑒𝑐𝑎𝑙𝑙𝛽2∙𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙 (3) where TP = true positive, i.e., the number of images that belong to a given class that are correctly classified as belonging to it; FP = false positive, i.e., the number of images that do not belong to a given class that are incorrectly classified as belonging to it; FN = false negative, i.e., the number of images that belong to a given class that are incorrectly classified as not belonging to it; and β is pre-defined coefficient used to adjust the importance of recall relative to precision. The value of β is set to one when precision and recall have the same weight; set to larger than one to give more importance to recall; and set to smaller than one to give more importance to precision. The arithmetic mean of all classes, referred to as macro averaging, is used to evaluate and compare the performance of different models. While the majority of the previous research uses F1 score, where β is set to one, to evaluate the model performance, Haurum and Moeslund (2021) suggest using the F2 score, where β is set to two, in addition for defective images, to give more importance to recall (i.e., placing emphasis on false negatives over false positives), given that the pipe would fail if the defect was left unnoticed. In this study, the macro averaged of the F1 and F2 scores over common defect classes that are found both in training and testing data are used to evaluate the model performance for identifying and classifying types of defects in the images. The F1 score associated with the ND class is used to evaluate how well the model identifies the ND images. 3.3 Results and Discussion The evaluation metric scores of each model are provided in Figure 6, where light blue-orange points represent all FB models, and light green-red points represent all VB models. Figure 6a shows that for the FB models, Utility A- and C-based models have macro averaged F1 and F2 34 scores which are close to one, while those for Utility B-based models range from 0.60 to 0.76.This relatively lower performance compared with Utility A- and C-based models may be attributed to the wider size- and shape-ranges of the sewers in Utility B, resulting in more variation in the location and appearance of defects, which confound classification, as well as the somewhat larger class imbalance of images in the data set, as exhibited by the relatively steeper class distribution shown in Figure 5a. For the VB models, all models for the three utilities have macro averaged F1 and F2 scores lower than 0.50, with those for Utility C being the lowest. Nevertheless, Figure 6b shows that all models can detect ND images with F1 scores higher than 0.50, with Utility A-based models being the highest at 0.80. 3.3.1 Evaluation of One- and Two-Stage Classification Models In Figure 6, the light blue-dark blue and yellow-orange points, represent the results of FB one- and two-stage models, respectively, and the light green-dark green and pink-red points, represent the results of VB one- and two-stage models, respectively. For FB and VB models, the two-stage models perform similarly to the one-stage models in classifying defective groups, except for Utility B, where the two-stage models are better, as shown by their macro averaged F1 and F2 scores in Figure 6a. The two-stage models have a somewhat higher macro averaged recall than the one-stage models, particularly for Utility B, for defective grouped classes. As shown in Figure 6b, the recall scores for the ND class in the two-stage models are comparatively lower than those of the one-stage models, particularly for the VB models, though this is less pronounced for Utility B. There are in general more images in the ND class than in any other defect class, and having been trained on a large number of ND images, the one-stage model is more likely to correctly classify an image as ND, than as belonging to other grouped classes. For the two-stage model, the first model is applied to classify ND and defective images, and in doing 35 so, class imbalance is reduced. The separated data have a higher proportion of defective images, and the model is more likely to correctly classify an image as belonging to a given defective group, than to the ND class. If it is more important to avoid misclassifying defective images as ND, than to misclassify ND images as defective, the two-stage model is more attractive than the one-stage model. By having a first model that separates ND from defective images, the two-stage model also has higher precision scores for the ND class than the one-stage model. Figure 6. Macro-averaged metric scores of a) defective grouped classes and b) the ND class For categories of similar defects (see Table 3), Figure 7 shows the percentage difference in the correctly identified images, i.e., TP images, and the incorrectly identified images, i.e., FN images, of the two-stage model relative to those for the one-stage model. High values of TP and low values of FN, are attractive as they correspond to high recall values. For the majority of defect categories, the two-stage models yield a decrease in the number of FN images, denoted by negative orange bars, and an increase in the TP images, denoted by positive blue bars, and are a) b) 36 generally more attractive. The classification procedure for the multi-label classification model can result in the same image being classified as belonging to two or more groups based on whether the likelihood of belonging is higher than the set probability threshold. Unlike the two-stage model, which separates ND images in the first stage, the one-stage model may indeed classify an image as being ND and as belonging to one or more defective grouped classes. Figure 7. Percent difference in TP and FN images of two-stage relative to one-stage models for a) FB ResNet50 models and b) VB ResNet50 models 3.3.2 Evaluation of Pre-trained Models The best pre-trained model used for defect multi-label classification varies depending on the utility. In Figure 6, the darkness of the data point represents the level of model complexity, with the lightest colored points corresponding to models based on MobileNet V3 Large, medium colored points corresponding to models based on ResNet50, and darkest colored points corresponding to models based on ResNet101. In general, models based on pre-trained ResNet 50 and ResNet101 exhibit superior performance compared with those based on MobileNet V3 Large. As shown in Figure 6a, there is little difference between the macro averaged F1 and F2 a) b) 37 scores of defective grouped classes observed when using the three different pre-trained models trained on Utilities A and C data sets. Models based on Utility B, which possesses the largest data set, exhibit the most pronounced differences among the pre-trained models. The Utility B-based model using pre-trained ResNet50 has the best performance among the FB models, where as the model using pre-trained ResNet101 has the best performance among the VB models when classifying different defective grouped classes. The two-stage ResNet50 model performs on par with that of ResNet101 and requires less computational effort. See Table 11 in the Appendix A for a comparison of the model computation time per 10,000 images. Figure 8 summarizes the performance of the two-stage ResNet50 model, providing the proportion of TP, denoted by hashed bars, and FN, denoted by non-hashed bars, in relation to the total number of images for each grouped class. The non-hashed bar with a darker shade represents the proportion of images that have been incorrectly categorized as belonging to different grouped classes. The non-hashed bar with the lightest shade represents the proportion of images that were not classified as belonging to any grouped class. For FB models, Figure 8a shows that, Utility A- and C-based models have TP values higher than 0.8. The Utility B-based model has TP values higher than 0.6 for most grouped classes, with the exception of the defective brick pipes (BW), defective point repair (PRD), shape/size change (MSC), defective taps (TD), and intruding taps (TI) grouped classes which are the five classes with the lowest frequently of occurrence in the Utility B data set. This relatively lower performance compared with Utility A- and C-based models, is also observed for all other one- and two-stage models as shown in Figure 6a. This may be attributed to the wider size- and shape-ranges of the sewers in Utility B, resulting in more variation in the location and appearance of defects, which confound classification, as well as the somewhat larger class 38 imbalance of images in the data set, as exhibited by the relatively steeper class distribution shown in Figure 5a. For VB models, Figure 8b shows that even though the models do not perform well in terms of classifying defective groups compared with the FB models, they can distinguish ND images from defective images, with all models having TP values for ND of more than 0.5, and as much as 0.8 for Utility A. In general, models based on the data from the three utilities, despite training on different the distribution of classes, can identify both roots and tap features more easily than other defects with the same number of images. In addition to applying transfer learning using pre-trained models that are based on generic sets of images and fine-tuning these models with training data for any one utility, transfer learning is also applied herein using the finely-tuned two-stage ResNet50 model for each utility as the pre-trained model and further fine-tuning it with training data from each of the other utilities. The resulting macro averaged F1 and F2 scores of the defective grouped classes are then evaluated with the testing data for these other utilities. This is undertaken to investigate whether using a pre-trained model that has been trained on sewage images from other utilities results in better performance than using a pre-trained model that has only been trained on a generic set of images. 39 Figure 8. Portion of true positive, misclassified, and unclassified images of each grouped class for a) FB two-stage ResNet50 and b) VB two-stage ResNet50 for each utility The macro averaged F1 and F2 scores of this transfer learning exercise are provided in Table 5. For comparison-sake, F1 and F2 values for the two-stage ResNet50 model based solely on a given utility, that is, the model that is only fine-tuned once with this utility’s data, are provided in bold on the diagonal. For example, for the FB model based solely on Utility A data, the macro averaged F1 and F2 scores are 0.96 and 0.95. If the pre-trained model is based on Utility B, and this model is further fine-tuned with data from Utility A, its macro averaged F1 and F2 scores are 0.96 and 0.96. Likewise, if the pre-trained model is based on Utility C, and this model is further fine-tuned with data from Utility A, the F1 and F2 scores are 0.96 and 0.95. a) b) 40 Table 5. Macro-averaged F1&F2 scores of the defective grouped classes for two-stage ResNet50 model pre-trained with data for a given utility (columns) and further fine-tuned with sewer images from other utilities (rows) Fine-Tuned with Utility Utility for which Pre-trained Model is Developed FB Models VB Models A B C A B C A 0.96&0.95a 0.96&0.96 0.96&0.95 0.34&0.30a 0.36&0.33 0.35&0.32 B 0.73&0.67 0.76&0.70a 0.73&0.67 0.26&0.25 0.25&0.24a 0.25&0.24 C 0.99&0.99 0.99&0.99 0.99&0.98a 0.22&0.20 0.23&0.20 0.22&0.20a aModel based solely on data for one utility, no additional data from any utility incorporated. Table 5 shows that using pre-trained models that are based on larger data sets and fine-tuning these with smaller data sets can improve performance of models for the utilities with the smaller data sets. For example, the slight improvement with respect to two-stage ResNet50 models based on Utility A or C data, when the pre-trained model based on Utility B is applied and further fine-tuned with data from Utility A or C. However, any increases in performance are more significant for VB models that initially have a relatively low level of performance, than for FB models, that are already performing well. Using pre-trained models for smaller data sets, and fine-tuning these with larger data sets, also show a slight improvement in performance for VB models. Conversely, such models degrade the performance of FB models developed for utilities with larger data sets. 3.3.3 Sensitivity Analyses A sensitivity analysis is conducted to evaluate the sensitivity of model performance to changes in the approach for grouping defects and in the model hyperparameters. Figure 9 presents the performance of the two-stage ResNet50 model when training and testing it for identifying the grouped classes, as well as for identifying all of the individual PACP classes present in the data, see Table 1 for the number of classes for each utility. As the ND image classes are identified in the first-stage model, they are the same in both cases, and only the macro averaged F1 and F2 41 values for the defective classes vary. For FB models, Figure 9a shows that there is a slight decrease in the model performance of defective classes for Utilities A- and C-based models and a significant decrease in performance for Utility B-based models, when models are developed for classifying individual defects rather than grouped defects. For VB models, Figure 9b shows the performance of all utility-based models greatly decrease when classifying individual defects rather than grouped defects. While the individual class approach has the potential for providing more information about specific defects than the grouped class approach, it exhibits relatively lower performance due to increased class imbalances where there are relatively fewer number of images for each class. Conversely, the grouped class approach is useful in that it is more accurate at indicating the existance general types of defects. Figure 9. Metric values of the grouped classes and individual classes a) FB two-stage ResNet50 and b) VB two-stage ResNet50 models To evaluate the performance sensitivity to variations in the model hyperparameters and to other enhancement methods, such as weighted loss function and vision augmentation, the two-stage a) b) 42 ResNet101 model based on data for Utility A is chosen as the reference model. A number of the model enhancements are computationally intensive, and Utility A satisfies a trade-off between data richness and computational effort (i.e., Utility A has the second largest data set, but the two-stage ResNet101 model based these data requires less than one-quarter of the computational time of the same model for Utility B). The ResNet101 model is also the best pre-trained model for Utility A. Application of the weighted loss function is examined because it addresses class imbalance, prioritizing the accurate identification of grouped classes with lower frequency over those with higher frequency. In order to create diverse variations of images within each class, i.e., more information regarding the class, vision augmentation techniques are applied. Here 50% of the images are randomly selected to undergo flipping, warping, rotating, and adjusting brightness simultaneously, where the characteristics of these techniques, such as magnitude and angle, are set at default values. In addition to these enhancement methods, variations in the value of the probability threshold, number of epochs for training the model with partially trainable parameters, number of epochs for training the model with fully trainable parameters, and ratio split of training and validation data are examined. Figure 10 shows the change in model performance for these modified models relative to the two-stage ResNet101 model based on data for Utility A. As the hyperparameters and enhancement methods are only changed in the second stage of the model, only the relative macro averaged F1 and F2 values for the defective classes are provided. The performance of FB models does not change significantly, less than by 1%, compared with VB models which change by on the order of 5-10%. Applying weighted loss function results in greater performance improvement than other model hyperparameter variations and enhancement methods. For the VB models, the performance increases when decreasing the number of epochs for training the model with 43 partially trainable parameters and increasing the number of epochs for training the model with fully trainable parameters. Lastly, the changes in the training and validation ratio have the smallest effect on model performance. Figure 10. Model performance relative to two-stage ResNet101 Utility A model for different hyperparameter values and enhancement methods for a) FB model and b) VB model Finally, as previously mentioned Utility B has a large range of pipe sizes, where pipes smaller than or equal to 750 mm are all circular, and larger pipes are comprised of a variety of shapes. To examine model performance sensitivity the pipe-size ranges for these data, three models are examined, namely the two-stage ResNet50 model for Utility B data, trained with i) all pipe size data, as previously analyzed, ii) pipes with diameter less than or equal to 750 mm, and iii) pipes with diameter greater than 750 mm. For all models, there is no improvement in the model performance when data are divided according to the pipe diameter. a) b) 44 3.4 Conclusions CCTV is a widely used tool for inspecting the condition of the interior of pipes and provides valuable support in the maintenance of deteriorating sewer system infrastructure. However, manual examination of the CCTV inspection video is subjective and time consuming. Previous studies have explored the use of deep learning to automate defect detection and classify CCTV inspection images. However, previous analyses were conducted solely on a single data set and only focused on a limited range of defect types. This study assesses the effectiveness of CNNs with varying complexity and configuration in detecting and classifying defects based on PACP codes. The defects are grouped into classes and the images are obtained from three data sets, each having different pipe and defect characteristics. The study evaluates the performance of the CNN models in terms of their effectiveness as a QA/QC assessment tool and their ability to accurately classify defects in unexamined videos. In the former case, high performing models provide a lens through which to evaluate whether inspections undertaken identify defects that are consistent with the model output. In the latter case, high performing models provide a means of identifying defects prior to, and thus in support of, future visual inspections. To undertake this study, a semi-automated image labelling process is developed that can be used to create ground truth data to save time when processing large data sets. The resulting models are able to learn specific image patterns and relate them to defect classes, minimizing reliance on subjective ground truth. The two-stage modelling approach is shown to perform better than one-stage modelling as the model results in higher recall scores of defective classes and a smaller number of unclassified images. It also eliminates the possibility of assigning both ND and defective classes to the same images due to the nature of its multi-label classification approach. Typically, increasing the 45 complexity of a model can improve the model accuracy, though for this study such improvements are most noticeable for Utility B, which has a large data set and a wide variation in image location and appearance, the analysis of which may benefit from increased model complexity. The usage of pre-trained models trained with data from different sewer utilities marginally enhances the performance in detecting and classifying defects for new data sets. Though preliminary, these results show that models trained with larger data sets may be used to support the development of models for explaining smaller data sets. This is helpful for utilities that have limited data. For QA/QC purposes, the performance of models based on data for Utilities A and C are better than those based on Utility B. This may be due to the fact that the Utility A- and C-based models are tested on smaller data sets that contain less image variation for each defective class compared with the Utility B testing data. Thus, the images in the testing data for Utilities A and C may be more similar to the images in their training and validation data set. In general, the models perform poorly when classifying defects in unexamined videos, and there is minimal difference in performance of such models trained on different utility data. Different hyperparameter values and applications of enhancement methods do not significantly affect performance when training of the models for QA/QC purposes. However, applying the weighted loss function, lower probability threshold, and lesser number of epochs when training models with partially trainable parameters can lead to a better performance when the model is trained to detect defects in unexamined videos. 46 While the proposed models would perform well if used for QA/QC purposes, improvements in the models for their use in identifying defects in unexamined CCTV videos are recommended. These may include obtaining more images of defective classes and developing approaches for identifying the optimal set of videos for training and testing VB models that ensure a stratified split among the data. Other applications of CV, such as object detection and image segmentation, should be investigated to improve the functionality of the models. Finally, future work should be undertaken toward advancing the application of the models to live-streaming CCTV video during the inspection process, and demonstrating their use in assisting live inspections. 47 Chapter 4: A Novel Sewer Condition Assessment Approach that Leverages Advanced Machine Learning (ML) Models 4.1 Introduction The majority of sewer networks in North America date to the 1960s, and are facing increased loadings related to population growth and climate change, exacerbating combined sewer overflow and other operational problems and their commensurate environmental and consumer consequences. Asset management strategies for such aged networks are hindered by the significant capital investment required and the inherent difficulty in assessing the condition, and thus service life, of buried sewers (Harvey and McBean 2014b; Tscheikner-Gratl et al. 2020). Typically, sewer condition is assessed by examining the sewer interior with a remotely operated CCTV rover, and operator inspection of the CCTV videos and estimation of the condition grade based on the quantity and types of defects observed, and their intensity. Given the expense of the sewer inspection process, and other physical limitations, researchers have developed sewer deterioration models to predict the condition of uninspected sewers. These models are based on relationships between deterioration factors, i.e., physical, environmental and operational variables, and estimated condition grades for inspected pipes in the network, where such relationships may be mechanistic, statistical, or based on a ML algorithms (Kley and Caradot 2013 ). While these models may be helpful in predicting condition of a network, they are limited in that they are specific to the single condition assessment standard applied during inspection (e.g., PACP by NASSCO), with a unique definition and scale of condition grade that is based on a unique method of defining, assessing and aggregating observed defects. Even when applying a single condition assessment standard, estimated condition grades only provide limited 48 information, due to the aggregation process. For example, two pipes with the same condition grade may have experienced different failure modes and these can often be explained by different deterioration factors (Rokstad and Ugarelli 2015). In addition, the condition grading process is subjective and dependent on inspector interpretation and experience (Rahman and Vanier 2004), and thus is subject to uncertainty (Rahman and Vanier 2004; Rokstad and Ugarelli 2015; Salihu et al. 2023), though Salihu et al. (2023) provide one means of addressing this subjectivity. This study develops and analyzes binary classification ML-based models for predicting whether or not specific defects may be expected and identifies the deterioration factors that contribute to the presence of these defects. The ML models investigated are based on variations of ensemble learning decision tree algorithms that have different capabilities, e.g., in terms of their computational efficiency and ability to accommodate categorical as well as numerical data types. The potential value of ML-based sewer deterioration models for single datasets is demonstrated with ANN-based algorithms (Najafi and Kulandaivel 2005; Tran et al. 2006; Khan et al. 2010) and SVM (Mashford et al. 2011). Syachrani et al. (2013) compare regression, ANNs, and decision tree algorithms for six datasets and find that the decision tree algorithm consistently outperforms the others. While, Sousa et al. (2014) show that an ANN algorithm outperforms an SVM, Harvey and McBean (2014a) show that a decision tree algorithm outperforms an SVM. In more recent studies, RF, which employs an ensemble of decision trees, has become the most popular algorithm for this application, providing high performance in predicting sewer condition (Vitorino et al. 2014; Harvey and McBean 2014b; Rokstad and Ugarelli 2015; Caradot et al. 2018; Hernández et al. 2018; Laakso et al. 2018). With the exception of Hernández et al. (2018), all of these studies develop the deterioration model using a single condition assessment standard 49 and many of them suffer from limited and imbalanced data, i.e., where the proportion of the condition grades of the pipes in the data set are not uniform. Caradot et al. (2018) note that such imbalances lead to biased predictions. Most of these studies analyse the importance of the deterioration factors for affecting condition. Malek Mohammadi et al. (2020) reviewed the findings of work from 2001 to 2019, and identify the several factors that affect condition and note cases in which past studies obtain contradictory results. Rather than predicting pipe condition, ML-based models that predict specific defects at the individual pipe level are developed herein, to target major maintenance concerns facing utilities, specifically structural defects and infiltration. The structural condition of a pipe is associated with its ability to resist external loads which could lead to deformation and collapse (Malek Mohammadi et al. 2020) and these, along with infiltration, if left unchecked, will reduce the hydraulic capacity of sewers and exacerbate further structural problems (Najafi and Kulandaivel 2005). Models based the following four well-known ensemble learning decision tree algorithms are explored: RF, eXtreme Gradient Boosting (XGBoost) (Chen and Guestrin 2016), Light Gradient Boosting Machine (LightGBM) (Ke et al. 2017), and Category Boosting (CatBoost) (Prokhorenkova et al. 2019). These algorithms develop multiple ML models built with different guiding principles, and integrate predictions from these in different ways. Three methods for managing imbalanced data are also examined, CSL, oversampling, and undersampling. The models are developed and evaluated for sewer inspection data for two utilities in Western Canada with widely varying pipe characteristics, one based on both PACP and MSCC by WRc condition assessment standards, and one based solely on PACP. Shapley Additive exPlanations (SHAP) (Lundberg and Lee 2017) are applied to determine the factors that contribute to the presence of the specific defects. The analysis of deterioration factor importance and the 50 exploration of the partial dependence plots are used to provide insights into the impact of various model predictor variables on different defects, or target variables, offering valuable information for future installations. The Methodology Section discusses the procedure for analysing ML algorithms for predicting sewer defects, including, data preparation, model development, model evaluation, and analysis of deterioration factor importance. The Results and Discussion evaluates the performance of the resulting ML models for predicting specific defects and for providing insights regarding influencing factors. The Conclusions and Recommendations summarize the findings and discuss possible future work. 4.2 Methodology For a pipe within the sewer network with a given pipe identification number (pipe ID), the ML-based models developed herein predict whether infiltration or structural defects are expected to occur based on the pipe characteristics as obtained from the utility Geographic Information System (GIS) database and the CCTV inspection reports. Inputs to the models include this information, as model predictor variables, and the model target variables are whether or not a known defect of a given type is located in the pipe. 4.2.1 Model Predictor Variables CCTV inspection reports and sample pipe information from two utilities, referred to as Utility A and Utility B, are summarized in Table 1. The reports cover a portion of the entire network where the network of Utility A consists of 400 km of sanitary sewage pipes and that of utility B of 553 km of combined and sanitary sewage pipes. 51 Table 6. CCTV and Sample Pipe Information Utility Utility A Utility A Utility B Condition assessment standard PACP WRc PACP No. CCTV inspection reports 799 2839 1376 Inspection years 2018 – 2022 1989 – 2015 2017 – 2022 No. pipes in sample (total length) 799 (48km) 2839 (172km) 1837 (184km) No. unique pipe IDa (total length, % of network total length) 698 (42km, 10%) 2231 (131km, 33%) 1439 (137km, 25%) Age range (years) 1 – 71b 1 – 71b 1 – 122 Types of pipe materialc (% of pipes of this type in sample) AC (2%), CP (12%), PVC (1%), RCP (<1%), SP (<1%), VCP (4%), ZZZ (<1%) AC (17%), CP (13%), PE (<1%), PVC (5%), VCP (26%), XXX (<1%), ZZZ (<1%) AC (3%), CP (58%), PE (<1%), PVC (11%), RCP (25%), SP (<1%), VCP (<1%), ZZZ (2%) Types of pipe shapec C C AP, B, C, E, H, RP Pipe height/diameter (mm) 150 – 600 150 – 700 200 – 4050 Note : No. = number; AC = Asbestos Cement; CP = Concrete Pipe; PE = Polyethylene; PVC = Polyvinyl Chloride; RCP = Reinforced Concrete Pipe; SP = Steel Pipe; VCP = Vitrified Clay Pipe; XXX = Not Known; and ZZZ = Other; A = Arched; B = Barrel; C = Circular; E = Egg-shaped; H = Horseshoe; R = Rectangular. aDue to some pipes being inspected more than once. bEstimated by utility. cBased on PACP condition assessment standard, except for AP and RP which are so noted to distinguish arched pipe from access point (A), and rectangular pipe from roots (R), respectively, see Table 3 (NASSCO 2022). Table 7. Model predictor variables Variable Name (unit) Data Type Description X-coordinate (m) numerical x-coordinate of pipe’s centroid Y-coordinate (m) numerical y-coordinate of pipe’s centroid Age (year) numerical pipe age at the time of inspection Pipe size (mm) numerical pipe height/diameter Pipe length (m) numerical pipe length Pipe shape categorical type of pipe shape Pipe material categorical type of pipe material The data in common for both utilities are selected as the predictor variables and these are listed in Table 7. To identify the pipe location, age at the time of inspection, and pipe length associated 52 with a pipe, and whether or not a defect of a given type is present in the pipe, the location and type of all defects are required. CCTV inspection often takes place between adjacent manholes (WRc 2004; NASSCO 2022). Inspection reports used herein provide the set of pipe IDs of the pipes viewed, the location of any defect from the starting point of the inspection, and the length of the inspection. The location of the defect in 2-space, is determined by noting the distance to defect relative to the starting point and the pipe IDs in the inspection report, and tracing the pipe IDs sequentially along polylines according to their order in the GIS database. Should only the ending point of the inspection be available in the GIS database, the defect may be located relative to the ending point by tracing them sequentially along the polyline in the reverse order of the pipe IDs. Like most utilities in the reviewed literature, Utilities A and B do not have extensive records regarding environmental information (Malek Mohammadi et al. 2020). In lieu of such information, the location of the pipe centroid, represented as two variables, is chosen as it may help relate the model findings to nearby features such as soil types, road cover, and traffic loads. Owing to repairs that may have taken place over time, a number of cases occur in which the pipe material, pipe size, or pipe shape documented in the CCTV inspection reports are different from the material associated with the respective pipes as reported in the GIS database. Since the information in the CCTV inspection reports is more current than the GIS data and is considered more reliable, the source of size, shape, and material information is the inspection reports. For Utility A in some cases, pipe sizes (42 pipe samples) and pipe materials (17 pipe samples) are not provided in the inspection reports, and in these cases, this information is obtained from the GIS database for the given pipe ID. 53 4.2.2 Model Target Variables While the PACP and WRc condition assessment standards are different, they are both comprised of a defect coding system which describes the defect type, the severity of defects, or the defect direction or location on the circumference of the pipe (WRc 2004; NASSCO 2022). While a variety of defect types are typically noted, for purposes of the ML-models developed herein, the relevant defects are sorted into two groups, that is: i) all defects associate with infiltration are considered one group, regardless of the level of distress, direction, and circumferential location identified; and ii) structural defects that may lead to collapse, including, cracks, fractures, breaks, holes, deformations, and collapses, are the other. While some information regarding defect severity is not considered, this approach is undertaken in order to align the defect codes of the two condition assessment standards, to reduce data thinness, and to account for the uncertainty resulting from subjectivity of inspector’s interpretations. The target variables of the binary classification ML-based models are whether or not either infiltration or structural defects are observed in the inspection. Different ML-based models are developed that relate the values of model predictor variables to one or the other of these model target variables, thus, there are two sets of binary classification ML-based models for each utility. The model sets are comprised of models that are based on one of four learning algorithms, one of three methods for addressing imbalanced data, and one of the two target variables. 4.2.3 Model Development The model predictor and target variable values for each of the pipes viewed in the inspection reports (799+2839=3638 and 1376, for Utilities A and B, respectively) were used to develop the 54 ML-based models, where for each model, 80% of the data sample are used to train and tune the respective algorithm and 20% are used to test it and conduct model comparisons. The pipe data are stratified split to preserve the distributions of the target classes in both the training and the testing datasets as shown in Table 8. The name of each model is denoted by utility name and one of two target classes, with “I” representing infiltration and “SD” representing structural defects. Table 8. Number of training and testing data Classification Model No. of Training Data (Target Class to No Target Class Ratio) No. of Testing Data (Target Class to No Target Class Ratio) Utility A-I 2910 (433:2477) 728 (108:620) Utility A-SD 2910 (860:2050) 728 (215:513) Utility B-I 1469 (430:1039) 368 (108:260) Utility B-SD 1469 (297:1172) 368 (74:294) Note: I=Infiltration; SD=Structural Defects. 4.2.3.1 Ensemble Learning Decision Tree Algorithms All four decision tree algorithms use a form of ensemble method to combine results from a collection of decision trees to determine the final model prediction. The RF algorithm uses a bagging method in which multiple decision trees are built in parallel based on a random subset of the data, which is obtained by sampling with replacement from the training data, and the predictions of these trees are aggregated to provide the trained RF-based model. The XGBoost, LightGBM and CatBoost algorithms use a gradient boosting method in which multiple decision trees are built in sequence and improve upon each other, all using the entire dataset. The other differences among these algorithms relate to: their method for guiding the tree branching process; i.e., how the data are split and evaluated conditioned on the values of the model predictor variables; their computational efficiencies’ the degree to which they overfit the data’ and their approach for supporting categorical data types. 55 A key factor that contributes to the efficiency of RF is its branching process which is guided by evaluating the information gain, i.e., prediction improvement, for any subset of model predictor variables and their values, measured with the Gini Index or Entropy. Its bagging ensemble method is known to be more robust than boosting against overfitting and noisy data (Fawagreh et al. 2014). It also requires a smaller number of hyperparameters to be tuned than the other algorithms XGBoost achieves increased computational efficiency by pre-sorting the data in the order of the model predictor variable values, before undergoing the tree branching process (Chen and Guestrin 2016). Thereafter, each possible of condition split is pursued and the information gain of predictions from each branch is based on the gradient and Hessian of the loss function, e.g., for binary classification models, a binary log loss function is used. However, compared with RF, XGBoost is more susceptible to overfitting if the hyperparameters are not properly turned. LightGBM realizes even more computational efficiency by applying Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) (Ke et al. 2017). To reduce the training dataset, GOSS retains all data points with large loss function gradients, and randomly selects and retains a sample of those with small loss function gradients. To reduce the model predictor variable set, EFB evaluates whether any of the variables are mutually exclusive, i.e., a variable is mutually exclusive from other variables, if when the variable is non-zero, the other variables have a value of zero. Unlike the other three algorithms that use level-wise growth (Korf 1985), where all immediately subsequent nodes (so called leaf nodes) at a given branch depth are split and evaluated before proceeding to the next branch depth, LigthGBM uses leaf-wise growth (Friedman et al. 2000), where the leaf node corresponding the highest improvement is pursued first, resulting in smaller 56 and faster models than the other three algorithms. However, this may lead to overfitting when the dataset is small if the maximum depth of a tree is not sufficiently tuned. While the aforementioned three algorithms are asymmetric in structure, CatBoost builds symmetric decision trees in which the predictor variable and variable value-pair resulting in the smallest loss function is applied to all the leaf nodes at given depth. This helps control overfitting and reduce prediction time. In addition, CatBoost further reduces its potential for overfitting by using ordered boosting. Unlike the classic boosting method applied in XGBoost and LightGBM, that uses the same data points to train the model and calculate the model fitting residuals, ordered boosting uses one subset of data to train and a different subset to calculate the residuals (Prokhorenkova et al. 2019). For all ML-based models, two model predictor variables are categorical, as listed in Table 2. At the time of the study, in the non-experimental version of RF and XGBoost algorithms used, a preprocessing step to convert categorical variables to numerical variables is required. One common process of conversion, known as One-Hot Encoding, involves creating a variable for each possible category of the variable, where a binary assignment is used, i.e., 1 or 0, indicating whether or not that category describes the categorical variable. In contrast, both LightGBM and CatBoost can support categorical variables internally without data preprocessing. The sklearn.ensemble.RandomForestClassifier API in Python, and the xgboost, lightgbm, and catboost Python packages, are used to implement RF, XGBoost, LightGBM, and CatBoost algorithms, respectively. To address categorical variables, One-Hot Encoding is implemented for the RF- or XGBoost-based models. 57 4.2.3.2 Methods for Addressing Imbalanced Data For binary classification ML-based models, non-uniformly distributed target variable values lead to model prediction bias because the model tends to minimize the loss function by focusing on the majority class and overlooking the minority class. The datasets for Utilities A and B are moderately imbalanced, with only 15% and 30% of pipe samples experiencing infiltration, and 30% and 20% of pipe samples experiencing structural defects, respectively. For these data, the majority class is the set of data for which there is no evidence of the given defect and the minority class is the set of data for which there is evidence of the defect. Three methods for addressing such imbalanced data are applied: i) the CSL approach; ii) the Synthetic Minority Oversampling Technique (SMOTE) (Bowyer et al. 2011) which may be applied for models that are strictly numerical, e.g., RF and XGBoost, or its extension, the Synthetic Minority Oversampling Technique for Nominal and Continuous features (SMOTE-NC), which may be applied for models that accommodate categorical and numerical data, e.g, LightGBM and CatBoost; and iii) Random Undersampling (RUS) (see also, Fernández et al. 2018 an extensive discussion of these and other methods for addressing imbalanced data). The objective of the CSL approach is to train a model to place a heavier weight on the class that has a higher cost in terms of the loss function, i.e., for binary models a higher weight is placed on false predictions of the minority class, than on false predictions of the majority class. For the algorithms applied herein, these weights are considered hyperparameters and the weight of the minority class (i.e., the presence of a given defect) relative to that of the majority class is the ratio of the number of majority data points and the number of minority data points. SMOTE, SMOTENC, and RUS require modifying training data to balance the class distribution prior to training and tuning the model. SMOTE and SMOTENC are oversampling methods in 58 which the minority class size is made equal to the majority class size by creating new minority class data points, through interpolation of data points between near neighbors. RUS is an undersampling technique in which some majority class data points are randomly eliminated to balance the size of the classes. While the dataset may be considered to be small, we explore RUS herein for completeness. The imblearn.over_sampling.SMOTE and imblearn.under_sampling.RandomUnderSampler API in Python are used to balance the class distribution. 4.2.3.3 Model Training and Tuning A systematic grid search procedure is employed for selecting combinations of hyperparameters and their values that are evaluated in terms of the resulting model validation score. For every model, training and hyperparameter tuning is undertaken with a repeated stratified k-fold cross-validation process, where the training dataset is partitioned into four subsets (k=4) and the training-tuning process is repeated five times, for a total of 20 model evaluations for each hyperparameter-value combination. The average of these 20 model evaluations is the validation score. Table 9 shows the list of hyperparameters used in each algorithm, indicated by the tick mark. The range of hyperparameter-value combinations for each of the models are provided in Table 12 in the Appendix B. The GridSearchCV and RepeatedStratifiedKFold from sklearn.model_selection module in Python are utilized to for grid search cross validation and the repeated 4-fold cross validation, respectively, in the optimal parameter selection process. Furthermore, the grid search cross validation algorithm is configured to run in parallell utilizing ‘n’ number of cores in the system, where ‘n’=16, thus resulting in a nearly 16-fold reduction in processing time. The computational savings is verified on a desktop computer with 13th Gen Intel(R) Core(TM) i7-13700KF CPU. 59 Table 9. List of hyperparameters Parameters Description RF XGBoost LightGBM CatBoost n_estimators/ iterations number of decision trees ✓ ✓ ✓ ✓ max_features maximum subsets of feature ✓ - - - class_weight/ scale_pos_weight weight assigned to majority and minority classes or majority class to minority class ratio ✓ ✓ ✓ ✓ criterion metric for impurity ✓ - - - max_depth/ depth maximum depth of a tree - ✓ ✓ ✓ learning_rate boosting learning rate - ✓ ✓ ✓ lambda/ lambda_l2/ l2_leaf_reg L2 regularization, prevent over fitting - ✓ ✓ ✓ num_leaves no. of leaves per tree - - ✓ - 4.2.4 Model Evaluation Metrics The overall performance of each classification model is evaluated using the areas under the Receiver Operating Characteristics (ROC) curve, denoted as AUC-ROC, and the Precision-Recall (PR) curve, denoted as AUC-PR. The ROC curve is considered to be robust to changes in the class distribution (Fawcett 2006), while the PR better represents the performance of models of imbalanced data (Davis and Goadrich 2006). The ROC curve shows the trade-off between the true positive rate (TPR) and false positive rate (FPR) at various probability thresholds. The probability threshold is the minimum likelihood level at which an identified defect is considered to be present. TPR [Eq. (4)] is the portion of pipes with a given defect, in which the given defect is correctly identified, and the FPR [Eq. (5)] is the portion of pipes that do not exhibit the defect, incorrectly being identified as having the defect. These are defined as: 𝑇𝑃𝑅 = 𝑇𝑃𝑇𝑃+𝐹𝑁 (4) 60 𝐹𝑃𝑅 = 𝐹𝑃𝐹𝑃+𝑇𝑁 (5) where TP = true positive, i.e., the number of pipes with a given defect that are correctly identified as having the defect; FN = false negative, i.e., the number of pipes with a given defect that are incorrectly classified as not having the defect; TN = true negative, i.e., the number of pipes that do not have a given defect and are correctly identified as not having the defect; FP = false positive, i.e., the number of pipes that do not have given defect that are incorrectly classified as having the defect. The PR curve represents a trade-off between precision and recall, at various probability thresholds. Recall is equivalent to the TPR [Eq. (4)]. Precision [Eq. (6)] is the portion of pipes that are correctly classified as having a given defect, among the total number of pipes identified as having the given defect. Precision is defined as: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃𝑇𝑃+𝐹𝑃 (6) The magnitude of AUC-ROC or AUC-PR may be used to compare different classification models and to evaluate whether any model performs better than a random classification. The ROC curve for random classification is a diagonal line between point (0,0) and point (1,1) i.e., the random classification baseline, where its area is 0.5. Thus, any models with an AUC-ROC better than 0.5 would perform better than a random classification. The PR curve for random classification is represented by the precision level at the point where recall is one, i.e., where all samples are classified has having defects. The random classification baseline, and the area under the baseline, for the PR curve, varies with the distribution of the minority and majority classes (Saito and Rehmsmeier 2015). 61 Fawcett (2006) asserts that in order to compare performance of any two models, using the ROC curve, multiple realizations of the data should be evaluated to consider the variance about the curves, to determine whether the models are indeed different. For this study, the relative model performance is evaluated based on twenty sets of testing data for each model, generated by executing a repeated stratified k-fold cross validation on the original testing data. To identify the mean and variance of the ROC curve for each model, a threshold averaging method is used, where TPR and FPR points are averaged, for a set of probability thresholds (see Fawcett 2006). The Mann-Whitney U test, a non-parametric statistical test, is used to compare any two sets of AUC-ROC generated for any two classification models. The distribution of AUC-ROC between the two models are found to be different if the p-value, a probability value that describes how likely they are to be the same, is less than or equal to a specified significance level, e.g., 0.05 in this study. If the models are considered to be different the U statistic, which indicates the general magnitude of the difference between two ROC curves, may be used to determine the relative performance of the models, i.e., the model with the larger U statistic is considered to be the better model. The same process is applied in this work, when comparing the AUC-PR between any two models. ROC curves and PR curves are computed using roc_curve and precision_recall_curve functions from the sklearn.metrics module in Python. AUC-ROC and AUC-PR are computed using auc functions from the same module. The Mann-Whitney U test is computed using mannwhitneyu from scipy.stats module in Python. 4.2.5 Shapley Additive exPlanations (SHAP) Analysis and Partial Dependence Plots SHAP, a class additive feature attribution method based on the coalitional game theory, is used to determine the model variable importance. SHAP is applied on the entire dataset (i.e., training 62 and testing) to estimate the degree to which each datapoint contributes to the model target variable value, where positive SHAP values have a positive influence on the value. SHAP dependence plots display the relationship between the model predictor variable values and the SHAP values. Partial dependence plots (PDP) are used to show the average effect of the locational coordinates, two model predictor variables, on the model target variable, to provide the likelihood of the given defect occurring at locations within the utility (Friedman 2001). The Python package shap and the sklearn.inspection.PartialDependenceDisplay API in Python are used for this analysis. 4.3 Results and Discussion 4.3.1 Classification Model Comparison For each algorithm, method for addressing imbalanced data, utility and target variable, the mean and standard deviation of the AUC-ROC and AUC-PR are reported in Table 13 – 14 in the Appendix B (48 models in total for each performance measure). Also provided are the p-values and U-statistics, as well as the rank ordered performance based on these, for cases where a rank order may be analyzed. When a rank or may be determined, i.e., cases for which some models are significantly different (highlighted in yellow in Table 13 – 14 in the Appendix B), evaluations in terms of AUC-ROC show that models based on the CSL approach are not significantly different from those based on SMOTE or SMOTE-NC, however, they both outperform RUS, and the latter generally outperforms the former in terms of their relative superiority to RUS. Evaluations in terms of AUC-PR indicate that CSL and SMOTE or SMOTE-NC approaches perform equally well, but outperform RUS. 63 For a general comparison of the algorithms, the ML-based models with the highest means of average AUC-ROC and AUC-PR based on 20 model executions and the corresponding standard deviations are summarized in Table 10. Regarding the AUC-ROC, for both utilities RF-based models perform well for classifying infiltration and LightGBM-based models perform well for classifying structural defects. Regarding the AUC-PR, for Utility A, RF- and CatBoost-based models have the highest mean AUC-PR when classifying infiltration and structural defects, respectively. For Utility B, XGBoost- and LightGBM-based models have the highest mean AUC-PR when classifying infiltration and structural defects, respectively. However, among the models provided in Table 10, only two of the models for classifying structural defects for Utility B are found to be statistically different from one another (i.e., the RF- and LightGBM-based models). Thus, only Utility B structural defects classifying models may be ranked relative to one another. Figure 11 provides the ROC and PR curves for these models. The curves for all other models are provided in Figure 21 – 23 in the Appendix B. 64 Table 10. Models with the Highest Mean AUC-ROC and AUC-PR for each algorithm Classification Model Highest Mean AUC-ROC ± 95% CIa Highest Mean AUC-PR ± 95% CIb RF XG-Boost Light-GBM Cat-Boost RF XG-Boost Light-GBM Cat-Boost Utility A-I 0.80 ± 0.06c 0.77 ± 0.06d 0.77 ± 0.05c 0.75 ± 0.07c 0.41 ± 0.14d 0.40 ± 0.15d 0.39 ± 0.13c 0.38 ± 0.17d Utility A-SD 0.71 ± 0.07d 0.71 ± 0.07d 0.72 ± 0.07d 0.71 ± 0.06c 0.50 ± 0.12b 0.49 ± 0.10c 0.50 ± 0.11c 0.51 ± 0.11d Utility B-I 0.82 ± 0.07c 0.81 ± 0.07c 0.81 ± 0.06c 0.80 ± 0.07c 0.61 ± 0.11d 0.63 ± 0.13d 0.62 ± 0.12d 0.63 ± 0.13d Utility B-SD 0.71 ± 0.10c 0.72 ± 0.10d 0.73 ± 0.10d 0.71 ± 0.10d 0.45 ± 0.16d 0.50 ± 0.17d 0.51 ± 0.16d 0.48 ± 0.15d aMean AUC-ROC and corresponding standard deviation for a 95% confidence interval, for 20 model evaluations. bMean AUC-PR and corresponding standard deviation for a 95% confidence interval, for 20 model evaluations. cThe model trained with SMOTE in the case of RF and XGBoost or SMOTE-NC in the case of LightGBM and CatBoost, to address imbalanced data. dThe model trained with CSL approach, to address imbalanced data. Figure 11. a) ROC and b) PR curves for models with the highest means for classifying structural defects for Utility B The curves in Figure 11 are close and cross one another several times, however, based on the Mann-Whitney U tests, those for RF and LightGBM are significantly different from one another. b) a) 65 While it appears that LightGBM has the highest overall mean AUC-ROC and AUC-PR, depending on the preferred level of precision desired, different models, and their respective probability threshold levels at the precision value may be preferred. The points on the graph correspond to the FPR and TPR on the ROC curves or recall and precision values on the PR curves that are the nearest distance to the ideal classification model points, i.e., point (0,1) and point (1,1), for ROC and PR curves, respectively. For the given model, these points correspond to the best probability threshold, a probability threshold that will yield the best trade-off between the two values, i.e., FPR and TPR for ROC curves and recall and precision for PR curves. For each algorithm, the best probability threshold for the ROC curve is different from that for the PR curve. The choice of which threshold is preferred depends on the importance placed on the minority class, e.g., if one were to place a high penalty on failing to identify the occurrence of a defect, the PR-based point is important. For a visual representation of the influence of class imbalance, the performance of LightGBM with the CSL approach for each of the different Utility-target variable combinations are compared in Figure 12. Comparing the ROC and PR curves for these models with the random classification baseline indicates that all models perform better than the random classification. It is important to note that the AUC-ROC value for random classification is 0.5, and the AUC-PR value for random classification is based on the distribution of the minority and majority classes. The effect of imbalanced data on the model performance is more prominent on the PR curves. The infiltration classification model for Utility A, trained with the most imbalanced data (approximately 15% minority class), has the lowest AUC-PR of all four algorithm-based models. However, this still has an AUC of 0.35 that is 2.33 times larger than that for random classification which has an AUC of 0.15. The structural defects classification model for Utility A 66 and infiltration classification models for Utility B have the lowest class imbalance (approximately 30% minority class). However, the former model has the second lowest AUC-PR due to its poor early retrieval, i.e., early severely low precision. The model has an AUC-PR that is 1.63 times larger than that for random classification. In contrast, the latter model has the best overall PR curve and AUC-PR among all models, and performs 2 times better than the random classification. The structural defects classification model for Utility B (approximately 20% minority class) structural shows good early retrieval, which has a high precision when the recall is small. The model performs 2.55 times better than the random classification. Figure 12. a) ROC and b) PR curves of the four classification models using LightGBM with the CSL approach 4.3.2 Feature Importance and Partial Dependence Plot of the Best Models To evaluate each model predictor variable in terms of its importance for each classification model, a SHAP summary plot based on the SHAP values for each model was drawn. The SHAP values corresponding to the model prediction variables of the x- and y-coordinates of the pipe a) b) 67 centroid were grouped together to see the overall effect of pipe location on the model prediction. The SHAP values can only be used to determine the contribution of the model predictor variables to the target variable relative to other model predictor variables in the same model. Figure 13 and Figure 14 show the distribution of all data points in the training and testing data and their corresponding SHAP values for each model predictor variable of LightGBM-based models with the CSL approach for infiltration and structural defects classification models for Utilities A and B, respectively. The order of model predictor variables presented in the figures corresponds to their contributions to the model prediction. The rank ordered model predictor variables based on the SHAP summary plots of other models can be found in Table 15 in the Appendix B. According to Figure 13, the pipe location and age are the top two model predictor variables followed by the pipe length and material for both infiltration and structural defects classification models for Utility A. The model prediction does not depend on pipe shape as all pipes have circular shapes. According to Figure 14, the pipe age and size are the top two model predictor variables, followed by the pipe location, length, and material for the infiltration classification model for Utility B. For the structural defect classification model, pipe age and length are the top two model predictor variables, followed by the pipe location, size, and material. The pipe shape does not have large impact on the model prediction for both models. 68 Figure 13. Model predictor variable importance for a) infiltration and b) structural defects classification models for Utility A Figure 14. Model predictor variable importance for a) infiltration and b) structural defects classification models for Utility B Even though different models have different rankings of model predictor variables, the SHAP dependence plots from each model are similar. All figures from now on are the results of LightGBM-based models with the CSL approach for addressing class imbalance. Figure 15 and Figure 16 are SHAP dependence plots of pipe material, age, size, and length for infiltration and structural defects, respectively, for Utility A. Similarly, Figure 17 and Figure 18 show the relationship between these model predictor variables and the infiltration and structural defects, respectively, for Utility B. The data points in the figures are coloured according to the pipe ages to observe the influence of the pipe age, one of the top two model predictor variables, on the model target variable, and on each model predictor variable’s SHAP value. The SHAP dependence plots of the pipe shape are omitted because it has little to no influence on the model prediction as shown in Figure 13 and Figure 14. Lastly, PDP are drawn as shown in Figure 19 and Figure 20 to show the average effect of the locational coordinates (i.e., x- and y- coordinates a) b) b) a) 69 variables) on the predictions of infiltration and structural defects for Utilities A and B, respectively. The SHAP dependence plot of the pipe age in Figure 15a suggests that the infiltration is more likely to happen from Year 0 to Year 30, as indicated by the positive SHAP values. The SHAP value shows a varied trend from Year 30 to Year 70, where an upward trend indicates that the chance of infiltration increases within the given age range and a downward trend indicates a decrease in infiltration within the given age range. Figure 15b shows that concrete (CP), and vitrified clay (VCP) pipes have positive SHAP values which indicate their positive effect on infiltration for Utility A and asbestos cement (AC) and polyvinyl Chloride (PVC) pipes have negative effect on infiltration, as indicated by negative SHAP values. In Figure 15c, the majority of points have positive SHAP values as pipe size increases indicating that larger pipes have more influence on infiltration than smaller pipes with the exception of pipes in the 500 mm range. The relationship between the infiltration and the pipe length is positive-linear. The longer the pipe is, the more likely the infiltration to be found. It should be noted that one may not find this relationship in other utilities. For example, if a certain material is likely to experience more infiltration and if most short pipes are made from that material, then one would see an inverse relationship with length. 70 Figure 15. SHAP dependence plots for Utility A infiltration classification model for: a) pipe age; b) material; c) size; and d) length Figure 16. SHAP dependence plots for Utility A structural defects classification model for: a) pipe age; b) material; c) size; and d) length a)c)b) d). a). c). b) d). 71 Figure 16a shows that SHAP values increase from Year 0 to Year 20, decrease from Year 20 to Year 40, and increase from Year 40 to Year 70. These trends indicate that structural defects are more likely to happen during the first 20 years of pipe life and after 40 years for Utility A. Similar to infiltration, Figure 16b shows that AC and PVC pipes have lesser contribution to structural defects than CP and VCP pipes. Figure 16c shows that the majority of points have negative SHAP values as pipe size increases indicating that structural defects are less likely to occur in larger pipes than smaller pipes. The opposite relationship is true for the pipe length and structural defects. Figure 17a and d show that SHAP values increase as pipe age and length increase indicating that older and longer pipes are more likely to have infiltration than newer and shorter pipes for Utility B. Figure 17c shows that SHAP values have an upward trend for pipe sizes increasing from 200 mm to 1700 mm and the downward trend from 1700 to 3000 mm. However, most pipes from 1000 mm onwards have positive SHAP values indicating they are likely to have infiltration. Figure 17b shows that AC, CP, PVC, and VCP pipes have a negative effect on infiltration, while reinforced concrete (RCP) and other material (ZZZ) pipes have a positive effect on infiltration. There is only a single data point for polyethylene (PE) and three data points for steel (SP) pipes. Thus, they are not used in the analysis. In addition, most of PVC pipes are less than 40 years old indicating that the pipe age may influence the PVC pipe SHAP values. However, this does not apply for other types of pipe material. 72 Figure 17. SHAP dependence plots for Utility B infiltration classification model for: a) pipe age; b) material; c) size; and d) length Figure 18. SHAP dependence plots for Utility B structural defects classification model for: a) pipe age; b) material; c) size; and d) length d). c). a). b). c). a). d). b). 73 The upward trend of SHAP values as the variable values increase is observed for the pipe age, size, and length for structural defects for Utility B, as shown in Figure 18a, Figure 18c, and Figure 18d. In addition, AC, CP, and PVC pipes have a lower likelihood to have structural defects, while RCP, VCP, and other pipes have a positive effect on structural defects as shown in Figure 18b. Lastly, for each classification model, there is no clear indication that shows that pipe age influences the other three model predictor variable SHAP values, as the colour distribution of the data points does not follow the pattern of the pipe age SHAP values. Figure 19 shows the likelihood that infiltration is found in a given area, denoted by numbers, where the darker the color the lower the likelihood. Here, infiltration is likely to be found in pipes located on the western side of Utility A, while structural defects are likely to be found in pipes located on the three-fourths upper portion of the western side. Figure 20 shows that one cannot divide Utility B into distinct infiltration and structural area clusters. Figure 19. Partial dependence plot between x- and y- coordinates, likelihood of a) infiltration and b) structural defects in Utility A where lighter colors represent a larger likelihood while darker colors suggest a lower likelihood a) b) 74 Figure 20. Partial dependence plot for x- and y- coordinates, likelihood of a) infiltration and b) structural defects for Utility B where lighter colors represent a larger likelihood while darker colors suggest a lower likelihood 4.4 Conclusions and Recommendations This chapter presented models that target the specific defects concerned by the utility as an alternative to a conventional sewer deterioration model. The results from the models can assist with the pipe inspection scheduling by generating the inventory of pipes that are likely to have specific defects, as well as identifying the areas within the network where these specific defects are more likely to occur and should be scheduled for inspection. The models are capable of accommodating data from various condition assessment standards, as the classification procedure requires only the grouping of defects with similar characteristics rather than pipe condition scores that could have different ranking criteria and scaling methods for different condition assessment standards. The models are trained and tested on data from two utilities to identify infiltration and structural defects within their network. Four ensemble learning decision tree-based algorithms and three methods for managing imbalanced data are explored. AUC-ROC and AUC-PR were used to evaluate the performance of each model and indicate that there is no statistical difference b) a) 75 between most models, except for one case where a LightGBM-based model outperforms an RF-based model. In most cases, models that employ CSL and SMOTE methods outperform models that apply RUS. One drawback of RUS for this study is that it decreases the data size, and RUS might be more suitable for a larger dataset. The performance of the models is affected by imbalanced data and such data-based models can be evaluated via PR curves. All novel classification models are proven to perform up to 2.5 times better than the random classification. Additionally, the SHAP dependence plots show PVC pipes have the less influence on the infiltration and structural defects for both utilities in comparison with other pipe materials. Therefore, it is recommended that both utilities consider using PVC pipes for future installation. In the future, environmental model predictor variables, such as soil conditions and surface loads, as well as road types and proximity to critical infrastructure, could be integrated as model predictor variables to relate these to sewer defects, and to improve model performance. Models based on these data can be further incorporated in LOF and COF analyses, and used to undertake the asset management of the network. 76 Chapter 5: Conclusions As a part of the ongoing discussion of methods for automating the CCTV inspection process, the study examines three CNN architectures and conducts investigations of their performance on three distinct utility data sets. The study develops modelling frameworks that show great potential for implementing a CCTV QA/QC assessment tool. However, further enhancement is required in order to effectively utilize these for identifying defects in unexamined CCTV inspection videos. This study also undertakes to help utilities undergo condition assessment which is one of the main components of asset management. The focus is on developing models that can be used to identify the presence of defects of concern within the sewer system as well as areas within the network where these defects are more likely to occur. The models have the ability to incorporate data from different condition assessment standards, which is beneficial when utilities are transitioning from one coding standard to another. The study has demonstrated how ML can be utilized to enable utilities to be proactive in their asset management. Future research should focus on advancing the application of automated sewer defect classification models for real-time CCTV inspection processes and demonstrating their use in supporting live inspections. Additional data such as soil conditions, road types, and proximity to critical infrastructure can be collected for COF analysis and included as predictor variables to further improve sewer defect prediction. The results from the COF analysis and models can be incorporated to determine the priority for pipe maintenance and replacement. 77 References ASCE (2021). "2021 Report Card for America's Infrastructure." <https://infrastructurereportcard.org/cat-item/wastewater-infrastructure/>. Bay, H., Ess, A., Tuytelaars, T., and Van Gool, L. (2008). "Speeded-Up Robust Features (SURF): Similarity Matching in Computer Vision and Multimedia." Computer vision and image understanding, 110(3), 346-359. Bowyer, K. W., Chawla, N. V., Hall, L. O., and Kegelmeyer, W. P. (2011). "SMOTE: Synthetic Minority Over-sampling Technique." CoRR, abs/1106.1813. Breiman, L. (2001). "Random Forests." Machine Learning, 45(1), 5-32. Caradot, N., Riechel, M., Fesneau, M., Hernandez, N., Torres, A., Sonnenberg, H., Eckert, E., Lengemann, N., Waschnewski, J., and Rouault, P. (2018). "Practical benchmarking of statistical and machine learning models for predicting the condition of sewer pipes in Berlin, Germany." Journal of Hydroinformatics, 20(5), 1131-1147. Caradot, N., Sonnenberg, H., Kropp, I., Ringe, A., Denhez, S., Hartmann, A., and Rouault, P. (2017). "The relevance of sewer deterioration modelling to support asset management strategies." Urban Water Journal, 14(10), 1007-1015. Chen, K., Hu, H., Chen, C., Chen, L., and He, C. "An Intelligent Sewer Defect Detection Method Based on Convolutional Neural Network." IEEE, 1301-1306. Chen, T., and Guestrin, C. (2016). "XGBoost: A Scalable Tree Boosting System." CoRR, abs/1603.02754. Chen, Z.-M., Wei, X.-S., Wang, P., and Guo, Y. "Multi-Label Image Recognition With Graph Convolutional Networks." IEEE, 5172-5181. Cheng, J. C. P., and Wang, M. (2018). "Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques." Automation in Construction, 95, 155-171. Dalal, N., and Triggs, B. "Histograms of oriented gradients for human detection." IEEE, 886-893 vol. 881. Davis, J., and Goadrich, M. (2006). "The relationship between Precision-Recall and ROC curves." Proceedings of the 23rd international conference on Machine learning, Association for Computing Machinery, Pittsburgh, Pennsylvania, USA, 233–240. Delleur, J. W., Reyna, S., and Vanegas, J. (1998). "Sewerage Rehabilitation." Hydroinformatics Tools for Planning, Design, Operation and Rehabilitation of Sewer Systems, J. Marsalek, C. Maksimovic, E. Zeman, and R. Price, eds., Springer Netherlands, Dordrecht, 415-450. Egger, C., Scheidegger, A., Reichert, P., and Maurer, M. (2013). "Sewer deterioration modeling with condition data lacking historical records." Water Research, 47(17), 6762-6779. 78 Elmasry, M., Hawari, A., and Zayed, T. (2017). "Defect based deterioration model for sewer pipelines using Bayesian belief networks." Canadian Journal of Civil Engineering, 44(9), 675-690. Fawagreh, K., Gaber, M. M., and Elyan, E. (2014). "Random forests: from early developments to recent advancements." Systems Science & Control Engineering, 2(1), 602-609. Fawcett, T. (2006). "An introduction to ROC analysis." Pattern Recognition Letters, 27(8), 861-874. Fernández, A., García, S., Galar, M., Prati, R. C., Krawczyk, B., and Herrera, F. (2018). "Learning from Imbalanced Data Sets." Springer Nature Switzerland AG. Friedman, J., Hastie, T., and Tibshirani, R. (2000). "Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)." The Annals of Statistics, 28(2), 337-407. Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine." The Annals of Statistics, 29(5), 1189-1232. Gurjar, A., and Voditel, P. (2023). "Incorporating Transfer Learning in CNN Architecture." International journal of next-generation computing, 14(1). Halfawy, M. R., and Hengmeechai, J. (2014). "Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine." Automation in Construction, 38, 1-13. Harvey, R. R., and McBean, E. A. (2014a). "Comparing the utility of decision trees and support vector machines when planning inspections of linear sewer infrastructure." Journal of Hydroinformatics, 16(6), 1265-1279. Harvey, R. R., and McBean, E. A. (2014b). "Predicting the structural condition of individual sanitary sewer pipes with random forests." Canadian Journal of Civil Engineering, 41(4), 294-303. Hassan, S. I., Dang, L. M., Mehmood, I., Im, S., Choi, C., Kang, J., Park, Y.-S., and Moon, H. (2019). "Underground sewer pipe condition assessment based on convolutional neural networks." Automation in construction, 106, 102849. Haurum, J. B., and Moeslund, T. B. (2021). "Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark." Cornell University Library, arXiv.org, Ithaca. Hawari, A., Alamin, M., Alkadour, F., Elmasry, M., and Zayed, T. (2018). "Automated defect detection tool for closed circuit television (cctv) inspected sewer pipelines." Automation in Construction, 89, 99-109. He, K., Zhang, X., Ren, S., and Sun, J. "Deep Residual Learning for Image Recognition." IEEE, 770-778. Hernández, N., Caradot, N., Sonnenberg, H., Rouault, P., and Torres, A. (2018). "Support tools to predict the critical structural condition of uninspected pipes for case studies of Germany and Colombia." Water Practice and Technology, 13(4), 794-802. 79 Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., Adam, H., and Le, Q. "Searching for MobileNetV3." IEEE, 1314-1324. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Hartwig, A. (2017). "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications." Cornell University Library, arXiv.org, Ithaca. Howard, J., and others (2018). "fastai." <https://github.com/fastai/fastai>. Ke, G., Meng, Q., Finely, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. "LightGBM: A Highly Efficient Gradient Boosting Decision Tree." Khan, Z., Zayed, T., and Moselhi, O. (2010). "Structural Condition Assessment of Sewer Pipelines." Journal of Performance of Constructed Facilities, 24(2), 170-179. Kleiner, Y., and Rajani, B. (2001). "Comprehensive review of structural deterioration of water mains: statistical models." Urban Water, 3(3), 131-150. Kley, G., and Caradot, N. (2013 ). "Review of sewer deterioration models." Kompetenzzentrum Wasser Berlin gGmbH. Kley, G., Kropp, I., Schmidt, T., and Caradot, N. (2013). "Review of available technologies and methodologies for sewer condition evaluation." Kompetenzzentrum Wasser Berlin gGmbH. Korf, R. E. (1985). "Depth-first iterative-deepening: An optimal admissible tree search." Artificial Intelligence, 27(1), 97-109. Krizhevsky, A., Sutskever, I., and Hinton, G. E. "ImageNet Classification with Deep Convolutional Neural Networks." Curran Associates, Inc. Kumar, S. S., Abraham, D. M., Jahanshahi, M. R., Iseley, T., and Starr, J. (2018). "Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks." Automation in Construction, 91, 273-283. Kumar, S. S., Wang, M., Abraham, D. M., Jahanshahi, M. R., Iseley, T., and Cheng, J. C. P. (2020). "Deep Learning–Based Automated Detection of Sewer Defects in CCTV Videos." Journal of computing in civil engineering, 34(1). Laakso, T., Kokkonen, T., Mellin, I., and Vahala, R. (2018). "Sewer Condition Prediction and Analysis of Explanatory Factors." Water, 10(9), 1239. Le Gat, Y. (2008). "Modelling the deterioration process of drainage pipelines." Urban Water Journal, 5(2), 97-106. Li, D., Cong, A., and Guo, S. (2019). "Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification." Automation in Construction, 101, 199-208. Li, D., Xie, Q., Yu, Z., Wu, Q., Zhou, J., and Wang, J. (2021). "Sewer pipe defect detection via deep learning with local and global feature fusion." Automation in Construction, 129, 103823. 80 Li, J., Li, X., Liu, H., Gao, L., Wang, W., Wang, Z., Zhou, T., and Wang, Q. (2023). "Climate change impacts on wastewater infrastructure: A systematic review and typological adaptation strategy." Water Research, 242, 120282. Lundberg, S. M., and Lee, S.-I. (2017). "A unified approach to interpreting model predictions." CoRR, abs/1705.07874. Malek Mohammadi, M., Najafi, M., Kaushal, V., Serajiantehrani, R., Salehabadi, N., and Ashoori, T. (2019). "Sewer Pipes Condition Prediction Models: A State-of-the-Art Review." Infrastructures, 4(4), 64. Malek Mohammadi, M., Najafi, M., Kermanshachi, S., Kaushal, V., and Serajiantehrani, R. (2020). "Factors Influencing the Condition of Sewer Pipes: State-of-the-Art Review." Journal of Pipeline Systems Engineering and Practice, 11(4), 03120002. Marsalek, J., and Schilling, W. (1998). "Operation of Sewer Systems." Hydroinformatics Tools for Planning, Design, Operation and Rehabilitation of Sewer Systems, J. Marsalek, C. Maksimovic, E. Zeman, and R. Price, eds., Springer Netherlands, Dordrecht, 393-414. Mashford, J., Marlow, D., Tran, D., and May, R. (2011). "Prediction of Sewer Condition Grade Using Support Vector Machines." Journal of Computing in Civil Engineering, 25(4), 283-290. Meijer, D., Scholten, L., Clemens, F., and Knobbe, A. (2019). "A defect classification methodology for sewer image sets with convolutional neural networks." Automation in Construction, 104, 281-298. Moradi, S., Zayed, T., and Golkhoo, F. (2019). "Review on Computer Aided Sewer Pipeline Defect Detection and Condition Assessment." Infrastructures, 4(1). Myrans, J., Everson, R., and Kapelan, Z. (2019). "Automated detection of fault types in CCTV sewer surveys." Journal of Hydroinformatics, 21(1), 153-163. Najafi, M., and Kulandaivel, G. (2005). "Pipeline Condition Prediction Using Neural Network Models." Pipelines 2005, 767-781. NASSCO (2022). "NASSCO's pipeline assessment and certification program, Metric Edition, v7.0.4." National Association of Sewer Service Companies, ed.USA. O’Shea, K., and Nash, R. (2015). "An Introduction to Convolutional Neural Networks." ArXiv, abs/1511.08458. Oliva, A., and Torralba, A. (2001). "Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope." International Journal of Computer Vision, 42(3), 145-175. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. (2019). "CatBoost: unbiased boosting with categorical features." Rahman, S., and Vanier, D. J. (2004). "An evaluation of condition assessment protocols for sewer management." Client Report (National Research Council of Canada. Institute for Research in Construction); no. B-5123.6, National Research Council of Canada. 81 Ridnik, T., Lawen, H., Noy, A., Ben, E., Sharir, B. G., and Friedman, I. "TResNet: High Performance GPU-Dedicated Architecture." IEEE, 1399-1408. Rijsbergen, C. J. V. (1979). Information Retrieval, Butterworth-Heinemann. Rokstad, M. M., and Ugarelli, R. M. (2015). "Evaluating the role of deterioration models for condition assessment of sewers." Journal of Hydroinformatics, 17(5), 789-804. Saito, T., and Rehmsmeier, M. (2015). "The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets." PLoS One, 10(3), e0118432. Salihu, C., Mohandes, S. R., Kineber, A. F., Hosseini, M. R., Elghaish, F., and Zayed, T. (2023). "A Deterioration Model for Sewer Pipes Using CCTV and Artificial Intelligence." Buildings, 13(4), 952. Salman, B., and Salem, O. (2012). "Risk Assessment of Wastewater Collection Lines Using Failure Models and Criticality Ratings." Journal of Pipeline Systems Engineering and Practice, 3(3), 68-76. Sousa, V., Matos, J. P., and Matias, N. (2014). "Evaluation of artificial intelligence tool performance and uncertainty for predicting sewer structural condition." Automation in Construction, 44, 84-91. Statistics Canada (2022). "A culvert operation: Flushing out the data on Canada's storm water and wastewater infrastructure." <https://www.statcan.gc.ca/o1/en/plus/2162-culvert-operation-flushing-out-data-canadas-storm-water-and-wastewater-infrastructure>. Statistics Canada (2023). "Table 36-10-0611-01 Infrastructure Economic Accounts, average age and remaining useful service life ratio by asset and asset function." <https://doi.org/10.25318/3610061101-eng>. Su, T.-C., Yang, M.-D., Wu, T.-C., and Lin, J.-Y. (2011). "Morphological segmentation based on edge detection for sewer pipe defects on CCTV images." Expert Systems with Applications, 38(10), 13094-13114. Sun, L., Zhu, J., Tan, J., Li, X., Li, R., Deng, H., Zhang, X., Liu, B., and Zhu, X. (2023). "Deep learning-assisted automated sewage pipe defect detection for urban water environment management." Sci Total Environ, 882, 163562. Syachrani, S., Jeong, H. S. D., and Chung, C. S. (2013). "Decision Tree–Based Deterioration Model for Buried Wastewater Pipelines." Journal of Performance of Constructed Facilities, 27(5), 633-645. Tan, Y., Cai, R., Li, J., Chen, P., and Wang, M. (2021). "Automatic detection of sewer defects based on improved you only look once algorithm." Automation in Construction, 131, 103912. Tran, D. H., Ng, A. W. M., Perera, B. J. C., Burn, S., and Davis, P. (2006). "Application of probabilistic neural networks in modelling structural deterioration of stormwater pipes." Urban Water Journal, 3(3), 175-184. 82 Tscheikner-Gratl, F., Caradot, N., Cherqui, F., Leitão, J. P., Ahmadi, M., Langeveld, J. G., Le Gat, Y., Scholten, L., Roghani, B., Rodríguez, J. P., Lepot, M., Stegeman, B., Heinrichsen, A., Kropp, I., Kerres, K., Almeida, M. d. C., Bach, P. M., Moy de Vitry, M., Sá Marques, A., Simões, N. E., Rouault, P., Hernandez, N., Torres, A., Werey, C., Rulleau, B., and Clemens, F. (2020). "Sewer asset management – state of the art and research needs." Urban Water Journal, 16(9), 662-675. Vitorino, D., Coelho, S. T., Santos, P., Sheets, S., Jurkovac, B., and Amado, C. (2014). "A Random Forest Algorithm Applied to Condition-based Wastewater Deterioration Modeling and Forecasting." Procedia Engineering, 89, 401-410. Vladeanu, G. J., and Matthews, J. C. (2019). "Consequence-of-Failure Model for Risk-Based Asset Management of Wastewater Pipes Using AHP." Journal of Pipeline Systems Engineering and Practice, 10(2). Wang, M. Z., and Cheng, J. C. P. (2019). "Semantic Segmentation of Sewer Pipe Defects Using Deep Dilated Convolutional Neural Network." IAARC Publications, Waterloo, 586-594. Wang, X., Stewart, M. G., and Nguyen, M. (2012). "Impact of climate change on corrosion and damage to concrete infrastructure in Australia." Climatic Change, 110(3), 941-957. Wang, Y., He, D., Li, F., Long, X., Zhou, Z., Ma, J., and Wen, S. (2020). "Multi-Label Classification with Label Graph Superimposing." Proceedings of the . AAAI Conference on Artificial Intelligence, 34(7), 12265-12272. Wang, Y., Li, P., and Li, J. (2022). "The monitoring approaches and non-destructive testing technologies for sewer pipelines." Water science and technology, 85(10), 3107-3121. WRc (2001). "Sewerage Rehabilitation Manual (SRM), Fourth Edition." Water Research Centre, ed.UK. WRc (2004). "Manual of Sewer Condition Classification, Fourth Edition." Water Research Centre, ed.UK. Wu, B., Chen, W., Fan, Y., Zhang, Y., Hou, J., Liu, J., and Zhang, T. (2019). "Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning." IEEE access, 7, 172683-172693. Xie, Q., Li, D., Xu, J., Yu, Z., and Wang, J. (2019). "Automatic Detection and Classification of Sewer Defects via Hierarchical Deep Learning." IEEE transactions on automation science and engineering, 16(4), 1836-1847. Xu, K., Luxmoore, A. R., and Davies, T. (1998). "Sewer pipe deformation assessment by image analysis of video surveys." Pattern Recognition, 31(2), 169-180. Yang, M.-D., and Su, T.-C. (2009). "Segmenting ideal morphologies of sewer pipe defects on CCTV images for automated diagnosis." Expert Systems with Applications, 36(2, Part 2), 3562-3573. 83 Appendices Appendix A A.1 Computational Time Table 11. Average training and prediction time of different models Training Time Testing Time Type of Model Pre-trained Model Average Time per 10,000 images (sec)a Ratio to One-Stage ResNet50 Average Time per 10,000 images (sec)a Ratio to One-Stage ResNet50 One-Stage MobileNet V3 Lb 11 0.6 0.010 0.4 ResNet50 19 1.0 0.024 1.0 ResNet101 29 1.5 0.038 1.6 Two-Stage MobileNet V3 L 21 1.1 0.015 0.6 ResNet50 38 2.0 0.039 1.6 ResNet101 59 3.2 0.063 2.6 a Average time across three utilities models b L stands for Large 84 Appendix B B.1 Hyperparameter Tunning Table 12. Range of hyperparameter values used during cross-validation grid search Parameters RF XGBoost LightGBM CatBoost n_estimators/ iterations [50, 100, 150, 200] [50, 100, 150, 200] [50, 100, 150, 200] [50, 100, 150, 200] max_features [None, sqrt, log2] - - - class_weight/ scale_pos_weight [None, balanced] majority class to minority class ratio majority class to minority class ratio majority class to minority class ratio criterion [gini, entropy] - - - max_depth/ depth - [10, 30, 50, 80, 100] [10, 30, 50, 80, 100] [4, 6, 8, 10] learning_rate - [0.03, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3] [0.03, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3] [0.03, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3] lambda/ lambda_l2/ l2_leaf_reg - [0, 1, 3, 9] [0, 1, 3, 9] [0, 1, 3, 9] num_leaves - - [10, 30, 50, 80, 100, 200, 300] - B.2 Evaluate the performance of different models using Mann-Whitney U tests In Table 13 and Table 14, models that are considered to be significantly different from one another are highlighted in yellow. The distribution of AUC-ROC or AUC-PR between the two models are found to be different if the p-value, a probability value that describes how likely they are to be the same, is less than or equal to a specified significance level, e.g., 0.05 in this study. If the 85 models are considered to be different, the U statistic, which indicates the general magnitude of the difference between two ROC or PR curves, may be used to determine the relative performance of the models, i.e., the model with the larger U statistic is considered to be the better model. The models may be ranked relative to one another if at least one pair of models is found to be different. Table 13. The area under the average ROC curves of each classification model and its U statistics, p-value, and rank when comparing its performance with model with other imbalanced data managing techniques, as determined by the Mann-Whitney U test at significance level equals to 0.05 Utility Target Variable Algorithm Imbalanced data managing technique mean AUC two standard deviations U Statistics when comparing the model to p-values when comparing the model to Rank CSL RUS SMOTE/ SMOTENC CSL RUS SMOTE/ SMOTENC Utility A Infiltration RF CSL 0.80 0.06 - 270 198 - 0.06 0.97 2 RUS 0.77 0.08 130 - 118 0.06 - 0.03 3 SMOTE 0.80 0.06 202 282 - 0.97 0.03 - 1 XGBoost CSL 0.77 0.06 - 304 213 - 0.01 0.74 1 RUS 0.73 0.09 96 - 106 0.01 - 0.01 3 SMOTE 0.77 0.06 187 294 - 0.74 0.01 - 1 LightGBM CSL 0.75 0.06 - 226 132 - 0.49 0.07 2 RUS 0.74 0.09 174 - 104 0.49 - 0.01 3 SMOTE-NC 0.77 0.05 268 296 - 0.07 0.01 - 1 CatBoost CSL 0.75 0.09 - 205.5 169 - 0.89 0.41 - RUS 0.74 0.10 194.5 - 159 0.89 - 0.27 - SMOTE-NC 0.75 0.07 231 241 - 0.41 0.27 - - Structural Defects RF CSL 0.71 0.07 - 226 208 - 0.49 0.84 - RUS 0.70 0.06 174 - 163 0.49 - 0.32 - SMOTE 0.71 0.07 192 237 - 0.84 0.32 - - XGBoost CSL 0.71 0.07 - 248 197 - 0.20 0.95 - RUS 0.70 0.08 152 - 146 0.20 - 0.15 - 86 SMOTE 0.71 0.06 203 254 - 0.95 0.15 - - LightGBM CSL 0.72 0.07 - 287 202 - 0.02 0.97 1 RUS 0.69 0.07 113 - 107 0.02 - 0.01 3 SMOTE-NC 0.71 0.07 198 293 - 0.97 0.01 - 1 CatBoost CSL 0.71 0.07 - 254 198.5 - 0.15 0.98 - RUS 0.70 0.06 146 - 148 0.15 - 0.16 - SMOTE-NC 0.71 0.06 201.5 252 - 0.98 0.16 - - Utility B Infiltration RF CSL 0.81 0.07 - 235 166.5 - 0.35 0.37 - RUS 0.80 0.07 165 - 142 0.35 - 0.12 - SMOTE 0.82 0.07 233.5 258 - 0.37 0.12 - - XGBoost CSL 0.80 0.08 - 248 182.5 - 0.20 0.65 - RUS 0.78 0.08 152 - 139 0.20 - 0.10 - SMOTE 0.81 0.07 217.5 261 - 0.65 0.10 - - LightGBM CSL 0.79 0.07 - 216.5 132.5 - 0.67 0.07 2 RUS 0.78 0.09 183.5 - 119.5 0.67 - 0.03 3 SMOTE-NC 0.81 0.06 267.5 280.5 - 0.07 0.03 - 1 CatBoost CSL 0.80 0.07 - 274 181.5 - 0.05 0.63 1 RUS 0.78 0.08 126 - 118 0.05 - 0.03 3 SMOTE-NC 0.80 0.07 218.5 282 - 0.63 0.03 - 1 Structural Defects RF CSL 0.69 0.11 - 206.5 167.5 - 0.87 0.39 - RUS 0.68 0.10 193.5 - 160.5 0.87 - 0.29 - SMOTE 0.71 0.10 232.5 239.5 - 0.39 0.29 - - XGBoost CSL 0.72 0.10 - 265 211 - 0.08 0.78 - RUS 0.69 0.09 135 - 149 0.08 - 0.17 - SMOTE 0.71 0.10 189 251 - 0.78 0.17 - - LightGBM CSL 0.73 0.10 - 215.5 222 - 0.68 0.56 - RUS 0.73 0.07 184.5 - 214 0.68 - 0.71 - SMOTE-NC 0.71 0.10 178 186 - 0.56 0.71 - - CatBoost CSL 0.71 0.10 - 234 250.5 - 0.36 0.18 - RUS 0.69 0.11 166 - 214 0.36 - 0.71 - SMOTE-NC 0.68 0.12 149.5 186 - 0.18 0.71 - - 87 Table 14. The area under the average PR curves of each classification model and its U statistics, p-value, and rank when comparing its performance with model with other imbalanced data managing techniques, as determined by the Mann-Whitney U test at significance level equals to 0.05 Utility Target Variable Algorithm Imbalanced data managing technique mean AUC two standard deviations U Statistics when comparing the model to p-values when comparing the model to Rank CSL RUS SMOTE/ SMOTENC CSL RUS SMOTE/ SMOTENC Utility A Infiltration RF CSL 0.41 0.14 - 274 216 - 0.05 0.68 1 RUS 0.36 0.13 126 - 140 0.05 - 0.11 3 SMOTE 0.40 0.13 184 260 - 0.68 0.11 - 2 XGBoost CSL 0.40 0.15 - 259 265 - 0.11 0.08 - RUS 0.35 0.15 141 - 209 0.11 - 0.82 - SMOTE 0.36 0.11 135 191 - 0.08 0.82 - - LightGBM CSL 0.35 0.13 - 159 136 - 0.27 0.09 - RUS 0.37 0.17 241 - 184 0.27 - 0.68 - SMOTE-NC 0.39 0.13 264 216 - 0.09 0.68 - - CatBoost CSL 0.38 0.17 - 241 212 - 0.27 0.76 - RUS 0.35 0.17 159 - 172 0.27 - 0.46 - SMOTE-NC 0.38 0.15 188 228 - 0.76 0.46 - - Structural Defects RF CSL 0.50 0.12 - 250 206 - 0.18 0.88 - RUS 0.48 0.09 150 - 151 0.18 - 0.19 - SMOTE 0.50 0.12 194 249 - 0.88 0.19 - - XGBoost CSL 0.49 0.10 - 244 186 - 0.24 0.71 - RUS 0.47 0.09 156 - 133 0.24 - 0.07 - SMOTE 0.49 0.10 214 267 - 0.71 0.07 - - LightGBM CSL 0.49 0.10 - 277 184 - 0.04 0.68 1 RUS 0.46 0.09 123 - 119 0.04 - 0.03 3 SMOTE-NC 0.50 0.11 216 281 - 0.68 0.03 - 1 88 CatBoost CSL 0.51 0.11 - 264 231 - 0.09 0.41 - RUS 0.48 0.09 136 - 160 0.09 - 0.29 - SMOTE-NC 0.49 0.09 169 240 - 0.41 0.29 - - Utility B Infiltration RF CSL 0.61 0.11 - 272 256 - 0.05 0.13 - RUS 0.57 0.13 128 - 181 0.05 - 0.62 - SMOTE 0.60 0.14 144 219 - 0.13 0.62 - - XGBoost CSL 0.63 0.13 - 271 271 - 0.06 0.06 - RUS 0.58 0.14 129 - 176 0.06 - 0.52 - SMOTE 0.59 0.12 129 224 - 0.06 0.52 - - LightGBM CSL 0.62 0.12 - 252 235 - 0.16 0.35 - RUS 0.58 0.14 148 - 177 0.16 - 0.54 - SMOTE-NC 0.60 0.12 165 223 - 0.35 0.54 - - CatBoost CSL 0.63 0.13 - 302 232 - 0.01 0.39 1 RUS 0.54 0.15 98 - 111 0.01 - 0.02 3 SMOTE-NC 0.61 0.11 168 289 - 0.39 0.02 - 1 Structural Defects RF CSL 0.45 0.16 - 191 221 - 0.82 0.58 - RUS 0.46 0.15 209 - 236 0.82 - 0.34 - SMOTE 0.44 0.15 179 164 - 0.58 0.34 - - XGBoost CSL 0.50 0.17 - 223 232 - 0.54 0.39 - RUS 0.47 0.16 177 - 212 0.54 - 0.76 - SMOTE 0.48 0.15 168 188 - 0.39 0.76 - - LightGBM CSL 0.51 0.16 - 240 259 - 0.29 0.11 - RUS 0.48 0.15 160 - 246 0.29 - 0.22 - SMOTE-NC 0.46 0.16 141 154 - 0.11 0.22 - - CatBoost CSL 0.48 0.15 - 227 257 - 0.47 0.13 - RUS 0.47 0.15 173 - 215 0.47 - 0.69 - SMOTE-NC 0.45 0.15 143 185 - 0.13 0.69 - - 89 B.3 Results from all models Table 15. Rank ordered model predictor variables from first to last place of each classification model Utility Target Variable Algorithm Imbalanced data managing technique Rank ordered model predictor variables Utility A Infiltration RF CSL location, length, age, material, size, shape RUS location, material, length, age, size, shape SMOTE location, material, age, size, length, shape XGBoost CSL location, length, age, material, size, shape RUS location, age, length, material, size, shape SMOTE location, material, length, age, size, shape LightGBM CSL location, age, length, material, size, shape RUS location, length, age, material, size, shape SMOTE-NC location, size, age, material, length, shape CatBoost CSL location, material, age, length, size, shape RUS material, location, length, age, size, shape SMOTE-NC location, material, size, age, length, shape Structural Defects RF CSL location, material, age, length, size, shape RUS location, material, age, length, size, shape SMOTE location, material, age, length, size, shape XGBoost CSL location, age, length, material, size, shape RUS location, age, length, material, size, shape SMOTE location, length, material, age, size, shape LightGBM CSL location, age, length, material, size, shape RUS location, age, length, material, size, shape SMOTE-NC location, age, length, material, size, shape CatBoost CSL material, age, length, location, size, shape RUS material, age, location, length, size, shape SMOTE-NC location, material, length, age, size, shape Utility B Infiltration RF CSL location, age, size, length, material, shape 90 RUS size, age, location, length, material, shape SMOTE size, location, age, length, material, shape XGBoost CSL age, size, location, length, material, shape RUS age, size, length, location, material, shape SMOTE size, age, location, length, material, shape LightGBM CSL age, size, location, length, material, shape RUS age, size, length, location, material, shape SMOTE-NC age, size, location, length, material, shape CatBoost CSL size, age, length, location, material, shape RUS size, age, length, material, location, shape SMOTE-NC location, age, size, length, material, shape Structural Defects RF CSL location, age, length, size, shape, material RUS age, length, location, material, size, shape SMOTE location, age, length, shape, size, material XGBoost CSL age, length, location, size, material, shape RUS age, length, location, material, size, shape SMOTE age, length, location, shape, size, material LightGBM CSL age, length, location, size, material, shape RUS age, length, size, location, material, shape SMOTE-NC age, length, location, size, material, shape CatBoost CSL age, length, location, material, size, shape RUS length, age, material, shape, size, location SMOTE-NC length, age, location, size, material, shape 91 Figure 21-23 show the ROC and PR curves for models with the highest means for each classifying case. Figure 21. a) ROC and b) PR curves for models with the highest means for classifying infiltration for Utility A a b) 92 Figure 22. a) ROC and b) PR curves for models with the highest means for classifying structural defects for Utility A a) b) 93 Figure 23. a) ROC and b) PR curves for models with the highest means for classifying infiltration for Utility B b) a)
Thesis/Dissertation
eng
Civil Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Lence, Barbara J.
Enhancing sewer asset management using machine learning algorithms
Text