Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Graham-Knight, John Brandon
Master of Science - MSc
University of British Columbia
Image segmentation is a popular topic enabled by rapid advances in neural network processing. There is also a trend towards computational efficiency in segmentation algorithms, brought about by economic and environmental interests, and often acutely motivated by constraints in how the solutions are deployed. Recent exploration of neural network computational efficiency has largely focused on the depth and width of the network, as well as image size. Ensembling and gradient boosting are two well-known methods for increasing performance through a collection of smaller networks. Ensembling is used when different models produce largely uncorrelated predictions; the combination of these predictions reduces overall error. Gradient boosting produces correlated models such that the sum of their losses is minimized; successive models are trained using the residual loss of previous models, with each new learner correcting mistakes of past learners. This thesis seeks to explore the scalability and efficiency of image segmentation neural networks by including the techniques of ensembling and gradient boosting. The approach is evaluated on both the Severstal Steel Defect Detection and Kidney Tumor Segmentation datasets, though it could easily be applied to any two-dimensional image segmentation task. The final boosted model produces results comparable to the baseline in 5 of the 6 evaluated classes, while using only 2.5% of the trainable parameters. The success of the approach has significant implications for deployed applications; not only is the overall network much smaller, the miniature sub-networks can each be calculated independently and aggregated with simple addition and averaging. This introduces a real possibility for efficient distributed inferencing, decoupling performance from the maximum size calculable by a single piece of hardware.
Efficient 2D Image SegmentationbyJohn Brandon Graham-KnightB.Eng., The University of Victoria, 2010A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)August 2020John Brandon Graham-Knight, 2020This work is licensed under a Creative CommonsAttribution-NonCommercial-ShareAlike 4.0 International License.The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Efficient 2D Image Segmentationsubmitted by John Brandon Graham-Knight in partial fulfilment of therequirements of the degree of Master of ScienceDr. Patricia Lasserre, Irving K. Barber Faculty of ScienceCo-SupervisorDr. Homayoun Najjaran, School of EngineeringCo-SupervisorDr. Mohammad Khalad Hasan, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Apurva Narayan, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Kenneth Chau, School of EngineeringUniversity ExamineriiAbstractImage segmentation is a popular topic enabled by rapid advances inneural network processing. There is also a trend towards computationalefficiency in segmentation algorithms, brought about by economic and envi-ronmental interests, and often acutely motivated by constraints in how thesolutions are deployed. Recent exploration of neural network computationalefficiency has largely focused on the depth and width of the network, aswell as image size. Ensembling and gradient boosting are two well-knownmethods for increasing performance through a collection of smaller networks.Ensembling is used when different models produce largely uncorrelated pre-dictions; the combination of these predictions reduces overall error. Gradi-ent boosting produces correlated models such that the sum of their lossesis minimized; successive models are trained using the residual loss of pre-vious models, with each new learner correcting mistakes of past learners.This thesis seeks to explore the scalability and efficiency of image segmenta-tion neural networks by including the techniques of ensembling and gradientboosting. The approach is evaluated on both the Severstal Steel Defect De-tection and Kidney Tumor Segmentation datasets, though it could easily beapplied to any two-dimensional image segmentation task. The final boostedmodel produces results comparable to the baseline in 5 of the 6 evaluatedclasses, while using only 2.5% of the trainable parameters. The success ofthe approach has significant implications for deployed applications; not onlyis the overall network much smaller, the miniature sub-networks can eachbe calculated independently and aggregated with simple addition and aver-aging. This introduces a real possibility for efficient distributed inferencing,decoupling performance from the maximum size calculable by a single pieceof hardware.iiiLay SummarySelf-driving cars, medical image analysis, and industrial manufacturingare a few examples of image segmentation applications; these require boththe type and precise location of objects in an image. There is also a trendtowards efficient computing, motivated by economic and environmental in-terests. Sometimes, additional constraints arise from how the solutions aredeployed; on mobile devices, for example, the challenge is to achieve the bestresult possible on limited hardware. In this thesis, an approach combiningmultiple, easy-to-calculate segmentations is explored; results can be com-puted on several small pieces of hardware. Steel defect and kidney tumordetection applications are used for evaluation, and show good performancewith much higher efficiency than state-of-the-art alternatives.ivContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivContents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvList of Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxList of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . .xxxiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxxivAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . .xxxixDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xlChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 1Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 32.1 Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.1 Computer Vision Tasks . . . . . . . . . . . . . . . . . 32.1.1.1 Object Recognition and Detection . . . . . . 42.1.1.2 Segmentation . . . . . . . . . . . . . . . . . . 52.1.1.3 Semantic (Multi-Class) Segmentation . . . . 52.1.2 Equivalence of Single- and Multi-Class Segmentation . 62.1.3 Cross-Correlation and Convolution . . . . . . . . . . . 62.1.3.1 Image Boundaries . . . . . . . . . . . . . . . 72.1.3.2 Transposed Convolution and Stride . . . . . 8vCONTENTS2.1.3.3 Multi-Channel Convolution . . . . . . . . . . 92.1.4 Receptive Field . . . . . . . . . . . . . . . . . . . . . . 92.1.5 Common Convolution Kernels . . . . . . . . . . . . . . 102.1.5.1 Low-Pass Filtering Kernels . . . . . . . . . . 102.1.5.2 Edge Detector Kernels . . . . . . . . . . . . 112.1.6 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . 132.1.6.1 Accuracy . . . . . . . . . . . . . . . . . . . . 132.1.6.2 Precision . . . . . . . . . . . . . . . . . . . . 142.1.6.3 Recall . . . . . . . . . . . . . . . . . . . . . . 142.1.6.4 Specificity . . . . . . . . . . . . . . . . . . . 152.1.6.5 F-score (Sørensen–Dice coefficient) . . . . . . 152.1.6.6 Jaccard Index . . . . . . . . . . . . . . . . . 152.1.6.7 Area Under the Curve . . . . . . . . . . . . . 162.2 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 172.2.1 Activation Functions . . . . . . . . . . . . . . . . . . . 182.2.1.1 Linear Activation . . . . . . . . . . . . . . . 192.2.1.2 Rectified Linear Unit Activation . . . . . . . 192.2.1.3 Self-Gated Activation Functions . . . . . . . 202.2.1.4 Logistic Activation . . . . . . . . . . . . . . . 212.2.1.5 Full-Domain Activations . . . . . . . . . . . 222.2.2 Convolutional Neural Networks . . . . . . . . . . . . . 222.3 Training using Supervised Machine Learning . . . . . . . . . 232.3.1 Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.3.1.1 Loss Function Properties . . . . . . . . . . . 232.3.1.1.1 Continuous . . . . . . . . . . . . . . 242.3.1.1.2 Differentiability . . . . . . . . . . . 242.3.1.1.3 Convexity . . . . . . . . . . . . . . . 242.3.1.2 Binary Cross-Entropy (Log Loss) . . . . . . . 242.3.1.3 Binary Focal Loss . . . . . . . . . . . . . . . 252.3.1.4 Dice Loss . . . . . . . . . . . . . . . . . . . . 262.3.2 Back Propagation . . . . . . . . . . . . . . . . . . . . 262.3.3 Fitting a Model . . . . . . . . . . . . . . . . . . . . . . 272.3.4 Optimization . . . . . . . . . . . . . . . . . . . . . . . 282.3.4.1 Stochastic Gradient Descent . . . . . . . . . 292.3.4.2 Adaptive Optimizers . . . . . . . . . . . . . . 302.3.5 The Vanishing / Exploding Gradient . . . . . . . . . . 322.3.6 Regularization . . . . . . . . . . . . . . . . . . . . . . 332.3.6.1 L1-L2 Kernel Regularization . . . . . . . . . 332.3.6.2 Data Augmentation . . . . . . . . . . . . . . 342.3.6.3 Batch Normalization . . . . . . . . . . . . . . 35viCONTENTS2.3.7 Cross Validation . . . . . . . . . . . . . . . . . . . . . 352.3.8 Scalability and Efficiency . . . . . . . . . . . . . . . . 362.3.8.1 Image Size . . . . . . . . . . . . . . . . . . . 362.3.8.2 Width . . . . . . . . . . . . . . . . . . . . . . 382.3.8.3 Depth . . . . . . . . . . . . . . . . . . . . . . 392.3.8.4 Ensembling . . . . . . . . . . . . . . . . . . . 392.3.8.5 Gradient Boosting . . . . . . . . . . . . . . . 39Chapter 3: Competitions . . . . . . . . . . . . . . . . . . . . . . 413.1 U-Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.1.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . 433.2 Lung Nodule Competitions . . . . . . . . . . . . . . . . . . . 443.2.1 Z-Net . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.2.2 Multi-Opening and Threshold (MOT M5Lv1) . . . . . 453.2.3 grt123 . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.3 Other Competitions . . . . . . . . . . . . . . . . . . . . . . . 493.4 Trustworthiness of Competition Results . . . . . . . . . . . . 49Chapter 4: Methodology . . . . . . . . . . . . . . . . . . . . . . 514.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.2 Model Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.2.1 Aggregation . . . . . . . . . . . . . . . . . . . . . . . . 544.3 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.4 Facilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.4.1 Software System Overview . . . . . . . . . . . . . . . . 584.4.2 Computational Resources . . . . . . . . . . . . . . . . 594.4.3 Evaluation and Visualization . . . . . . . . . . . . . . 594.4.4 Data Processing . . . . . . . . . . . . . . . . . . . . . 604.4.5 Job Configuration . . . . . . . . . . . . . . . . . . . . 604.4.6 Checkpointing . . . . . . . . . . . . . . . . . . . . . . 60Chapter 5: Results and Discussion . . . . . . . . . . . . . . . . 615.1 F1 Score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.2 Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.3 Area Under the Curve Evaluation . . . . . . . . . . . . . . . . 645.4 Layer Outputs and Class Confusion . . . . . . . . . . . . . . 665.5 Selected Predictions . . . . . . . . . . . . . . . . . . . . . . . 69Chapter 6: Conclusion and Future Work . . . . . . . . . . . . . 72viiCONTENTSBibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Appendix A: Dataset Properties . . . . . . . . . . . . . . . . . . . 86A.1 Severstal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86A.2 Kits19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Appendix B: Full Results . . . . . . . . . . . . . . . . . . . . . . . 94B.1 Severstal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95B.1.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . 95B.1.2 Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . 96B.1.2.1 Baseline Class 1 . . . . . . . . . . . . . . . . 96B.1.2.2 Best Performing Threshold Histogram . . . . 97B.1.2.3 Baseline Class 2 . . . . . . . . . . . . . . . . 98B.1.2.4 Best Performing Threshold Histogram . . . . 99B.1.2.5 Baseline Class 3 . . . . . . . . . . . . . . . . 100B.1.2.6 Best Performing Threshold Histogram . . . . 101B.1.2.7 Baseline Class 4 . . . . . . . . . . . . . . . . 102B.1.2.8 Best Performing Threshold Histogram . . . . 103B.1.3 Weak Learners . . . . . . . . . . . . . . . . . . . . . . 104B.1.3.1 Weak Learners Class 1 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 104B.1.3.2 Weak Learners Class 1 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 105B.1.3.3 Weak Learners Class 1 - Noisy Or Aggregator 106B.1.3.4 Weak Learners Class 1 - Vote Aggregator . . 107B.1.3.5 Weak Learners Class 1 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 108B.1.3.6 Best Performing Threshold Histograms . . . 109B.1.3.7 Weak Learners Class 2 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 111B.1.3.8 Weak Learners Class 2 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 112B.1.3.9 Weak Learners Class 2 - Noisy Or Aggregator 113B.1.3.10 Weak Learners Class 2 - Vote Aggregator . . 114B.1.3.11 Weak Learners Class 2 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 115B.1.3.12 Best Performing Threshold Histograms . . . 116B.1.3.13 Weak Learners Class 3 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 118viiiCONTENTSB.1.3.14 Weak Learners Class 3 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 119B.1.3.15 Weak Learners Class 3 - Noisy Or Aggregator 120B.1.3.16 Weak Learners Class 3 - Vote Aggregator . . 121B.1.3.17 Weak Learners Class 3 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 122B.1.3.18 Best Performing Threshold Histograms . . . 123B.1.3.19 Weak Learners Class 4 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 125B.1.3.20 Weak Learners Class 4 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 126B.1.3.21 Weak Learners Class 4 - Noisy Or Aggregator 127B.1.3.22 Weak Learners Class 4 - Vote Aggregator . . 128B.1.3.23 Weak Learners Class 4 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 129B.1.3.24 Best Performing Threshold Histograms . . . 130B.1.4 Boosted Learners . . . . . . . . . . . . . . . . . . . . . 132B.1.4.1 Boosted Learners Class 1 - Average AfterAggregator . . . . . . . . . . . . . . . . . . . 132B.1.4.2 Boosted Learners Class 1 - Average BeforeAggregator . . . . . . . . . . . . . . . . . . . 133B.1.4.3 Boosted Learners Class 1 - Noisy Or Aggre-gator . . . . . . . . . . . . . . . . . . . . . . 134B.1.4.4 Boosted Learners Class 1 - Vote Aggregator 135B.1.4.5 Boosted Learners Class 1 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 136B.1.4.6 Best Performing Threshold Histograms . . . 137B.1.4.7 Boosted Learners Class 2 - Average AfterAggregator . . . . . . . . . . . . . . . . . . . 139B.1.4.8 Boosted Learners Class 2 - Average BeforeAggregator . . . . . . . . . . . . . . . . . . . 140B.1.4.9 Boosted Learners Class 2 - Noisy Or Aggre-gator . . . . . . . . . . . . . . . . . . . . . . 141B.1.4.10 Boosted Learners Class 2 - Vote Aggregator 142B.1.4.11 Boosted Learners Class 2 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 143B.1.4.12 Best Performing Threshold Histograms . . . 144B.1.4.13 Boosted Learners Class 3 - Average AfterAggregator . . . . . . . . . . . . . . . . . . . 146ixCONTENTSB.1.4.14 Boosted Learners Class 3 - Average BeforeAggregator . . . . . . . . . . . . . . . . . . . 147B.1.4.15 Boosted Learners Class 3 - Noisy Or Aggre-gator . . . . . . . . . . . . . . . . . . . . . . 148B.1.4.16 Boosted Learners Class 3 - Vote Aggregator 149B.1.4.17 Boosted Learners Class 3 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 150B.1.4.18 Best Performing Threshold Histograms . . . 151B.1.4.19 Boosted Learners Class 4 - Average AfterAggregator . . . . . . . . . . . . . . . . . . . 153B.1.4.20 Boosted Learners Class 4 - Average BeforeAggregator . . . . . . . . . . . . . . . . . . . 154B.1.4.21 Boosted Learners Class 4 - Noisy Or Aggre-gator . . . . . . . . . . . . . . . . . . . . . . 155B.1.4.22 Boosted Learners Class 4 - Vote Aggregator 156B.1.4.23 Boosted Learners Class 4 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 157B.1.4.24 Best Performing Threshold Histograms . . . 158B.1.5 Strong Learners . . . . . . . . . . . . . . . . . . . . . . 160B.1.5.1 Strong Learners Class 1 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 160B.1.5.2 Strong Learners Class 1 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 161B.1.5.3 Strong Learners Class 1 - Noisy Or Aggregator162B.1.5.4 Strong Learners Class 1 - Vote Aggregator . 163B.1.5.5 strong Learners Class 1 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 164B.1.5.6 Best Performing Threshold Histograms . . . 165B.1.5.7 Strong Learners Class 2 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 167B.1.5.8 Strong Learners Class 2 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 168B.1.5.9 Strong Learners Class 2 - Noisy Or Aggregator169B.1.5.10 Strong Learners Class 2 - Vote Aggregator . 170B.1.5.11 strong Learners Class 2 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 171B.1.5.12 Best Performing Threshold Histograms . . . 172B.1.5.13 Strong Learners Class 3 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 174xCONTENTSB.1.5.14 Strong Learners Class 3 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 175B.1.5.15 Strong Learners Class 3 - Noisy Or Aggregator176B.1.5.16 Strong Learners Class 3 - Vote Aggregator . 177B.1.5.17 strong Learners Class 3 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 178B.1.5.18 Best Performing Threshold Histograms . . . 179B.1.5.19 Strong Learners Class 4 - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 181B.1.5.20 Strong Learners Class 4 - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 182B.1.5.21 Strong Learners Class 4 - Noisy Or Aggregator183B.1.5.22 Strong Learners Class 4 - Vote Aggregator . 184B.1.5.23 strong Learners Class 4 - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 185B.1.5.24 Best Performing Threshold Histograms . . . 186B.2 KiTS19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188B.2.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . 188B.2.2 Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . 189B.2.2.1 Baseline Class Kidney . . . . . . . . . . . . . 189B.2.2.2 Best Performing Threshold Histogram . . . . 190B.2.2.3 Baseline Class Tumor . . . . . . . . . . . . . 191B.2.2.4 Best Performing Threshold Histogram . . . . 192B.2.3 Weak Learners . . . . . . . . . . . . . . . . . . . . . . 193B.2.3.1 Weak Learners Kidney- Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 193B.2.3.2 Weak Learners Kidney - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 194B.2.3.3 Weak Learners Kidney - Noisy Or Aggregator 195B.2.3.4 Weak Learners Kidney - Vote Aggregator . . 196B.2.3.5 Weak Learners Kidney - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 197B.2.3.6 Best Performing Threshold Histogram . . . . 198B.2.3.7 Weak Learners Tumor- Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 200B.2.3.8 Weak Learners Tumor - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 201B.2.3.9 Weak Learners Tumor - Noisy Or Aggregator 202B.2.3.10 Weak Learners Tumor - Vote Aggregator . . 203xiCONTENTSB.2.3.11 Weak Learners Tumor - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 204B.2.3.12 Best Performing Threshold Histogram . . . . 205B.2.4 Boosted Learners . . . . . . . . . . . . . . . . . . . . . 207B.2.4.1 Boosted Learners Kidney - Average AfterAggregator . . . . . . . . . . . . . . . . . . . 207B.2.4.2 Boosted Learners Kidney - Average BeforeAggregator . . . . . . . . . . . . . . . . . . . 208B.2.4.3 Boosted Learners Kidney - Noisy Or Aggre-gator . . . . . . . . . . . . . . . . . . . . . . 209B.2.4.4 Boosted Learners Kidney - Vote Aggregator 210B.2.4.5 Boosted Learners Kidney - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 211B.2.4.6 Best Performing Threshold Histogram . . . . 212B.2.4.7 Boosted Learners Tumor - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 214B.2.4.8 Boosted Learners Tumor - Average BeforeAggregator . . . . . . . . . . . . . . . . . . . 215B.2.4.9 Boosted Learners Tumor - Noisy Or Aggre-gator . . . . . . . . . . . . . . . . . . . . . . 216B.2.4.10 Boosted Learners Tumor - Vote Aggregator . 217B.2.4.11 Boosted Learners Tumor - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 218B.2.4.12 Best Performing Threshold Histogram . . . . 219B.2.5 Strong Learners . . . . . . . . . . . . . . . . . . . . . . 221B.2.5.1 Strong Learners Kidney - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 221B.2.5.2 Strong Learners Kidney - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 222B.2.5.3 Strong Learners Kidney - Noisy Or Aggregator223B.2.5.4 Strong Learners Kidney - Vote Aggregator . 224B.2.5.5 Strong Learners Kidney - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 225B.2.5.6 Best Performing Threshold Histogram . . . . 226B.2.5.7 Strong Learners Tumor - Average After Ag-gregator . . . . . . . . . . . . . . . . . . . . . 228B.2.5.8 Strong Learners Tumor - Average Before Ag-gregator . . . . . . . . . . . . . . . . . . . . . 229B.2.5.9 Strong Learners Tumor - Noisy Or Aggregator230B.2.5.10 Strong Learners Tumor - Vote Aggregator . . 231xiiCONTENTSB.2.5.11 Strong Learners Tumor - Aggregator Inde-pendent . . . . . . . . . . . . . . . . . . . . . 232B.2.5.12 Best Performing Threshold Histogram . . . . 233B.3 Selected Predictions . . . . . . . . . . . . . . . . . . . . . . . 235xiiiList of TablesTable 3.1 Lung Nodule Detection Competitions . . . . . . . . . . 45Table 4.1 Component size statistics for the Severstal classes. . . 52Table 4.2 A comparison of the configurations evaluated. . . . . 55Table 4.3 A comparison of the four aggregation methods evalu-ated. . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Table B.1 Summary of Performance Metrics . . . . . . . . . . . 94Table B.2 Summary of Activation Layer Outputs . . . . . . . . 94Table B.3 Summary of Convolution Layer Outputs . . . . . . . . 94Table B.4 Class 1 Results . . . . . . . . . . . . . . . . . . . . . . 95Table B.5 Class 2 Results . . . . . . . . . . . . . . . . . . . . . . 95Table B.6 Class 3 Results . . . . . . . . . . . . . . . . . . . . . . 95Table B.7 Class 4 Results . . . . . . . . . . . . . . . . . . . . . . 95Table B.8 Kidney Results . . . . . . . . . . . . . . . . . . . . . . 188Table B.9 Tumor Results . . . . . . . . . . . . . . . . . . . . . . 188xivList of FiguresFigure 2.1 Object Detection . . . . . . . . . . . . . . . . . . . . 5Figure 2.2 Semantic Segmentation . . . . . . . . . . . . . . . . . 6Figure 2.3 Convolution . . . . . . . . . . . . . . . . . . . . . . . 8Figure 2.4 Example 2-Strided Convolution . . . . . . . . . . . . 9Figure 2.5 The Receptive Field of Convolution . . . . . . . . . . 10Figure 2.6 3x3 Averaging Kernel . . . . . . . . . . . . . . . . . . 11Figure 2.7 Gaussian Kernel . . . . . . . . . . . . . . . . . . . . . 11Figure 2.8 Laplacian Kernel . . . . . . . . . . . . . . . . . . . . . 12Figure 2.9 Horizontal Sobel Kernel . . . . . . . . . . . . . . . . . 12Figure 2.10 Vertical Sobel Kernel . . . . . . . . . . . . . . . . . . 13Figure 2.11 Example ROC and PRC Curves . . . . . . . . . . . . 16Figure 2.12 The Perceptron . . . . . . . . . . . . . . . . . . . . . 17Figure 2.13 The Multi-Layer Perceptron . . . . . . . . . . . . . . 18Figure 2.14 Example Rectified Linear Unit Activation Functions . 20Figure 2.15 Swish Activation . . . . . . . . . . . . . . . . . . . . . 21Figure 2.16 Example Logistic Activation Functions . . . . . . . . 22Figure 2.17 Example Full Domain Activation Functions . . . . . . 22Figure 2.18 Binary Focal Loss . . . . . . . . . . . . . . . . . . . . 25Figure 2.19 Back Propagation . . . . . . . . . . . . . . . . . . . . 27Figure 3.1 Computer Aided Diagnosis . . . . . . . . . . . . . . . 41Figure 3.2 U-Net architecture . . . . . . . . . . . . . . . . . . . . 44Figure 3.3 Radiomics Workflow . . . . . . . . . . . . . . . . . . . 46Figure 3.4 Example grt123 Pre-Processing Results . . . . . . . . 48Figure 4.1 Class Instance Counts . . . . . . . . . . . . . . . . . . 52Figure 4.2 Components in the KiTS19 Dataset . . . . . . . . . . 53Figure 4.3 Model Construction . . . . . . . . . . . . . . . . . . . 56Figure 4.4 Aggregation . . . . . . . . . . . . . . . . . . . . . . . 56Figure 4.5 Boosted Model Training . . . . . . . . . . . . . . . . . 57Figure 4.6 System Overview . . . . . . . . . . . . . . . . . . . . 58xvLIST OF FIGURESFigure 5.1 Average After Sigmoid F1 Score by Model and Class 62Figure 5.2 Average Before Sigmoid F1 Score by Model and Class 62Figure 5.3 Noisy Or F1 Score by Model and Class . . . . . . . . 62Figure 5.4 Vote F1 Score by Model and Class . . . . . . . . . . . 63Figure 5.5 KiTS19 Boosting Improvement . . . . . . . . . . . . . 63Figure 5.6 Severstal Boosting Improvement . . . . . . . . . . . . 64Figure 5.7 Severstal Class 1 AUC . . . . . . . . . . . . . . . . . 65Figure 5.8 Severstal Class 2 AUC . . . . . . . . . . . . . . . . . 65Figure 5.9 Severstal Class 3 AUC . . . . . . . . . . . . . . . . . 65Figure 5.10 Severstal Class 4 AUC . . . . . . . . . . . . . . . . . 66Figure 5.11 KiTS19 Kidney AUC . . . . . . . . . . . . . . . . . . 66Figure 5.12 KiTS19 Tumor AUC . . . . . . . . . . . . . . . . . . 66Figure 5.13 Convolution Layer Output Histograms . . . . . . . . 67Figure 5.14 Activation Layer Output Histograms . . . . . . . . . 67Figure 5.15 Severstal Confusion matrices . . . . . . . . . . . . . . 68Figure 5.16 KiTS19 Confusion matrices . . . . . . . . . . . . . . . 68Figure 5.17 KiTS19 Kidney - Case 00003, Slice 62 . . . . . . . . . 69Figure 5.18 KiTS19 Tumor - Case 00003, Slice 62 . . . . . . . . . 70Figure 5.19 Severstal Class 1 - 3fdc95c42 . . . . . . . . . . . . . . 71Figure 6.1 Augmented Reality Smartphone Application . . . . . 74Figure A.1 Class Instance Counts in the Severstal Dataset . . . . 86Figure A.2 Class 1 Coverage in the Severstal Dataset . . . . . . . 87Figure A.3 Class 2 Coverage in the Severstal Dataset . . . . . . . 87Figure A.4 Class 3 Coverage in the Severstal Dataset . . . . . . . 88Figure A.5 Class 4 Coverage in the Severstal Dataset . . . . . . . 88Figure A.6 Components in Class 1 of the Severstal Dataset . . . 89Figure A.7 Components in Class 2 of the Severstal Dataset . . . 89Figure A.8 Components in Class 3 of the Severstal Dataset . . . 90Figure A.9 Components in Class 4 of the Severstal Dataset . . . 90Figure A.10 Class Instance Counts in the KiTS19 Dataset . . . . 91Figure A.11 Class Kidney Coverage in the KiTS19 Dataset . . . . 91Figure A.12 Class Tumor Coverage in the KiTS19 Dataset . . . . 92Figure A.13 Components in Class Kidney of the KiTS19 Dataset . 92Figure A.14 Components in Class Tumor of the KiTS19 Dataset . 93Figure B.1 Severstal Class 1 Metrics for Baseline Model Dummy 96Figure B.2 Severstal Class 1 AUC for Baseline Model Dummy . . 96Figure B.3 Severstal Class 1 Layer Outputs for Baseline Model . 97xviLIST OF FIGURESFigure B.4 Class 1 F1 Score for threshold 0.5 with Baseline Modeland Severstal Dataset . . . . . . . . . . . . . . . . . . 97Figure B.5 Severstal Class 2 Metrics for Baseline Model Dummy 98Figure B.6 Severstal Class 2 AUC for Baseline Model Dummy . . 98Figure B.7 Severstal Class 2 Layer Outputs for Baseline Model . 99Figure B.8 Class 2 F1 Score for threshold 0.5 with Baseline Modeland Severstal Dataset . . . . . . . . . . . . . . . . . . 99Figure B.9 Severstal Class 3 Metrics for Baseline Model Dummy 100Figure B.10 Severstal Class 3 AUC for Baseline Model Dummy . . 100Figure B.11 Severstal Class 3 Layer Outputs for Baseline Model . 101Figure B.12 Class 3 F1 Score for threshold 0.5 with Baseline Modeland Severstal Dataset . . . . . . . . . . . . . . . . . . 101Figure B.13 Severstal Class 4 Metrics for Baseline Model Dummy 102Figure B.14 Severstal Class 4 AUC for Baseline Model Dummy . . 102Figure B.15 Severstal Class 4 Layer Outputs for Baseline Model . 103Figure B.16 Class 4 F1 Score for threshold 0.5 with Baseline Modeland Severstal Dataset . . . . . . . . . . . . . . . . . . 103Figure B.17 Severstal Class 1 Metrics for Weak Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 104Figure B.18 Severstal Class 1 AUC for Weak Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 104Figure B.19 Severstal Class 1 Metrics for Weak Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 105Figure B.20 Severstal Class 1 AUC for Weak Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 105Figure B.21 Severstal Class 1 Metrics for Weak Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 106Figure B.22 Severstal Class 1 AUC for Weak Model and Noisy OrAggregation . . . . . . . . . . . . . . . . . . . . . . . 106Figure B.23 Severstal Class 1 Metrics for Weak Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 107Figure B.24 Severstal Class 1 AUC for Weak Model and Vote Ag-gregation . . . . . . . . . . . . . . . . . . . . . . . . . 107Figure B.25 Severstal Class 1 Layer Outputs for Weak Model . . . 108Figure B.26 Class 1 F1 Score for threshold 0.29, Weak Learners,Average After Aggregation, and Severstal Dataset . . 109Figure B.27 Class 1 F1 Score for threshold 0.28, Weak Learners,Average Before Aggregation, and Severstal Dataset . 109Figure B.28 Class 1 F1 Score for threshold, 0.97 Weak Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 110xviiLIST OF FIGURESFigure B.29 Class 1 F1 Score for threshold 0.00, Weak Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 110Figure B.30 Severstal Class 2 Metrics for Weak Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 111Figure B.31 Severstal Class 2 AUC for Weak Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 111Figure B.32 Severstal Class 2 Metrics for Weak Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 112Figure B.33 Severstal Class 2 AUC for Weak Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 112Figure B.34 Severstal Class 2 Metrics for Weak Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 113Figure B.35 Severstal Class 2 AUC for Weak Model and Noisy OrAggregation . . . . . . . . . . . . . . . . . . . . . . . 113Figure B.36 Severstal Class 2 Metrics for Weak Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 114Figure B.37 Severstal Class 2 AUC for Weak Model and Vote Ag-gregation . . . . . . . . . . . . . . . . . . . . . . . . . 114Figure B.38 Severstal Class 2 Layer Outputs for Weak Model . . . 115Figure B.39 Class 2 F1 Score for threshold 0.34, Weak Learners,Average After Aggregation, and Severstal Dataset . . 116Figure B.40 Class 2 F1 Score for threshold 0.33, Weak Learners,Average Before Aggregation, and Severstal Dataset . 116Figure B.41 Class 2 F1 Score for threshold 0.99, Weak Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 117Figure B.42 Class 2 F1 Score for threshold 0.00, Weak Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 117Figure B.43 Severstal Class 3 Metrics for Weak Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 118Figure B.44 Severstal Class 3 AUC for Weak Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 118Figure B.45 Severstal Class 3 Metrics for Weak Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 119Figure B.46 Severstal Class 3 AUC for Weak Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 119Figure B.47 Severstal Class 3 Metrics for Weak Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 120Figure B.48 Severstal Class 3 AUC for Weak Model and Noisy OrAggregation . . . . . . . . . . . . . . . . . . . . . . . 120xviiiLIST OF FIGURESFigure B.49 Severstal Class 3 Metrics for Weak Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 121Figure B.50 Severstal Class 3 AUC for Weak Model and Vote Ag-gregation . . . . . . . . . . . . . . . . . . . . . . . . . 121Figure B.51 Severstal Class 3 Layer Outputs for Weak Model . . . 122Figure B.52 Class 3 F1 Score for threshold 0.30, Weak Learners,Average After Aggregation, and Severstal Dataset . . 123Figure B.53 Class 3 F1 Score for threshold 0.30, Weak Learners,Average Before Aggregation, and Severstal Dataset . 123Figure B.54 Class 3 F1 Score for threshold 0.97, Weak Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 124Figure B.55 Class 3 F1 Score for threshold 0.00, Weak Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 124Figure B.56 Severstal Class 4 Metrics for Weak Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 125Figure B.57 Severstal Class 4 AUC for Weak Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 125Figure B.58 Severstal Class 4 Metrics for Weak Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 126Figure B.59 Severstal Class 4 AUC for Weak Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 126Figure B.60 Severstal Class 4 Metrics for Weak Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 127Figure B.61 Severstal Class 4 AUC for Weak Model and Noisy OrAggregation . . . . . . . . . . . . . . . . . . . . . . . 127Figure B.62 Severstal Class 4 Metrics for Weak Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 128Figure B.63 Severstal Class 4 AUC for Weak Model and Vote Ag-gregation . . . . . . . . . . . . . . . . . . . . . . . . . 128Figure B.64 Severstal Class 4 Layer Outputs for Weak Model . . . 129Figure B.65 Class 4 F1 Score for threshold 0.31, Weak Learners,Average After Aggregation, and Severstal Dataset . . 130Figure B.66 Class 4 F1 Score for threshold 0.30, Weak Learners,Average Before Aggregation, and Severstal Dataset . 130Figure B.67 Class 4 F1 Score for threshold 0.98, Weak Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 131Figure B.68 Class 4 F1 Score for threshold 0.00, Weak Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 131Figure B.69 Severstal Class 1 Metrics for Boosted Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 132xixLIST OF FIGURESFigure B.70 Severstal Class 1 AUC for Boosted Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 132Figure B.71 Severstal Class 1 Metrics for Boosted Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 133Figure B.72 Severstal Class 1 AUC for Boosted Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 133Figure B.73 Severstal Class 1 Metrics for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 134Figure B.74 Severstal Class 1 AUC for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 134Figure B.75 Severstal Class 1 Metrics for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 135Figure B.76 Severstal Class 1 AUC for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 135Figure B.77 Severstal Class 1 Layer Outputs for Boosted Model . 136Figure B.78 Severstal Class 1 Boosting Improvement . . . . . . . 136Figure B.79 Class 1 F1 Score for threshold 0.30, Boosted Learners,Average After Aggregation, and Severstal Dataset . . 137Figure B.80 Class 1 F1 Score for threshold 0.30, Boosted Learners,Average Before Aggregation, and Severstal Dataset . 137Figure B.81 Class 1 F1 Score for threshold, 0.97 Boosted Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 138Figure B.82 Class 1 F1 Score for threshold 0.00, Boosted Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 138Figure B.83 Severstal Class 2 Metrics for Boosted Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 139Figure B.84 Severstal Class 2 AUC for Boosted Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 139Figure B.85 Severstal Class 2 Metrics for Boosted Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 140Figure B.86 Severstal Class 2 AUC for Boosted Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 140Figure B.87 Severstal Class 2 Metrics for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 141Figure B.88 Severstal Class 2 AUC for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 141Figure B.89 Severstal Class 2 Metrics for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 142Figure B.90 Severstal Class 2 AUC for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 142xxLIST OF FIGURESFigure B.91 Severstal Class 2 Layer Outputs for Boosted Model . 143Figure B.92 Severstal Class 2 Boosting Improvement . . . . . . . 143Figure B.93 Class 2 F1 Score for threshold 0.34, Boosted Learners,Average After Aggregation, and Severstal Dataset . . 144Figure B.94 Class 2 F1 Score for threshold 0.33, Boosted Learners,Average Before Aggregation, and Severstal Dataset . 144Figure B.95 Class 2 F1 Score for threshold 0.98, Boosted Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 145Figure B.96 Class 2 F1 Score for threshold 0.00, Boosted Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 145Figure B.97 Severstal Class 3 Metrics for Boosted Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 146Figure B.98 Severstal Class 3 AUC for Boosted Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 146Figure B.99 Severstal Class 3 Metrics for Boosted Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 147Figure B.100Severstal Class 3 AUC for Boosted Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 147Figure B.101Severstal Class 3 Metrics for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 148Figure B.102Severstal Class 3 AUC for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 148Figure B.103Severstal Class 3 Metrics for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 149Figure B.104Severstal Class 3 AUC for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 149Figure B.105Severstal Class 3 Layer Outputs for Boosted Model . 150Figure B.106Severstal Class 3 Boosting Improvement . . . . . . . 150Figure B.107Class 3 F1 Score for threshold 0.33, Boosted Learners,Average After Aggregation, and Severstal Dataset . . 151Figure B.108Class 3 F1 Score for threshold 0.32, Boosted Learners,Average Before Aggregation, and Severstal Dataset . 151Figure B.109Class 3 F1 Score for threshold 0.98, Boosted Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 152Figure B.110Class 3 F1 Score for threshold 0.00, Boosted Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 152Figure B.111Severstal Class 4 Metrics for Boosted Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 153Figure B.112Severstal Class 4 AUC for Boosted Model and Aver-age After Aggregation . . . . . . . . . . . . . . . . . . 153xxiLIST OF FIGURESFigure B.113Severstal Class 4 Metrics for Boosted Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 154Figure B.114Severstal Class 4 AUC for Boosted Model and Aver-age Before Aggregation . . . . . . . . . . . . . . . . . 154Figure B.115Severstal Class 4 Metrics for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 155Figure B.116Severstal Class 4 AUC for Boosted Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 155Figure B.117Severstal Class 4 Metrics for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 156Figure B.118Severstal Class 4 AUC for Boosted Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 156Figure B.119Severstal Class 4 Layer Outputs for Boosted Model . 157Figure B.120Severstal Class 4 Boosting Improvement . . . . . . . 157Figure B.121Class 4 F1 Score for threshold 0.32, Boosted Learners,Average After Aggregation, and Severstal Dataset . . 158Figure B.122Class 4 F1 Score for threshold 0.31, Boosted Learners,Average Before Aggregation, and Severstal Dataset . 158Figure B.123Class 4 F1 Score for threshold 0.98, Boosted Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 159Figure B.124Class 4 F1 Score for threshold 0.00, Boosted Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 159Figure B.125Severstal Class 1 Metrics for Strong Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 160Figure B.126Severstal Class 1 AUC for Strong Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 160Figure B.127Severstal Class 1 Metrics for Strong Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 161Figure B.128Severstal Class 1 AUC for Strong Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 161Figure B.129Severstal Class 1 Metrics for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 162Figure B.130Severstal Class 1 AUC for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 162Figure B.131Severstal Class 1 Metrics for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 163Figure B.132Severstal Class 1 AUC for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 163Figure B.133Severstal Class 1 Layer Outputs for Strong Model . . 164xxiiLIST OF FIGURESFigure B.134Class 1 F1 Score for threshold 0.33, Strong Learners,Average After Aggregation, and Severstal Dataset . . 165Figure B.135Class 1 F1 Score for threshold 0.32, Strong Learners,Average Before Aggregation, and Severstal Dataset . 165Figure B.136Class 1 F1 Score for threshold, 0.98 Strong Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 166Figure B.137Class 1 F1 Score for threshold 0.00, Strong Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 166Figure B.138Severstal Class 2 Metrics for Strong Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 167Figure B.139Severstal Class 2 AUC for Strong Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 167Figure B.140Severstal Class 2 Metrics for Strong Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 168Figure B.141Severstal Class 2 AUC for Strong Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 168Figure B.142Severstal Class 2 Metrics for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 169Figure B.143Severstal Class 2 AUC for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 169Figure B.144Severstal Class 2 Metrics for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 170Figure B.145Severstal Class 2 AUC for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 170Figure B.146Severstal Class 2 Layer Outputs for Strong Model . . 171Figure B.147Class 2 F1 Score for threshold 0.33, Strong Learners,Average After Aggregation, and Severstal Dataset . . 172Figure B.148Class 2 F1 Score for threshold 0.32, Strong Learners,Average Before Aggregation, and Severstal Dataset . 172Figure B.149Class 2 F1 Score for threshold 0.99, Strong Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 173Figure B.150Class 2 F1 Score for threshold 0.10, Strong Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 173Figure B.151Severstal Class 3 Metrics for Strong Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 174Figure B.152Severstal Class 3 AUC for Strong Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 174Figure B.153Severstal Class 3 Metrics for Strong Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 175xxiiiLIST OF FIGURESFigure B.154Severstal Class 3 AUC for Strong Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 175Figure B.155Severstal Class 3 Metrics for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 176Figure B.156Severstal Class 3 AUC for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 176Figure B.157Severstal Class 3 Metrics for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 177Figure B.158Severstal Class 3 AUC for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 177Figure B.159Severstal Class 3 Layer Outputs for Strong Model . . 178Figure B.160Class 3 F1 Score for threshold 0.34, Strong Learners,Average After Aggregation, and Severstal Dataset . . 179Figure B.161Class 3 F1 Score for threshold 0.34, Strong Learners,Average Before Aggregation, and Severstal Dataset . 179Figure B.162Class 3 F1 Score for threshold 0.98, Strong Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 180Figure B.163Class 3 F1 Score for threshold 0.00, Strong Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 180Figure B.164Severstal Class 4 Metrics for Strong Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 181Figure B.165Severstal Class 4 AUC for Strong Model and AverageAfter Aggregation . . . . . . . . . . . . . . . . . . . . 181Figure B.166Severstal Class 4 Metrics for Strong Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 182Figure B.167Severstal Class 4 AUC for Strong Model and AverageBefore Aggregation . . . . . . . . . . . . . . . . . . . 182Figure B.168Severstal Class 4 Metrics for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 183Figure B.169Severstal Class 4 AUC for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 183Figure B.170Severstal Class 4 Metrics for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 184Figure B.171Severstal Class 4 AUC for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 184Figure B.172Severstal Class 4 Layer Outputs for Strong Model . . 185Figure B.173Class 4 F1 Score for threshold 0.32, Strong Learners,Average After Aggregation, and Severstal Dataset . . 186Figure B.174Class 4 F1 Score for threshold 0.31, Strong Learners,Average Before Aggregation, and Severstal Dataset . 186xxivLIST OF FIGURESFigure B.175Class 4 F1 Score for threshold 0.98, Strong Learners,Noisy Or Aggregation, and Severstal Dataset . . . . . 187Figure B.176Class 4 F1 Score for threshold 0.00, Strong Learners,Vote Aggregation, and Severstal Dataset . . . . . . . 187Figure B.177KiTS19 Class Kidney Metrics for Baseline Model . . 189Figure B.178KiTS19 Class Kidney AUC for Baseline Model . . . 189Figure B.179KiTS19 Class Kidney Layer Outputs for Baseline Model190Figure B.180Kidney F1 Score for threshold 0.5 with Baseline Modeland KiTS19 Dataset . . . . . . . . . . . . . . . . . . . 190Figure B.181KiTS19 Class Tumor Metrics for Baseline Model . . 191Figure B.182KiTS19 Class Tumor AUC for Baseline Model . . . . 191Figure B.183KiTS19 Class Tumor Layer Outputs for Baseline Model192Figure B.184Tumor F1 Score for threshold 0.5 with Baseline Modeland KiTS19 Dataset . . . . . . . . . . . . . . . . . . . 192Figure B.185KiTS19 Class Kidney Metrics for Weak Model andAverage After Aggregation . . . . . . . . . . . . . . . 193Figure B.186KiTS19 Class Kidney AUC for Weak Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 193Figure B.187KiTS19 Class Kidney Metrics for Weak Model andAverage Before Aggregation . . . . . . . . . . . . . . 194Figure B.188KiTS19 Class Kidney AUC for Weak Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 194Figure B.189KiTS19 Class Kidney Metrics for Weak Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 195Figure B.190KiTS19 Class Kidney AUC for Weak Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 195Figure B.191KiTS19 Class Kidney Metrics for Weak Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 196Figure B.192KiTS19 Class Kidney AUC for Weak Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 196Figure B.193KiTS19 Class Kidney Layer Outputs for Weak Model 197Figure B.194Kidney F1 Score for threshold 0.32, Boosted Learners,Average After Aggregation, and KiTS19 Dataset . . . 198Figure B.195Kidney F1 Score for threshold 0.31, Boosted Learners,Average Before Aggregation, and KiTS19 Dataset . . 198Figure B.196Kidney F1 Score for threshold 0.98, Boosted Learners,Noisy Or Aggregation, and KiTS19 Dataset . . . . . 199Figure B.197Kidney F1 Score for threshold 0.10, Boosted Learners,Vote Aggregation, and KiTS19 Dataset . . . . . . . . 199xxvLIST OF FIGURESFigure B.198KiTS19 Class Tumor Metrics for Weak Model andAverage After Aggregation . . . . . . . . . . . . . . . 200Figure B.199KiTS19 Class Tumor AUC for Weak Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 200Figure B.200KiTS19 Class Tumor Metrics for Weak Model andAverage Before Aggregation . . . . . . . . . . . . . . 201Figure B.201KiTS19 Class Tumor AUC for Weak Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 201Figure B.202KiTS19 Class Tumor Metrics for Weak Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 202Figure B.203KiTS19 Class Tumor AUC for Weak Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 202Figure B.204KiTS19 Class Tumor Metrics for Weak Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 203Figure B.205KiTS19 Class Tumor AUC for Weak Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 203Figure B.206KiTS19 Class Tumor Layer Outputs for Weak Model 204Figure B.207Tumor F1 Score for threshold 0.20, Boosted Learners,Average After Aggregation, and KiTS19 Dataset . . . 205Figure B.208Tumor F1 Score for threshold 0.15, Boosted Learners,Average Before Aggregation, and KiTS19 Dataset . . 205Figure B.209Tumor F1 Score for threshold 0.85, Boosted Learners,Noisy Or Aggregation, and KiTS19 Dataset . . . . . 206Figure B.210Tumor F1 Score for threshold 0.00, Boosted Learners,Vote Aggregation, and KiTS19 Dataset . . . . . . . . 206Figure B.211KiTS19 Class Kidney Metrics for Boosted Model andAverage After Aggregation . . . . . . . . . . . . . . . 207Figure B.212KiTS19 Class Kidney AUC for Boosted Model andAverage After Aggregation . . . . . . . . . . . . . . . 207Figure B.213KiTS19 Class Kidney Metrics for Boosted Model andAverage Before Aggregation . . . . . . . . . . . . . . 208Figure B.214KiTS19 Class Kidney AUC for Boosted Model andAverage Before Aggregation . . . . . . . . . . . . . . 208Figure B.215KiTS19 Class Kidney Metrics for Boosted Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 209Figure B.216KiTS19 Class Kidney AUC for Boosted Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 209Figure B.217KiTS19 Class Kidney Metrics for Boosted Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 210xxviLIST OF FIGURESFigure B.218KiTS19 Class Kidney AUC for Boosted Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 210Figure B.219KiTS19 Class Kidney Layer Outputs for Boosted Model211Figure B.220KiTS19 Class Kidney Boosting Improvement . . . . . 211Figure B.221Kidney F1 Score for threshold 0.33, Boosted Learners,Average After Aggregation, and KiTS19 Dataset . . . 212Figure B.222Kidney F1 Score for threshold 0.31, Boosted Learners,Average Before Aggregation, and KiTS19 Dataset . . 212Figure B.223Kidney F1 Score for threshold 0.98, Boosted Learners,Noisy Or Aggregation, and KiTS19 Dataset . . . . . 213Figure B.224Kidney F1 Score for threshold 0.10, Boosted Learners,Vote Aggregation, and KiTS19 Dataset . . . . . . . . 213Figure B.225KiTS19 Class Tumor Metrics for Boosted Model andAverage After Aggregation . . . . . . . . . . . . . . . 214Figure B.226KiTS19 Class Tumor AUC for Boosted Model andAverage After Aggregation . . . . . . . . . . . . . . . 214Figure B.227KiTS19 Class Tumor Metrics for Boosted Model andAverage Before Aggregation . . . . . . . . . . . . . . 215Figure B.228KiTS19 Class Tumor AUC for Boosted Model andAverage Before Aggregation . . . . . . . . . . . . . . 215Figure B.229KiTS19 Class Tumor Metrics for Boosted Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 216Figure B.230KiTS19 Class Tumor AUC for Boosted Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 216Figure B.231KiTS19 Class Tumor Metrics for Boosted Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 217Figure B.232KiTS19 Class Tumor AUC for Boosted Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 217Figure B.233KiTS19 Class Tumor Layer Outputs for Boosted Model218Figure B.234KiTS19 Class Tumor Boosting Improvement . . . . . 218Figure B.235Tumor F1 Score for threshold 0.20, Boosted Learners,Average After Aggregation, and KiTS19 Dataset . . . 219Figure B.236Tumor F1 Score for threshold 0.20, Boosted Learners,Average Before Aggregation, and KiTS19 Dataset . . 219Figure B.237Tumor F1 Score for threshold 0.90, Boosted Learners,Noisy Or Aggregation, and KiTS19 Dataset . . . . . 220Figure B.238Tumor F1 Score for threshold 0.00, Boosted Learners,Vote Aggregation, and KiTS19 Dataset . . . . . . . . 220Figure B.239KiTS19 Class Kidney Metrics for Strong Model andAverage After Aggregation . . . . . . . . . . . . . . . 221xxviiLIST OF FIGURESFigure B.240KiTS19 Class Kidney AUC for Strong Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 221Figure B.241KiTS19 Class Kidney Metrics for Strong Model andAverage Before Aggregation . . . . . . . . . . . . . . 222Figure B.242KiTS19 Class Kidney AUC for Strong Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 222Figure B.243KiTS19 Class Kidney Metrics for Strong Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 223Figure B.244KiTS19 Class Kidney AUC for Strong Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 223Figure B.245KiTS19 Class Kidney Metrics for Strong Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 224Figure B.246KiTS19 Class Kidney AUC for Strong Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 224Figure B.247KiTS19 Class Kidney Layer Outputs for Strong Model 225Figure B.248Kidney F1 Score for threshold 0.33, Strong Learners,Average After Aggregation, and KiTS19 Dataset . . . 226Figure B.249Kidney F1 Score for threshold 0.32, Strong Learners,Average Before Aggregation, and KiTS19 Dataset . . 226Figure B.250Kidney F1 Score for threshold 0.99, Strong Learners,Noisy Or Aggregation, and KiTS19 Dataset . . . . . 227Figure B.251Kidney F1 Score for threshold 0.10, Strong Learners,Vote Aggregation, and KiTS19 Dataset . . . . . . . . 227Figure B.252KiTS19 Class Tumor Metrics for Strong Model andAverage After Aggregation . . . . . . . . . . . . . . . 228Figure B.253KiTS19 Class Tumor AUC for Strong Model and Av-erage After Aggregation . . . . . . . . . . . . . . . . . 228Figure B.254KiTS19 Class Tumor Metrics for Strong Model andAverage Before Aggregation . . . . . . . . . . . . . . 229Figure B.255KiTS19 Class Tumor AUC for Strong Model and Av-erage Before Aggregation . . . . . . . . . . . . . . . . 229Figure B.256KiTS19 Class Tumor Metrics for Strong Model andNoisy Or Aggregation . . . . . . . . . . . . . . . . . . 230Figure B.257KiTS19 Class Tumor AUC for Strong Model and NoisyOr Aggregation . . . . . . . . . . . . . . . . . . . . . 230Figure B.258KiTS19 Class Tumor Metrics for Strong Model andVote Aggregation . . . . . . . . . . . . . . . . . . . . 231Figure B.259KiTS19 Class Tumor AUC for Strong Model and VoteAggregation . . . . . . . . . . . . . . . . . . . . . . . 231Figure B.260KiTS19 Class Tumor Layer Outputs for Strong Model 232xxviiiLIST OF FIGURESFigure B.261Tumor F1 Score for threshold 0.25, Strong Learners,Average After Aggregation, and KiTS19 Dataset . . . 233Figure B.262Tumor F1 Score for threshold 0.20, Strong Learners,Average Before Aggregation, and KiTS19 Dataset . . 233Figure B.263Tumor F1 Score for threshold 0.93, Strong Learners,Noisy Or Aggregation, and KiTS19 Dataset . . . . . 234Figure B.264Tumor F1 Score for threshold 0.00, Strong Learners,Vote Aggregation, and KiTS19 Dataset . . . . . . . . 234Figure B.265KiTS19 Kidney - Case 00000, Slice 28 . . . . . . . . . 235Figure B.266KiTS19 Tumor - Case 00012, Slice 40 . . . . . . . . . 236Figure B.267KiTS19 Kidney - Case 00003, Slice 62 . . . . . . . . . 237Figure B.268KiTS19 Tumor - Case 00003, Slice 62 . . . . . . . . . 238Figure B.269Severstal Class 1 - 3fdc95c42 . . . . . . . . . . . . . . 239Figure B.270Severstal Class 2 - 17ac2163c . . . . . . . . . . . . . . 240Figure B.271Severstal Class 3 - 0ebb34d89 . . . . . . . . . . . . . 241Figure B.272Severstal Class 4 - 5c8104408 . . . . . . . . . . . . . . 242xxixList of Equations2.0 A Computer Vision Task . . . . . . . . . . . . . . . . . . . . . . 42.1 2D Cross Correlation . . . . . . . . . . . . . . . . . . . . . . . . 72.2 2D Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3 Multiple Channel Convolution . . . . . . . . . . . . . . . . . . . 92.4 Multi-Dimensional Gaussian Filter . . . . . . . . . . . . . . . . . 102.5 Two-Dimensional Gaussian Filter . . . . . . . . . . . . . . . . . 112.6 One-Dimensional Gaussian Filter . . . . . . . . . . . . . . . . . . 112.7 Evaluation Metric . . . . . . . . . . . . . . . . . . . . . . . . . . 132.8 Accuracy Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.9 Precision Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.10 Recall Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.11 Specificity Metric . . . . . . . . . . . . . . . . . . . . . . . . . . 152.12 Generalized F-Score . . . . . . . . . . . . . . . . . . . . . . . . . 152.13 Sørensen–Dice coefficient . . . . . . . . . . . . . . . . . . . . . . 152.14 Jaccard Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.15 The Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.16 The Multi-Layer Perceptron . . . . . . . . . . . . . . . . . . . . 182.17 Linear Activation Function . . . . . . . . . . . . . . . . . . . . . 192.18 The Rectified Linear Unit . . . . . . . . . . . . . . . . . . . . . . 202.19 The Leaky Rectified Linear Unit . . . . . . . . . . . . . . . . . . 202.20 The Exponential Linear Unit [11] . . . . . . . . . . . . . . . . . 202.21 Swish Activation [53] . . . . . . . . . . . . . . . . . . . . . . . . 202.22 Sigmoid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.23 Hyperbolic Tangent . . . . . . . . . . . . . . . . . . . . . . . . . 212.24 General Loss Function . . . . . . . . . . . . . . . . . . . . . . . . 232.25 Binary Cross-Entropy Loss . . . . . . . . . . . . . . . . . . . . . 242.26 Binary Focal Loss . . . . . . . . . . . . . . . . . . . . . . . . . . 252.27 Dice Loss [31] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.28 The Chain Rule of Calculus . . . . . . . . . . . . . . . . . . . . . 262.29 Noisy Ground Truth . . . . . . . . . . . . . . . . . . . . . . . . . 272.30 Optimization Goal . . . . . . . . . . . . . . . . . . . . . . . . . . 29xxxLIST OF EQUATIONS2.31 Initial Weight Distribution [20] . . . . . . . . . . . . . . . . . . . 322.32 Parameter Regularization [22] . . . . . . . . . . . . . . . . . . . 332.33 Decomposed Computer Vision Task . . . . . . . . . . . . . . . . 362.34 Relationship Between Image Size and Network Size . . . . . . . 372.35 The Number of Convolutional Parameters . . . . . . . . . . . . . 382.36 Relationship Between Channels and Network Size . . . . . . . . 382.37 Ensembling to Reduce Average Error [22] . . . . . . . . . . . . . 392.38 A Boosted Learner . . . . . . . . . . . . . . . . . . . . . . . . . . 392.39 Boosted Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.40 Residual Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 404.0 Logits Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 55xxxiList of AcronymsAI Artificial Intelligence. 1ANODE09 Automatic Nodule Detection 2009. 44, 45AR Augmented Reality. 74, 75AUC Area Under Curve. 16, 42, 45, 61CAD Computer Aided Diagnosis. 41CNN Convolutional Neural Network. 22, 23, 26, 27, 36, 38CT Computed Tomography. 45, 47, 73DCNN Deep Convolutional Neural Network. 1DSB17 Kaggle Data Science Bowl 2017. 42, 44, 45, 47, 49, 50FCN Fully-Convolutional Network. 3FN False Negative. 13FP False Positive. 13GPU Graphical Processing Units. 1, 23HCI Human-Computer Interaction. 73HU Hounsfield Units. 47ICLR International Conference on Learning Representation. 32ICS Internal Covariate Shift. 35ILSVRC ImageNet Large Scale Visual Recognition Challenge. 23xxxiiList of AcronymsIOU Intersection Over Union. 15, 42IOUS Intraoperative Ultrasounds. 73KiTS19 Kidney Tumor Segmentation Challenge 2019. 1, 2, 43, 49–51, 53,61, 65, 72LUNA16 Lung Nodule Analysis 2016. 44, 45, 49, 50MLP Multi-Layer Perceptron. 17, 18MOT M5Lv1 Multi-Opening and Threshold. 45PRC Precision-Recall Curve. 16, 61, 64, 65ROC Receiver Operating Characteristic Curve. 16SGD Stochastic Gradient Descent. 29–32SNR Signal-to-Noise Ratio. 32TN True Negative. 13TP True Positive. 13VR Virtual Reality. 74, 75XOR Exclusive Or. 17xxxiiiGlossaryAccuracy is the ratio of correct predictions to the total number of predic-tions made. 13Area Under Curve refers to the area present under either the ROC Curveor PRC. Values can range from 0 to 1; however, a value of 0.5 or belowshows that the classifier is unable to meaningfully distinguish classes.16Classification is the process of mapping input data into a single member ofa set such that similar inputs are mapped to the same member anddifferent inputs are mapped to different members. 4Computer Vision is the discipline of teaching computers to understand im-ages. 1, 3Computer vision tasks are methods for transforming digital images into arepresentation easily understood by a computer. 3Convexity is a property of a function when, for any two points in the func-tion, a straight line (or plane or hyperplane in higher dimensions) join-ing those points would not intersect the function at any other point[46]. 24Convolution is a similar operation to cross-correlation, but the kernel isflipped left-to-right before calculating the sliding dot product. Flip-ping the kernel makes the operation commutative. 7Cross-correlation is a measure of how similar two matrices are as a functionof displacement between the two matrices. 6Cross-Validation is an evaluation technique where multiple trained modelsare produced; each model holds back a different portion of the overalldataset for validation. 35xxxivGlossaryDeep Learning is a family of machine learning techniques which utilize neu-ral networks with many hidden layers. 1, 3Dice Score (also called the Dice Similarity Coefficient, or Sørensen-Dice Co-efficient) is the F1 score and the harmonic mean of precision and recall.15Differentiable Function is one where the derivative is defined at all pointsin the function. 24Epoch is a training iteration over the entire dataset, done to make sure thatall examples equally contribute to the loss. 29Evaluation metrics produce a scalar rating M of a computer vision taskgiven prediction Yˆ and ground truth Y. 13F-Score is a weighted combination of precision and recall and a commonlyused evaluation metric in classification tasks. The weighting betweenthese two values is controlled by parameter β such that recall is βtimes more important than precision. 15Feed forward network is one where no feedback is present, that is no neuronsform a cycle. A feed forward network can be represented as a directedacyclic graph. 26Fold refers a combination of training and validation data used in cross-validation. 35Gradient Boosting is a method for combining weak learners such that theoverall loss is minimized. This usually means that subsequently trainedlearners are encouraged to correct mistakes made by their predecessors.39Ground truth Y is the perfect result of a task. 4Image Processing is the discipline of manipulating or enhancing images. 3Jaccard Index is the ratio of correct predictions to number of predictionsplus the number of expected predictions from the ground truth. 15Logistic activation functions return values in the range [0, 1] or [−1, 1] witha sharp transition near x = 0. 21xxxvGlossaryLoss L is a scalar value calculated as a function of Y and Yˆ given a definedloss function L(Y, Yˆ). 23Margin refers to how far a correctly classified pixel is away from being mis-classified. A classification which is only barely correct has a smallmargin, and similar features are more likely to be incorrectly classi-fied in other datasets. A classification with large margin is likely toencourage similar features to also be classified correctly. 24Max Pooling is a dimensionality reduction technique where a section in animage is replaced by the maximum pixel intensity in the section. Itpreserves dominant features while reducing dimensions and allowingfor small spatial translations in the original image. 37Momentum is intuitively named after the physical concept that an object inmotion tends to stay in motion; here, it means that gradient descentwill prefer to continue in the same direction across updates. Math-ematically, momentum is an infinite impulse response low-pass filterwhich produces an exponentially-weighted moving average of the gra-dient [52]. 29Object Detection is a computer vision task where both classification andlocation of one or more objects in the image is produced. 4Object Recognition is a classification task to identify object(s) within theimage, without any location information. 4One-Hot Coding represents information through a group of entries whichtake on values in the set {0,1}. For each group, only one entry isallowed to take on the value 1, and the remaining must take on thevalue 0. It is commonly used in mutually-exclusive classification tasks,where the value 1 shows inclusion to the corresponding class and thevalue 0 shows exclusion. In a one-hot coded segmentation, for an imagewith R rows, C columns, and K classes the segmentation takes on theshape RxCxK. 5Optimization is the process of systematically improving a result throughiteration. 28Overfitting is learning an overly complex function which is able to perfectlyrepresent the training set, but is unable to generalize to any otherdataset. 28xxxviGlossaryPatch is a subsection of a image, usually of fixed size, which can be usedfor training. Smaller patches reduce the size of the overall network.Multiple patches can be created from the same image. 37Patience is the number of epochs without training improvement, or thesize of the plateau, before lowering the learning rate or stopping thetraining run. 31Plateau is when the validation loss does not decrease for a certain numberof epochs. 31Precision measures the likelihood that a prediction of a pixel being in a classis correct. 14Prediction Yˆ is the output of function f and is an approximation the groundtruth Y. 4Radiomics seeks to extract large amounts of quantitative features from med-ical images using data-characterisation algorithms. It is an alternativeto deep learning, breaking the overall task into steps with, potentially,a machine learning classifier at the end. 46Recall measures the likelihood that a pixel which should be in a certainclass is correctly predicted to be in that class. 14Receiver Operating Characteristic Curve is a plot of sensitivity (recall) vs.specificity (false-positive rate). A straight line indicates that the clas-sifier is unable to meaningfully distinguish classes. 16Receptive Field is the area of input that can affect an output pixel. Convo-lution is a localized operation, and for a single convolution operationthe receptive field is the kernel size. Repeating convolutional opera-tions allow the receptive field to spread linearly for a fixed kernel size.The receptive field of a pixel can also be increased by increasing theconvolutional kernel size. Portions of the input that cannot effect apixel are not part of that pixel’s receptive field [45]. 9Rectified Linear Functions are piecewise linear above some input value, butrectified to a constant below that value. 19Regression is the process of mapping input data into one or more continuousoutput values designed to indicate the strength of statistical relation-ship between the input and output(s). 4xxxviiGlossaryRegularization is a loosely defined term referring to any strategy which re-duces the generalization error of a learning algorithm without overfit-ting. 33Segmentation is a binary classification task where each pixel in the outputwill be classified as either part of the class or part of the background.5Self-gated activation functions are able to approximate either a linear ofrectified linear function. 20Semantic Segmentation is an extension of segmentation which allows formultiple classes in the output. 5Specificity measures how many instances which were not in a class werepredicted to not be in that class. 15Stochastic Gradient Descent is an optimization method which adjusts theloss function through repeat sampling of minibatches. The loss gra-dient is calculated on a minibatch, and the parameters of the modeladjusted slightly to move the output closer to the desired ground truth.This process is repeated until some stopping criteria is reached; usu-ally, the stopping criteria is either a fixed number of iterations or whenthe loss no longer continues to decrease for some number of iterations.30Stride refers to skipping some overlappings of the kernel and image duringthe process of convolution. 8, 36Supervised Machine Learning is the process of fitting a function f to gener-ate prediction Yˆ from input data X; fitting uses known examples (X,Y). 23Transposed convolution is a corollary operation to convolution. It consistsof multiplying a scalar value in the image with the kernel, then over-lapping the resulting matrices to produce the output. 8Underfitting results in a model that does not maximize learning from thetraining set. 28Validation is the evaluation of a model on data not used in training toapproximate the true, generalizable performance of the model. It isused to measure if a model is overfitting. 28, 35xxxviiiAcknowledgementsI would like to acknowledge the hard work and support of my supervisors,Dr. Patricia Lasserre and Dr. Homayoun Najjaran. I would also like tothank Abtin Djavadifar, who worked closely with me on several projectsand often helped to focus idea into reality.This research was enabled in part by support provided by ComputeOntario, WestGrid, and Compute Canada.xxxixDedicationThis work is dedicated to my wonderful wife Cassie and our new sonGriffith. Thank you both for your patience.xlChapter 1IntroductionThe vast parallel compute facilities offered by Graphical Processing Units(GPU) have facilitated a paradigm shift in Artificial Intelligence (AI). Theseadvances have been epitomized by Deep Learning, a family of machinelearning techniques which utilize neural networks with many hidden lay-ers. Where computers used to be deployed primarily on problems whichcould be described precisely with mathematics, machines can now learn byexample; this allows near human-level performance on tasks which appearsuperficially easy to us, but are nearly impossible for mathematics to fullymodel. There is no better example of this advancement than in the field ofComputer Vision, the discipline of teaching computers to understand im-ages. Tasks which had previously required specific, hand-tuned approachesare now better solved by a Deep Convolutional Neural Network (DCNN).The end result is giving computers at least an approximation of intuition,being able to crunch high-dimensional data to produce the most likely result.Image segmentation, a popular topic enabled by rapid advances in DC-NNs, provides a pixel-level labelling of an image. Applications of imagesegmentation are vast and varied, as it provides both object class and fine-grained location information through pixel-wise labelling. Self-driving cars,medical image analysis, and industrial manufacturing are but a few examplesof how segmentation is currently being used. Ulku and Akagunduz providean excellent overview of deep learning-based semantic segmentation architec-tures for 2D images [68], including: encoder-decoder, led by U-Net [10, 55]and Seg-Net [3]; spatial pyramid pooling, which allows different image sizesinto the network [23]; and feature concatenation, popularized by DeepMask[51] and ParseNet [43]. Recently, DeepLabV3+ has emerged as a promisinghybrid method, employing both auto-encoder and spatial-pyramid poolingarchitectures [9].Recent state-of-the-art segmentation results have been achieved by auto-encoder networks. 3D U-Net was applied by Isensee et. al to win boththe Medical Image Segmentation Decathlon [61] and the Kidney TumorSegmentation Challenge 2019 (KiTS19) [25, 26, 30]. DeepLabV3+ set newbenchmarks on both the PASCAL VOC 2012 and Cityscapes datasets [9].1Chapter 1. IntroductionThere is also a trend towards computational efficiency in neural networkprocessing, brought about by economic and environmental interests andoften acutely motivated by constraints in how the networks are deployed.However, the exploration of efficiency has largely focused on the depth andwidth of the network, as well as image size [64]; the efficiency is often forcedby the constrained capacity of mobile devices, where the challenge is toachieve the best result possible on limited hardware [63].Ensembling and gradient boosting are two well-known methods for in-creasing performance through a collection of networks. Ensembling is usedwhen different models produce largely uncorrelated predictions; the combi-nation of these predictions reduces overall error. In practice, it is easy toproduce different predictions by training several models with different data;this is a major advantage of cross-validation [22]. Gradient Boosting was firstapplied to neural networks by Schwenk and Bengio in 1998, and producescorrelated models such that the sum of their losses is minimized; successivemodels are trained using the residual loss of previous models, with each newlearner correcting mistakes of past learners [17, 18, 59]. Recent work byBardirli et al. applied boosting techniques to neural networks for the tasksof binary classification, regression, and ranking, surpassing XGBoost in allcases [2].This thesis seeks to explore the scalability and efficiency of image seg-mentation neural networks by including the techniques of ensembling andgradient boosting. The approach is evaluated on both the Severstal SteelDefect Detection [27] and KiTS19 [15] datasets, though it could easily beapplied to any two-dimensional image segmentation task. The success of theapproach has significant implications for deployed applications; not only isthe overall network much smaller, the miniature sub-networks can each becalculated independently and aggregated with simple addition and averag-ing. This introduces a real possibility for efficient distributed inferencing,decoupling performance from the maximum size calculable by a single pieceof hardware.2Chapter 2BackgroundThis chapter provides the foundation required for proficiency in 2D imagesegmentation. It begins with the definition of a computer vision task, specif-ically the prolific tasks of object recognition, object detection, and segmen-tation. These tasks have been enabled by the rise of the Fully-ConvolutionalNetwork (FCN), and the mathematics of the underlying convolution opera-tion and some applications are presented here as well.Machine learning tasks can be broken into three main components: eval-uation, representation, and optimization. Evaluation is a property of thetask, in this case segmentation, and so is covered first. Deep Learning isa family of machine learning techniques which utilize neural networks withmany hidden layers. Neural networks, and the underlying perceptron, formthe representation of the selected approach; a summary of their develop-ment and theory is presented. Finally, and overview of the optimizationtechniques needed to train neural networks is provided. Optimization is acomplex topic on its own, so the concepts of backpropagation, loss functions,and model fitting introduced here should be considered a starting point.2.1 Computer VisionComputer Vision is the discipline of teaching computers to understandimages. This is a related, but fundamentally different, discipline than ImageProcessing, which focuses on the manipulation and enhancement of images.Examples of image processing tasks include deblurring or denoising images.Image processing is often used to improve image quality for human con-sumption; computer vision algorithms also often employ image processingtechniques.2.1.1 Computer Vision TasksComputer vision tasks are methods for transforming digital images intoa representation easily understood by a computer. Understanding is a verybroad term, and can include many concepts such as the presence of an event,32.1. Computer Visionpresence of an object, predicting missing sections, or estimating propertiessuch as pose or movement of an object. Tasks described in this section arelimited to the identification of an object.Computer vision tasks are often classification or regression tasks. Clas-sification is the process of mapping input data into a single member of aset such that similar inputs are mapped to the same member and differ-ent inputs are mapped to different members. Regression is the process ofmapping input data into one or more continuous output values designedto indicate the strength of statistical relationship between the input andoutput(s). Classification is often preceded by regression; first, the input ismapped to several continuous output values indicating probability of be-longing to each class; next, the most probable class is selected as the finaloutput.Computer vision tasks are, mathematically, a function f of an inputimage X:Yˆ = f(X), (2.1)where Yˆ is an approximation of ground truth Y.Ground truth Y is the perfect result of a task. Prediction Yˆ is theoutput of function f and is an approximation the ground truth Y. Sinceeach pixel in a computer vision task is a dimension, these tasks have highdimensionality, the functions are complex, and the process is prone to error;thus, Yˆ is almost always an approximation of the Y.2.1.1.1 Object Recognition and DetectionObject Recognition is a classification task to identify object(s) within theimage, without any location information. Object Detection is a computervision task where both classification and location of one or more objectsin the image is produced. Usually, object location is produced in the formof a bounding box; ideally, the box is drawn as small as possible to fullyencompass the object in the image. Yˆ is an array of tensors with a lengthequal to the number of objects predicted to be in the image; each tensorconsists of: the member of the set to which this section of the image belongs,and co-ordinates for the box bounding of the section of the image whichcontains the object.42.1. Computer VisionFigure 2.1: Object Detection. From [70].2.1.1.2 SegmentationSegmentation is a binary classification task where each pixel in the out-put will be classified as either part of the class or part of the background.The end result of a segmentation task is a binary mask of the same dimen-sions of the input. This provides both classification and localization, wherethe localization can be precise to the pixel. Yˆ is a matrix of the same di-mensions of X, where each value in Yˆ is from the set {0, 1}; 0 indicates theco-located pixel in the input is not part of the class being segmented, and 1indicates that it is part of the class being segmented.Background removal is a very common application of segmentation,where the foreground object is first segmented and then retained.2.1.1.3 Semantic (Multi-Class) SegmentationSemantic Segmentation is an extension of segmentation which allowsfor multiple classes in the output. Rather than a binary pixel mask withboolean answers to a pixel being part of an object, the pixel mask can nowtake on many values where each value is from a set of possible classes. Yˆis a matrix of the same dimensions of X, where each value in the Yˆ is fromthe set {0, 1, ..., k}; 0 indicates the co-located pixel in the input is not partof any class being segmented, and a value greater than 0 indicates thatthe co-located pixel in the input is part of the corresponding class beingsegmented.Semantic segmentation can also be represented as a one-hot coded maskof classes. One-Hot Coding represents information through a group of entrieswhich take on values in the set {0,1}. For each group, only one entry isallowed to take on the value 1, and the remaining must take on the value0. It is commonly used in mutually-exclusive classification tasks, where the52.1. Computer Visionvalue 1 shows inclusion to the corresponding class and the value 0 showsexclusion. In a one-hot coded segmentation, for an image with R rows, Ccolumns, and K classes the segmentation takes on the shape RxCxK.Figure 2.2: Semantic Segmentation. From [69].2.1.2 Equivalence of Single- and Multi-Class SegmentationSegmentation can be viewed as a specific case of semantic segmentationwhere the number of available classes to choose from is 1. Similarly, a multi-class segmentation can be completed as a composition of multiple single-classsegmentation tasks.Single- and multi-class segmentation produce different shapes for Yˆ. Asingle-class segmentation for an image with R rows and C columns takes onthe shape RxCx1. A one-hot coded multi-class segmentation for the sameimage and K classes takes on the shape RxCxK. This can be viewed asK RxCx1 single-class segmentations. Similarly, if K single-class segmenta-tion tasks are run on the same image, and the resulting outputs stacked,a RxCxK matrix is produced; this is the equivalent of the multi-class seg-mentation result.There is an edge case where the one-hot coding restrictions are not met inthe stacked, single-class segmentation representation. When two single-classsegmentations both predict a pixel’s inclusion in mutually exclusive classes,a conflict resolution strategy may need to be devised. However, based onthe intended application, it’s also possible that a pixel’s predicted inclusionin two different classes is not an issue.2.1.3 Cross-Correlation and ConvolutionCross-correlation is a measure of how similar two matrices are as a func-tion of displacement between the two matrices. In image processing, the62.1. Computer Visionlarger matrix is the image and the smaller is called the kernel. Conceptu-ally, the kernel is moved across all points in the image; at each point thesimilarity is calculated as the dot product of the matrices. This operationis called the sliding dot product. Mathematically, cross-correlation isI ? J =∑m∑nI(i+m, j + n)K(m,n), (2.2)where I is the image,K is the kernel,i is the number of columns in the image,j is the number of rows in the image,m is the number of columns in the kernel,and n is the number of rows in the kernel.Convolution is a similar operation to cross-correlation, but the kernel isflipped left-to-right before calculating the sliding dot product. Flipping thekernel makes the operation commutative. The equation for convolution isI ∗K =∑m∑nI(i−m, j − n)K(m,n), (2.3)where I is the image,K is the kernel,i is the number of columns in the image,j is the number of rows in the image,m is the number of columns in the kernel,and n is the number of rows in the kernel.The flipping of the kernel can be seen in the equation by changing theaddition operator to subtraction [22].2.1.3.1 Image BoundariesDuring convolution or cross correlation, when the center of the kernel isat the edge of the image, a portion of the kernel hangs over the edge. Thereare no image values to multiply in such a case, so an alternate approachneeds to be adopted. There are two possible options: first, one can padthe edges of the image with some value or values, and use those values as72.1. Computer Vision0 1 1 1 0 0 00 0 1 1 1 0 00 0 0 1 1 1 00 0 0 1 1 0 00 0 1 1 0 0 00 1 1 0 0 0 01 1 0 0 0 0 01 0 10 2 01 0 11 5 4 5 11 2 5 4 41 2 4 5 11 4 4 1 14 4 1 1 0∗ =IKI ∗K×1 ×0 ×1×0 ×2 ×0×1 ×0 ×1Figure 2.3: Convolution. An example convolution operation with Image I,Kernel K, and result I ∗K.the image values in the convolution operation. Examples are constant valuepadding with either zero or the mean intensity of the image, or reflectingthe image across its axis and using the resulting values. Another method isto simply not allow the kernel to go to the edge of the image; that is, stopwhen the edge of the kernel aligns with the edge of the image. Notably, thelatter solution results in an output image which is smaller than the originalimage.During transposed convolution, when the center of the kernel is at theedge of the image, the portion hanging over the edge is possible to calculate.Values in this case take on the kernel values at the edge multiplied by thescalar values in the image. In this case, the output can be larger than theoriginal image.2.1.3.2 Transposed Convolution and StrideTransposed convolution is a corollary operation to convolution. It con-sists of multiplying a scalar value in the image with the kernel, then over-lapping the resulting matrices to produce the output.Stride refers to skipping some overlappings of the kernel and image dur-ing the process of convolution. This is an equivalent operation to calculatingfull convolution, then keeping only every nth value, where n is the width ofthe stride. Since only every nth value needs to be calculated, this is moreefficient than performing the convolution and downsampling separately.82.1. Computer Visionx1 x2 x3 x4 x5y1 y3 y5Figure 2.4: Example 2-Strided Convolution. This is equivalent to perform-ing full convolution, then keeping only every 2nd value.When using strided transposed convolution, the output is larger thannon-strided convolution. In the deep learning world, this process has takenthe unfortunate name of deconvolution. The term deconvolution has beenused for decades to refer to inverse convolution. That is, the process ofreversing the convolution operation, or restoring the original image afterconvolution has been applied. Applications of this deconvolution includedeblurring images or removing noise. This is not the same as strided trans-posed convolution, which is commonly used for upsampling in deep learning.2.1.3.3 Multi-Channel ConvolutionIt is possible to convolve to and/or from an arbitrary number of channels.A color image, for example, is usually represented as 3-channels where themeaning of the channels depends on the colour space. WithN input channelsand M output channels, the convolution requires N ·M convolutional kernelsCˆM =∑i∈{0,1,...,N}Ci ∗Ki,M . (2.4)where the summation is piecewise addition of the convolution outputs.The resulting output channels are then stacked to produce the final output[14]. The number of convolution operations is O(N ·M) ≈ O(n2) in networkwidth.2.1.4 Receptive FieldThe Receptive Field is the area of input that can affect an output pixel.Convolution is a localized operation, and for a single convolution operationthe receptive field is the kernel size. Repeating convolutional operations al-low the receptive field to spread linearly for a fixed kernel size. The receptivefield of a pixel can also be increased by increasing the convolutional kernel92.1. Computer Visionsize. Portions of the input that cannot effect a pixel are not part of thatpixel’s receptive field [45].a1 a2 a3 a4 a5b1 b2 b3 b4 b5c1 c2 c3 c4 c5Figure 2.5: The Receptive Field of Convolution. Through two successiveconvolution operations, the receptive field of C3 is five input pixels.2.1.5 Common Convolution KernelsConvolution is a widespread operation in image processing and computervision. Some basic functions and corresponding kernels are described hereto illustrate the power of operations based on local information in the image.2.1.5.1 Low-Pass Filtering KernelsLow-pass filtering kernels are used to remove high-frequency componentsfrom an image. High-frequency components are rapid spatial-intensity-changes such as sharp edges or noise. These filters blur the image, generallyreducing noise and sharpness of edges.An averaging kernel has all elements in the kernel set to 1n where n isthe size of the kernel. In the example shown in Figure 2.6, filtering reducesthe standard deviation in the resulting image from 0.5 to 0.14. The centerpixel, previously an strong outlier from its neighbours, now takes on thesame value as its neighbouring pixels.A similar result to the averaging kernel is achieved by applying a Gaus-sian kernel as shown in Figure 2.7. The Gaussian function,G(X;σ) =1(√2piσ)Ne−|X|22σ2 , (2.5)exhibits many desirable properties mathematically; most notably, the deriva-tive of a Gaussian function is itself a Gaussian function. Moreover, a two-102.1. Computer Visiondimensional Gaussian kernel,G(x, y;σ) =12piσ2e−x2+y22σ2 , (2.6)is separable; that is, it can be represented as two one-dimensional Gaussiankernels,G(x;σ) =1√2piσe−x22σ2 . (2.7)The two Gaussian kernels can be applied in series to achieve the same resultas a single two-dimensional kernel. The separated kernels are more efficientto compute, and there is usually a preference for a Gaussian kernel to beused for low-pass filtering when possible.0 0 0 0 0 0 00 1 1 1 1 1 00 1 1 1 1 1 00 1 1 0 1 1 00 1 1 1 1 1 00 1 1 1 1 1 00 0 0 0 0 0 00.11 0.11 0.110.11 0.11 0.110.11 0.11 0.110.44 0.66 0.66 0.66 0.440.66 0.88 0.88 0.88 0.660.66 0.88 0.88 0.88 0.660.66 0.88 0.88 0.88 0.660.44 0.66 0.66 0.66 0.44∗ =IKI ∗K×0.11×0.11×0.11×0.11×0.11×0.11×0.11×0.11×0.11Figure 2.6: 3x3 Averaging Kernel.0 0 0 0 0 0 00 1 1 1 1 1 00 1 1 1 1 1 00 1 1 0 1 1 00 1 1 1 1 1 00 1 1 1 1 1 00 0 0 0 0 0 00.08 0.12 0.080.12 0.2 0.120.08 0.12 0.080.52 0.72 0.72 0.72 0.520.72 0.92 0.88 0.92 0.720.72 0.88 0.8 0.88 0.720.72 0.92 0.88 0.92 0.720.52 0.72 0.72 0.72 0.52∗ =IKI ∗K×0.08×0.12×0.08×0.12 ×0.2×0.12×0.08×0.12×0.08Figure 2.7: A Gaussian Kernel. A 3x3 intensity-normalized gaussian kernelwith σ = 1.2.1.5.2 Edge Detector KernelsEdge detectors attempt to preserve high-frequency edges while discard-ing other information. The most common edge detectors approximate either112.1. Computer Visionthe first- or second-derivative of the image.In practice, high-pass filters are very sensitive to high-frequency noise.As such, many edge detection techniques use multiple convolutions. Anexample of this is the Laplacian-of-Gaussian method [47], where the edge-detector is applied after a low-pass filter. Another common approach is toemploy the difference of two low-pass filters with different corner frequencies.An example of this is the Difference-of-Gaussian method [47].Edge detection is commonly used in image sharpening. First, the edgesof the image are found, and then summed with the original image. Thisaccentuates the edges in the resulting image, giving a sharpening effect.0 0 0 0 0 0 00 1 1 1 1 1 00 1 1 1 1 1 00 1 1 0 1 1 00 1 1 1 1 1 00 1 1 1 1 1 00 0 0 0 0 0 0−1 −1 −1−1 8 −1−1 −1 −15 3 3 3 53 1 1 1 33 1 −8 1 33 1 1 1 35 3 3 3 5∗ =IKI ∗K×− 1×− 1×− 1×− 1 ×8×− 1×− 1×− 1×− 1Figure 2.8: A Laplacian Kernel. The Laplacian Kernel approximates the2nd derivative, with edges at zero-crossings.0 0 0 0 0 0 00 1 1 1 1 1 00 1 1 1 1 1 00 1 1 0 1 1 00 1 1 1 1 1 00 1 1 1 1 1 00 0 0 0 0 0 01 0 −12 0 −21 0 −1−3 0 0 0 3−4 1 0 −1 4−4 2 0 −2 4−4 1 0 −1 4−3 0 0 0 3∗ =IKI ∗K×1 ×0×− 1×2 ×0×− 2×1 ×0×− 1Figure 2.9: A Horizontal Sobel Kernel. The kernel approximates the 1stderivative in the x-direction.122.1. Computer Vision0 0 0 0 0 0 00 1 1 1 1 1 00 1 1 1 1 1 00 1 1 0 1 1 00 1 1 1 1 1 00 1 1 1 1 1 00 0 0 0 0 0 01 2 10 0 0−1 −2 −1−3 −4 −4 −4 −30 1 2 1 00 0 0 0 00 −1 −2 −1 03 4 4 4 3∗ =IKI ∗K×1 ×2 ×1×0 ×0 ×0×− 1×− 2×− 1Figure 2.10: A Vertical Sobel Kernel. The kernel approximates the 1stderivative in the y-direction.2.1.6 Evaluation MetricsEvaluation metrics produce a scalar ratingM of a computer vision taskgiven prediction Yˆ and ground truth Y,M = M(Y, Yˆ). (2.8)Metrics for semantic segmentation tasks are generally an aggregation ofdiscrete pixel classification results across an image. In a given prediction,each pixel can be one of:− False Positive (FP) where the prediction was an incorrect class.− True Positive (TP) where the prediction was the correct class.− False Negative (FN) where the prediction was incorrectly no class.− True Negative (TN) where the prediction was correctly no class.2.1.6.1 AccuracyAccuracy is the ratio of correct predictions to the total number of pre-dictions made,MAccuracy = TP + TNTP + TN + FP + FN=TP + TNImage Area in Pixels. (2.9)132.1. Computer VisionAn accuracy of 100% means that the prediction matches the groundtruth perfectly; 0% means that none of the predictions matched the groundtruth.Accuracy values between 0% and 100% cannot be interpreted with gen-eral rules or guidelines; the value of an accuracy score is a function of theproperties of the ground truth. For example, in an image where only 10%of the ground truth is taken up by a certain class, an accuracy of 90% canbe achieved simply by guessing that there are no instances of that classanywhere. Obviously, such a classifier has no practical significance despiteachieving what, on the surface, appears to be a very high accuracy score.Because of the this limitation, accuracy is not commonly used as an evalu-ation metric, and should be avoided entirely if the data set has unbalancedclasses.2.1.6.2 PrecisionPrecision measures the likelihood that a prediction of a pixel being in aclass is correct. Mathematically, it is the ratio of correct positive predictionsto the total number of positive predictions made,MPrecision = TPTP + FP. (2.10)Precision is meaningful for data sets with sparse classes as each incorrectprediction is normalized by the total number of predictions made; however,a single correct prediction can yield a precision score of 100% despite a largenumber of false negatives present in the image.2.1.6.3 RecallRecall measures the likelihood that a pixel which should be in a certainclass is correctly predicted to be in that class. It is the ratio of correctpredictions to the total number of instances of that class in the groundtruth,MRecall = TPTP + FN. (2.11)Recall suffers from the lack of consideration of False Positives in themetric. Predicting that the entire image is part of a class will always yielda recall score of 100% for that class.142.1. Computer Vision2.1.6.4 SpecificitySpecificity measures how many instances which were not in a class werepredicted to not be in that class,MSpecificity = TNTN + FP. (2.12)Specificity can be seen as a corollary of recall. A perfect specificity scorecan be trivially achieved by simply not predicting any instances of a class.2.1.6.5 F-score (Sørensen–Dice coefficient)As can be seen with the above evaluation metrics, all can trivially achievehigh scores despite very poor intuitive performance of the model. However,by combining several of these metrics, an evaluation metric can be devisedwhich both improves as the prediction better matches the ground truth andis unable to achieve good performance with trivial predictions. F-Score is aweighted combination of precision and recall and a commonly used evalua-tion metric in classification tasks. The weighting between these two valuesis controlled by parameter β such that recall is β times more important thanprecision,MFScore = Fβ = (1 + β2) · precision · recall(β2 · precision) + recall . (2.13)The Dice Score (also called the Dice Similarity Coefficient, or Sørensen-Dice Coefficient) is the F1 score and the harmonic mean of precision andrecall. [13, 62],MDice = F1 = 2 · TP2 · TP + FP + FN =2 · |Y ∩ Yˆ ||Y |+ |Yˆ | . (2.14)2.1.6.6 Jaccard IndexJaccard Index is the ratio of correct predictions to number of predictionsplus the number of expected predictions from the ground truth. It is alsocommonly called Intersection Over Union (IOU). It is mathematically verysimilar to the Sørensen–Dice coefficient, but the weighting of TP values by2 is omitted,152.1. Computer VisionMJaccardIndex = Y ∩ YˆY ∪ Yˆ =TPTP + FN + FP. (2.15)In practice, it is generally considered better to include a pixel in thecorrect class than to remove a pixel from an incorrect class; as such, theDice Similarity Coefficient is often preferred to the Jaccard Index.2.1.6.7 Area Under the CurveThe Receiver Operating Characteristic Curve is a plot of sensitivity (re-call) vs. specificity (false-positive rate). A straight line indicates that theclassifier is unable to meaningfully distinguish classes. The false positiverate is defined as 1−MSpecificity. It is common to evaluate classifiers usingthe Area Under Curve (AUC). Area Under Curve refers to the area presentunder either the ROC Curve or PRC. Values can range from 0 to 1; however,a value of 0.5 or below shows that the classifier is unable to meaningfullydistinguish classes. For sparse datasets, the number of predictions is smallrelative to the image size, so spurious predictions do not change the false pos-itive rate drastically. A common method for evaluating segmentation tasksis to use the Precision-Recall Curve (PRC), by replacing the false-positiverate with the false-discovery rate in the Receiver Operating CharacteristicCurve (ROC) [68].(a) Receiver Operating Characteristic (b) Precision-Recall CurveFigure 2.11: Example ROC and PRC Curves.162.2. Neural Networks2.2 Neural NetworksThe perceptron, featuring learnable weights, was invented by F. Rosen-blatt in 1958. It was a simplified model of the neuron used by “higherorganisms for perceptual recognition, generalisation, recall, and thinking[56].” Rosenblatt showed that the perceptron is quite powerful in solvinglinearly separable problems. The equation for the perceptron can be writtenasy(x0, ..., xn) = A(wb +n∑i=0xi · wi), (2.16)where x0, ..., xn is the tensor of inputs,w0, ..., wn is the tensor of weights,wb is the bias weight,and A is the activation function.xnx1x01∑wnw1w0wbinputsweightsactivationFigure 2.12: The Perceptron.It was discovered in 1969 that a single perceptron was incapable of mod-elling a simple Exclusive Or (XOR) function [50]. This problem was solveddecades later with the advent of the Multi-Layer Perceptron (MLP), wheremany layers of perceptrons are cascaded together. Each layer maps thefeature space into a different representation, allowing one layer to create alinearly separable representation while the next would perform the separa-tion [22].172.2. Neural Networksxmx01xm+1xn1∑wmw0wb0∑wm+1wnwb1∑wywx1 wb2Figure 2.13: The Multi-Layer Perceptron.The equation for the MLP in figure 2.13 can be written asy = A2(wb2 +A0(wb0 +m∑i=0xi · wi) +A1(wb1 +n∑j=m+1xj · wj)), (2.17)where x0, ..., xn is the tensor of inputs,w0, ..., wn is the tensor of weights,wbn is the bias weight of perceptron n,and An is the activation function of perceptron n.2.2.1 Activation FunctionsActivation functions, an integral part of the neural network, are appliedat the output of each perceptron. Like neural networks, they were initiallymodelled off biological organisms and represented the action potential of acell [56]. Today, many varieties of activation functions have been exploredwhich all introduce different properties into the network.182.2. Neural Networks2.2.1.1 Linear ActivationA linear activation function A maps input to output using a linear equa-tion such as:Alinear(x) = mx+ b. (2.18)The multi-layer perceptron itself becomes a linear operation when allactivation functions used are linear. From the multi-layer-perceptron equa-tion 2.17, when the activation functions are linear, A0, A1, and A2 can bereplaced with constants c0, c1, and c2. Setting m to 1 and b to 0 in all casessimplifies the equation and yieldsy = c2(wb2 + c0(wb0 +m∑i=0xi · wi) + c1(wb1 +n∑j=m+1xj · wj))= c2 ·wb2 +c2 ·c0 ·wb0 +c2 ·c0m∑i=0(xi · wi)+c2 ·c1 ·wb1 +c2 ·c1n∑j=m+1(xj · wj)= K + c2 · c0m∑i=0(xi · wi) + c2 · c1n∑j=m+1(xj · wj),where K is c2 · wb2 + c2 · c0 · wb0 + c2 · c1 · wb1 .To approximate a non-linear function, the activation function used shouldbe non-linear.2.2.1.2 Rectified Linear Unit ActivationRectified Linear Functions are piecewise linear above some input value,but rectified to a constant below that value. In computer vision, it is com-mon that pixels can only take on intensity values greater than zero; this canbe modelled with a rectified linear function, which makes these functionsintuitively valuable for computer vision applications.The theoretical advantage of a rectified function is the promotion ofsparse connections [21]. As the authors state, sparsity, in theory, allowsthe neural network to generalize better through information disentangling.192.2. Neural NetworksThat is, in a sparse network, small changes in single neurons are not able tocascade widely throughout subsequent layers.The ReLU function,AReLU (x) ={0 x ≤ 0x x > 0, (2.19)promotes sparsity by rectifying values in Quadrant III to 0. Leaky ReLU,ALeakyReLU (x) ={βx x ≤ 0; 0 < β < 1x x > 0, (2.20)allows some transmission of values in Quadrant III, scaled by tunable pa-rameter β. The exponential linear unit (ELU) function,AELU (x) ={α · ex − 1 x ≤ 0x x > 0, (2.21)models Leaky ReLU, but is continuous whereas ReLU and Leaky ReLU arediscontinuous at X = 0.Figure 2.14: Example Rectified Linear Unit Activation Functions.2.2.1.3 Self-Gated Activation FunctionsSelf-gated activation functions are able to approximate either a linearof rectified linear function. Work by Ramachandran et al. discovered theSwish activation function,202.2. Neural NetworksAswish(x) = x · σ(βx), (2.22)where σ(z) is 11+e−z ,and β is a tunable parameter.When β = 0, the activation becomes f(x) = x2 . When β →∞, the σ(βx)component approximates a unit step function, and the overall activationapproximates ReLU. The authors suggest a β value near 1.0, confirmedthrough experimentation in learning β during training.Figure 2.15: Swish Activation.2.2.1.4 Logistic ActivationLogistic activation functions return values in the range [0, 1] or [−1, 1]with a sharp transition near x = 0. These activation functions are commonlyused when assigning probabilities to classes, as they can be seen to saturatevery quickly at one of the two possible values. The sigmoid function,ASigmoid(x) =exex + 1, (2.23)is widely used as it takes the range [0, 1] which matches probability values.LeCun et al. showed that the non-zero mean of the sigmoid function canbe problematic [39]. An alternative which saturates between [-1, -1] is thehyperbolic tangentAtanh(x) =ex − e−xex + e−x. (2.24)212.2. Neural NetworksFigure 2.16: Example Logistic Activation Functions.2.2.1.5 Full-Domain ActivationsFull-domain activations can be broadly divided into three further sub-categories. The first category has a non-linearity near x=0, but becomespseudo-linear for values |x| >> 0. The second category is made up of non-linear polynomials which are able to realize the full range of values and arenon-linear at all points; functions in this category explode very quickly forsmall changes in the input, which generally yields unstable outputs. Thethird category is a linear function summed with a periodic function, whichis also non-linear at all points.(a) Pseudo-Linear (b) Polynomial (c) Linear-PeriodicFigure 2.17: Example Full Domain Activation Functions.2.2.2 Convolutional Neural NetworksYann LeCun used the power of local information to create the Convo-lutional Neural Network (CNN) in 1989 [38]. Motivated by the power ofconvolution, he introduced the key concepts of sparse connections and pa-rameter sharing to neural networks. The end result were networks where222.3. Training using Supervised Machine Learningeach neuron was connected only to its neighbours, and weights were sharedacross all neurons in the layer. This reduced the computational complexityand memory requirements of neural networks, ultimately leading to theirwidespread practical adoption.CNNs are significantly computationally simpler than the fully-connectednetworks, with computation complexity of O(k × n) rather than O(m× n)(m inputs, n outputs, k connection limit). Moreover, memory requirementsare reduced to the number of connection parameters; these parameters areshared, similar to how a 2D kernel uses the same kernel values as it is movedacross pixels.Before neural networks could emerge as a leading contender in imageprocessing, a technical advancement was necessary. The ImageNet LargeScale Visual Recognition Challenge (ILSVRC) was won in 2012 by AlexNet,which was one of the first entry of a neural network computed on a GPU.Since that victory, GPUs have been widely adopted to process CNNss andtheir success in computer vision tasks has been dominant.2.3 Training using Supervised Machine LearningSupervised Machine Learning is the process of fitting a function f togenerate prediction Yˆ from input data X; fitting uses known examples (X,Y). f is found through a training process which minimizes loss L over allexamples.2.3.1 LossLoss is a scalar value calculated as a function of Y and Yˆ given a definedloss functionL = L(Y, Yˆ). (2.25)When Yˆ and Y match exactly, the loss should be at the global minimum;when Yˆ and the Y differ significantly, the loss value should be high.2.3.1.1 Loss Function PropertiesLoss is used in the training process, and as such is subject to any limita-tions enforced by the training process employed. Evaluation metrics, on theother hand, are not used in optimization; as such, they do not suffer fromthe same limitations. Thus, evaluation metrics are more indicative of true232.3. Training using Supervised Machine Learningperformance by allowing operations such as thresholding. Common proper-ties which distinguish loss functions from evaluation metrics are describedhere.2.3.1.1.1 Continuous Loss functions for computer vision tasks wereexplored by Masanadi-Shirazi et. al [48]. They state that continuous lossfunctions are useful for training a neural network, as they promote movementaway from the incorrect case, but also promote movement away from acorrect, but highly uncertain prediction; this helps to produce a classifier ofmaximum margin.2.3.1.1.2 Differentiability A Differentiable Function is one where thederivative is defined at all points in the function. A differentiable loss func-tion allows some understanding of how L changes as a function of Yˆ. Thiscan inform the training process, which attempts to vary Yˆ in such a waythat L is minimized across all examples. This allows a more systematictraining than random guessing or grid search.2.3.1.1.3 Convexity Convexity is a property of a function when, forany two points in the function, a straight line (or plane or hyperplane inhigher dimensions) joining those points would not intersect the function atany other point [46]. Convexity guarantees that the global optimum canbe reached from any starting point through monotonically increasing ormonotonically decreasing the output. When the function is convex, everylocal minimum is also a global minimum [6]. When the function is non-convex, no such guarantee necessarily exists, so there is a possibility ofoptimization finding only a local minimum.2.3.1.2 Binary Cross-Entropy (Log Loss)Binary Cross-Entropy is also called log loss. Each pixel is comparedwith the ground truth on a logarithmic scale, then summed to produce lossacross the image. The non-linearity of the logarithmic function encouragesimproving the margin of the classifier. Margin refers to how far a correctlyclassified pixel is away from being misclassified. A classification which is onlybarely correct has a small margin, and similar features are more likely to beincorrectly classified in other datasets. A classification with large margin islikely to encourage similar features to also be classified correctly.242.3. Training using Supervised Machine LearningLBinaryCrossEntropy = − 1nn∑i=1[yi log (yˆi) + (1− yi) log (1− yˆi)], (2.26)where yˆi is the predicted class at pixel i,and yi is the ground-truth class at pixel i.2.3.1.3 Binary Focal LossBinary focal loss is an extension of binary cross entropy, adding a tunableparameter which can be used to downplay the significance of classificationsas they improve,LBinaryFocal = − 1nn∑i=1[(1−yˆi)γ ·yi log (yˆi)+(yˆi)γ ·(1−yi) log (1− yˆi)], (2.27)where yˆi is the predicted class at pixel i,yi is the ground-truth class at pixel i,and γ is a tunable focusing factor.This is particularly important in sparse or unbalanced datasets. By down-playing the significance of the over-represented class, optimization will morelikely prefer to reduce greatly misclassified examples in the sparse set [42].Figure 2.18: Binary Focal Loss.252.3. Training using Supervised Machine Learning2.3.1.4 Dice LossDice Loss is similar to the Dice Score (see section 2.1.6.5); however,where the Dice Score is calculated on discrete predictions in the set {0, 1},Dice Loss is calculated on continuous probabilities in the range [0, 1]:Ldice =∑i∈I yˆiyi∑i∈I yˆi +∑i∈I yi, (2.28)where yˆi is the predicted class at pixel i,and yi is the ground-truth class at pixel i.Higher probabilities for true positive predictions yield a higher numera-tor, whereas additional predictions as false positives increase the denomina-tor. Using continuous probabilities rather than discrete predictions makesthe overall function differentiable.2.3.2 Back PropagationBackpropagation was discovered by Y. LeCun in 1989. In two seminalpapers, LeCun defined the sparse connections and parameter sharing whichare now widely adopted in CNNs, developed backpropagation, and appliedthe techniques to perform digit recognition in a hand-drawn number dataset[37, 38].A neuron not firing as expected can be corrected by either correcting theweights directly entering the neuron, or by changing the firing of connectedneurons in the previous layer. This is repeated for every neuron, until allfire as expected. Mathematically, this is represented by the chain rule ofcalculus,dydx=dydududx. (2.29)Given a scalar loss value L = L(Y, Yˆ) and prediction Yˆ = f(X), we canapply the chain rule of calculus to represent the change in L as a function ofthe change in X: dLdX =dLdYˆdYˆdX . The gradient of L as a function of outputsYˆ can be calculated directly if the loss function is differentiable. The chainrule can then be applied to each neuron in the preceding layer.In the example shown in Figure 2.19, a single neuron is present in eachlayer. For simplicity, the network is assumed to be feed forward. A Feed262.3. Training using Supervised Machine Learningy0A(a0)a0A(b0)b0A(x0)x0dy0da0da0db0db0dx0dy0db0dy0dx0××f ′f ′f ′Figure 2.19: Back Propagation. This is a modified version of the graphicpresented in [22].forward network is one where no feedback is present, that is no neuronsform a cycle. A feed forward network can be represented as a directedacyclic graph. CNNs are feed-forward networks. The chain rule is thenapplied recursively through the network until dYˆdX is found.2.3.3 Fitting a ModelThe goal in machine learning is to produce a generalizable function.When noise is considered, the ground truth can be represented asY = f(X) + , (2.30)where f(X) is the learnable portion of the ground truth,and  is non-learnable noise.The goal is to learn function f(X), which is assumed to be constantacross all training examples. The noise, however, changes across samples.A perfect model would learn the underlying function without modelling anynoise. In practice, however, it is very difficult to separate the underlyingfunction from the noise. Every learning approach makes a trade-off between272.3. Training using Supervised Machine Learningthese two pieces; in general, better learning of the function will include morenoise.Overfitting is learning an overly complex function which is able to per-fectly represent the training set, but is unable to generalize to any otherdataset. When the function is overfit, the noise from the training set islearned; this appears, incorrectly, to be improvement while training. How-ever, since the noise across data sets is highly unlikely to be the same, theend result is worse when applied to new data.There are many techniques to help combat overfitting, but the only wayto measure its impact on a model is to test the model’s generalizabilitythrough validation. Validation is the evaluation of a model on data notused in training to approximate the true, generalizable performance of themodel. It is used to measure if a model is overfitting.Carune, Lawrence, and Giles explored the overfitting of neural networks;they concluded that large models are more likely to overfit than smaller,properly sized models, as the larger network is able to model a more complexfunction. However, they concluded that the larger network first learns afunction comparable to a well-sized network, and then additional trainingleads to overfitting. Their recommendation is to employ early stopping,which uses validation to stop training when the model reaches minimal losson a data set unseen in training [8].Underfitting results in a model that does not maximize learning fromthe training set. Usually, underfitting occurs as a response to the fear ofoverfitting. Unlike overfitting, however, underfitting is very hard to measure.Overfitting can be quantified as the difference in performance between thetraining set and some unseen validation set. Quantifying underfitting wouldrequire the difference in performance between the training set and somehypothetical perfect model. Of course, if such a perfect model existed, itwould benefit the researcher to simply use it, which would render the processof learning redundant. Needless to say, in almost all cases such a model doesnot exist. Another perspective is that proving that a model is not underfitrequires proving that no better model exists; this is an unfeasible task whichwould require testing all models.2.3.4 OptimizationOptimization is the process of systematically improving a result throughiteration. Optimization in supervised machine learning relies heavily on alabelled data set; examples are used to generate predicted output, and acomparison of the prediction and ground truth identifies errors which can282.3. Training using Supervised Machine Learningbe minimized.Given a set of examples {X0,X1,X2, ...Xn}, the goal is to minimizeaverage loss over the entire setL = 1NN∑n=1L(Yn, Yˆn), (2.31)where N is the number of examples in the training set.Where the loss is calculated as a single scalar, the change in loss as afunction of X, dLdX , is found through backpropagation. For a single example,it is trivial to minimize the loss by simply applying the calculated gradient.For larger training sets, however, this is a futile task as adjusting the functionfor a perfect result to a single example will almost certainly increase theloss for other examples. A better algorithm needs to be devised which canreliably provide good results across the entire training set.A deterministic approach would be to process the entire training set atthe same time, then find the average gradient for all examples. However,this is both computationally inefficient and produces diminishing returns,as the standard error of the mean is inversely proportional to the square-root of the number of samples in the batch. Because of this non-linearrelationship between the number of samples used to produce the gradientand the reduction in the expected error, batching algorithms are often usedwhich employ a subset of the training samples in each iteration.2.3.4.1 Stochastic Gradient DescentStochastic Gradient Descent (SGD) is an optimization method whichadjusts the loss function through repeat sampling of minibatches. The lossgradient is calculated on a minibatch, and the parameters of the modeladjusted slightly to move the output closer to the desired ground truth.This process is repeated until some stopping criteria is reached; usually,the stopping criteria is either a fixed number of iterations or when the lossno longer continues to decrease for some number of iterations. Training isoften done in iterations over the full training set, with opportunity to stoptraining only between these iterations; this helps ensure that all examplesequally contribute to the loss. Such an iteration is called an Epoch.Note parameter V in Algorithm 1. This parameter is momentum. Mo-mentum is intuitively named after the physical concept that an object in292.3. Training using Supervised Machine LearningAlgorithm 1: Stochastic Gradient DescentInput: Learning Rate 1 W← sampled randomly from intialization distribution ;2 V← zero matrix ;3 while stopping criterion not met do4 Sample a minibatch from the training set {X0,X1, ...Xm} ;5 gˆ← 1m∑mi=0∇WL(Yi, f(Wk,Xi)) ;6 V← γV + (1− γ) ·  · gˆ ;7 W←W −V ;8 endmotion tends to stay in motion; here, it means that gradient descent willprefer to continue in the same direction across updates. Mathematically,momentum is an infinite impulse response low-pass filter which produces anexponentially-weighted moving average of the gradient [52].The infinite impulse response filter is bounded-input, bounded-outputstable if 0 ≤ γ ≤ 1. When γ = 1, the gradient updates are rejected com-pletely; when γ = 0, no filtering is performed, and thus no momentum ispreserved. SGD without momentum can then be seen as a specific case ofSGD with momentum where γ = 0.Stochastic Gradient Descent handles samples which are corrupted bynoise relatively well. Recall from Equation 2.30 that learning examplescan be split into a generalizable function and non-generalizable noise. Ifthe minibatch were to consider all training examples concurrently, the noisewould be constant in each iteration; it would then be easy to perfectly modelboth the function and the error, leading to a well-performing, but poorlygeneralizable model. Random minibatching causes the noise to vary acrossiterations, and thus the optimizer does not have as much opportunity tooverfit.2.3.4.2 Adaptive OptimizersStochastic Gradient Descent can be slow to converge. One possiblemethod of acceleration is to use a high learning rate. While this can leadto faster convergence, at least initially, the final stages of learning sufferas optimization can jump across critical points without descending to theminimum; in the worst case, the result can actually diverge as the optimizerclimbs by jumping back and forth across the function on either side of thecritical point. A common technique to combat this is to reduce the learning302.3. Training using Supervised Machine Learningrate over time. Two common methods for managing the learning rate areto use a fixed schedule or to decrease the learning rate on a plateau. APlateau is when the validation loss does not decrease for a certain numberof epochs. Patience is the number of epochs without training improvement,or the size of the plateau, before lowering the learning rate or stopping thetraining run.Even if the perfect learning rate strategy is found through a notoriouslydifficult hyperparameter tuning task, the learning rate for SGD is a constantvalue for all parameters to be learned. It is easy to envision a situation wheresome, even most, of the parameters of a model are less important to the finaloutput than a select few. Momentum is one strategy to help deal with such asituation, where parameter updates will continue to change those parameterswhich usually change rapidly. Another strategy is to maintain separatelearning rates for each parameter, and to somehow change the learning ratefor each parameter independently. Such is the strategy adopted by adaptiveoptimizers.Algorithm 2: AdamInput: Learning Rate Input: Exponential Decay Rate for First Moment β1Input: Exponential Decay Rate for Second Moment β21 W← sampled randomly from intialization distribution ;2 s← zero matrix ;3 r← zero matrix ;4 t← 0 ;5 while stopping criterion not met do6 t← t+ 17 Sample a minibatch from the training set {X0,X1, ...Xm} ;8 gˆ← 1m∑mi=0∇WL(Yi, f(Wk,Xi)) ;9 s← β1s + (1− β1)gˆ ; // 1st order moment (mean)10 r← β2r + (1− β2)gˆ2 ; // 2nd order raw moment (variance)11 sˆ← s1−βt1 ;12 rˆ ← r1−βt2 ;13 W←W −  sˆ√rˆ; // Element-wise square-root, division14 endOne extremely popular adaptive optimizer is Adam. Developed in 2014by Kingma and Ba [35], it is actually an extension and combination of several312.3. Training using Supervised Machine Learningideas which were introduced in the early 2010s. Rather than introduce allsuch developments here, only Adam itself will be discussed and the readershould consult one of the many resources which cover this topic for furtherinformation [22, 57, 76].Parameters s and r from Adam should be recognizable from StochasticGradient Descent. s is a new symbol for the concept of momentum; simi-larly, β1 is γ, a low-pass filtering parameter on momentum. r is similar tomomentum; it is a low-pass filtered square of the gradient, also called thesecond raw moment.The authors of the Adam paper refer to the parameter sˆ√rˆas an ap-proximation of the Signal-to-Noise Ratio (SNR). When optimization nearsa critical point, uncertainty builds about whether the calculated gradientis a good approximation of the true gradient; this uncertainty yields to asmaller SNR value and thus smaller updates [35]. A statistical interpretationis that the term approximates the inverse of the coefficient of variation. Thecoefficient of variation is a measure of population variation relative to themean. Higher variation implies higher uncertainty; by inverting the ratio,this leads to smaller updates when uncertainty is high.There is controversy in the deep learning community as to which opti-mizer to use. Adaptive optimizers are widely used because of their speedof convergence. However, recent works have shown that in some cases theyconverge to a worse value than SGD with momentum, or even diverge insome circumstances because of extremely large or small learning rates atthe end of training [54, 77]. Recent work claims to have reconciled theseissues through an optimizer which begins with adaptive techniques whilesmoothly moving to SGD as training progresses; this paper was acceptedto International Conference on Learning Representation (ICLR) 2019, butwidespread adoption will likely be needed to determine its practicality [44].2.3.5 The Vanishing / Exploding GradientThe gradient in multi-layered neural networks has an exponential rela-tionship with previous layers. This leads to a gradient which either explodesquickly to infinity or vanishing quickly to zero [4, 20].Glorot and Bengio show through experimentation that correct distri-bution of the initial weights can cause the activation value histogram tospread out and remain relatively similar across all layers. They suggest arandomized weight initialization bounded by the properties of the network:322.3. Training using Supervised Machine LearningW ∼ U[−√6√nj + nj+1,√6√nj + nj+1], (2.32)where nj is the number of neurons in layer j,and U is the uniform distribution.Interestingly, the authors equate the exploding gradient issue with acti-vation values clustered near 0 [20]. More recent work, however, has shownthat for extremely deep networks, correct initialization may not be sufficientto keep the gradient stable. He et al. attempted to rectify this issue byadding skip layers, or summing the inputs of a layer with its convolutionaloutput to produce the final output. As a result, rather than optimizingthe convolutional output of the layer F(x), optimization is performed onF(x) + x [24]. Using the residual learning building block, the authors ex-plored the viability of making very deep neural networks. They concludedthat error on the CIFAR-10 test set was minimized when 110 layers wereused [24].2.3.6 RegularizationRegularization is a loosely defined term referring to any strategy whichreduces the generalization error of a learning algorithm without overfitting.The concept of overfitting was introduced in Section 2.3.3 and Equation2.30. Regularization is anything which promotes better learning of f(X)without also modelling noise.There are many regularization strategies; selecting, evaluating, and bal-ancing them is one of the primary tasks of the machine learning researcher.One form of regularization has already been introduced in Section 2.3.4.1:sampling data in minibatches causes the noise to change across iterationsreducing overfitting. There is a danger of applying too much regularization;in such a case, learning is slowed so much that the model, for all practicalpurposes, never converges and the resulting model is underfit.2.3.6.1 L1-L2 Kernel RegularizationL1 and L2 kernel regularization penalize kernel weights as they move farfrom the origin. They are forms of parameter regularization, which givenan function in terms of W, the parameters of the function, the objectivefunction for optimization is changed to332.3. Training using Supervised Machine LearningJ˜(W;L) = J(W;L) + αΩ(W), (2.33)where W is the set of weights of the neural networkand a is a weighting term.In L1 regularization, Ω(W) = ‖W‖1. Adding this regularization termchanges the weight update step on line 7 of 1 to W → α · sign(W) + V,where sign(W) is the sign of W applied element-wise. The end result is apreference for weights with a value of 0, which promotes sparsity within thenetwork [22].In L2 regularization, Ω(W) =12‖W‖22. Adding this regularization termchanges the weight update step on line 7 of 1 to W → (1 − α)W + V,reducing the weights in the network by a constant factor at each step. Theend result is a preference for weights near zero [22].2.3.6.2 Data AugmentationData augmentation is the practice of modifying X to produce a morerobust model. It can be seen as a form regularization, where the changes toX can be seen a modifications to noise. In fact, it is of extreme importancethat Y remains unchanged, or that these changes are properly reflected. Ina classification task, for example, data augmentation should not change theunderlying class of the image [22].In a segmentation task, a guarantee that no classes have been changedthrough data augmentation is more difficult to achieve; the classification isapplied pixel-wise, and many pixels may be modified. Many libraries havebeen developed to apply these augmentations to both the image and maskin segmentation tasks, but the researcher much remain vigilant that changesmade through augmentation properly generalize. The best known methodto ensure generalizability is validation on a dataset which has not undergoneaugmentation.Augmentations for segmentation tasks commonly include flipping, bothvertically and horizontally, as well as gamma correction and elastic defor-mations [32, 55]. The importance of data augmentation is a function of thesize of the dataset; if the dataset is sufficiently large and diverse to representall significant examples, data augmentation will be of little benefit.342.3. Training using Supervised Machine Learning2.3.6.3 Batch NormalizationBatch Normalization layers maintain parameters needed to normalizeeach minibatch to a similar normal distribution. It was motivated to resolveInternal Covariate Shift (ICS) through guaranteeing distribution parame-ters [28]. However, there is recent research which shows that, while batchnormalization works, it does not work by correcting ICS. Santurkar et al.argue that batch normalization appears to not reduce ICS at all; instead, thebenefits come from smoothing the optimization landscape, allowing higherlearning rates and faster convergence [58]. Another possible explanation isthat batch normalization seems to force sparsity in the network, with ap-proximately a quarter and half of all feature channels being pruned throughbatch normalization [16]. Regardless of the explanation, there is widespreadacceptance that batch normalization is doing something to improve neuralnetwork performance, and the original paper has been cited almost 20,000times.2.3.7 Cross ValidationValidation requires that some available data be held back from training.This can have a negative impact on the training process, particularly iftraining is data limited. In a small data set, it is possible that not allfeatures are represented in many images; as such, when removing data fromthe training set to use in the validation set, features can be removed entirelyand never seen in training.Cross-Validation is an evaluation technique where multiple trained mod-els are produced; each model holds back a different portion of the overalldataset for validation. Cross-Validation is a common technique in trainingneural networks, particularly when the data set is small. A Fold refers acombination of training and validation data used in cross-validation. Eachpiece of data is used only once in the validation set; in the remaining folds,the data is used in the training set. Each fold produces a different model;by combining the models after training, each piece of data can be properlyrepresented in the final results. Note that each fold is a full training run;when many folds are used, this training can be a time consuming endeavour.However, if not enough folds are employed, then significant portions of thedata set are held back in each fold which can reduce the quality of eachtrained fold.352.3. Training using Supervised Machine Learning2.3.8 Scalability and EfficiencyRecent work has focused on automating the search for the optimal net-work size. Tan et al. employed reinforcement learning to size networks formobile devices; they allowed different layer architectures in different blocksand employed a compound search which optimized both accuracy and la-tency. The model which they produced exceeded accuracy baselines whilemaintaining comparable size and latency [63]. Tan and Le then expandedupon their previous work by developing EfficientNetB0 using a grid search ofheight, width, and depth parameters. They showed that this model is ableto meet or surpass accuracy baselines with 8.4x parameter and 16x FLOPSreduction [64].Tan et al. broke down the network into pieces, showing that the overallfunction is a composition of smaller functions of each layer:Yˆ = f(X) =⊙i=1,2,...,LFi(X<hi,wi,ci>), (2.34)where hi is the height of the network at layer i,and wi is the width of the network at layer i,and ci is the number of feature channels in the network at layer i,and L is the number of layers in the network.The size of a network can be evaluated by three significant parameters:P , the number of trainable parameters in the network; M , the amount ofmemory the network consumes; and FLOPS, the number of floating pointoperations required to compute the network. These parameters roughlycorrelate to particular properties of the network: P relates to the totalcount of convolutional kernel values within the network; M relates to thetotal number of neurons in the network; and FLOPS relates to the numberof connections within the network.2.3.8.1 Image SizeWhen using CNNs, the dimensions h and w at layer l are not indepen-dent of those at layer l− 1. Instead, they follow a proportional relationship.The constant of proportionality between layers is controlled by the operationbetween layers. When using convolution, the change in dimension is con-trolled primarily through Stride (see 2.1.3.2). Another common operationof dimensionality reduction is pooling. There are many forms of pooling,362.3. Training using Supervised Machine Learningbut the most common is Max Pooling. Max Pooling is a dimensionality re-duction technique where a section in an image is replaced by the maximumpixel intensity in the section. It preserves dominant features while reducingdimensions and allowing for small spatial translations in the original image.Either technique will end up with a dimensionality reduction propor-tional to a constant R, which corresponds to stride of the convolution orto the pooling size. Since dimensionality reduction throughout the networkmakes the size of each layer proportional to the size of the previous layer,the original image size is of great import. When 0 < R < 1, the networkperforms downsampling, and the total size of the network can be solved asa geometric series. When R > 1, the network performs upsampling andconvergence on the sum is only guaranteed for finite L.The total memory consumption of the network can be approximated byM ≈∑l={0,1,2,...,L}hl · wl= h1 · w1 + h2 · w2 + ..., hL · wL= h1 · w1 +R · h1 ·R · w1 + ..., RL · h1 ·RL · w1= h1 · w1(1 +R2 + ...R2L), (2.35)where R is the size proportionality constant between layersand hl is the height of the network at layer land wl is the width of the network at layer lNote that this equation holds only when R is constant between layers. Inmany cases, R is not constant across the entire network; this is particularlytrue when a single network does both downsampling and subsequent up-sampling. In such cases, total M could be calculated by summing Equation2.35 across all sections in which R is constant.Reducing the size of the initial image has an impact on the size of everysubsequent layer. This reduces both M , the amount of memory required tohold the model, and the number of FLOPS required to compute the model.It has no impact on P , the number of trainable parameters. However, thereis a trade off in that fine features in the image may be distorted or removedentirely in the downsampling process used to reduce the image size. Thus,reducing image size size should be done with care.Rather than downsampling the entire image, training can be done onpatches. A Patch is a subsection of a image, usually of fixed size, which can372.3. Training using Supervised Machine Learningbe used for training. Smaller patches reduce the size of the overall network.Multiple patches can be created from the same image. Patching is oftenrequired with very large source images, especially for 3D datasets where inaddition to height and width, the third dimension must also be multipliedin Equation 2.35 making the overall size O(n3) rather than O(n2). Thedownside of patching is the introduction of additional edges and truncationof connected features that could affect overall results.2.3.8.2 WidthIncreasing the number of channels allows the network to maintain manydifferent features which may be of use in calculating the output. Recall thatfor a CNN, sparse connections and parameter sharing reduce the number oftrainable parameters per convolution operation to K, where K is the size ofthe convolutional kernel. Recall also from Section 2.1.3.3 that the numberof convolution operations between layers l and l+1 is the product of thenumber of channels in layers l and l+1.The total number of channels C is directly proportional to the number oftrainable parameters P , where the constant of proportionality is the kernelsize:P ≈ k · C, (2.36)where k is the size of the convolutional kernel.Usually, the kernel size is fixed for the whole network and is often 3x3.From Section 2.1.4, the receptive field of convolution can become large evenwith a small, fixed kernel size through repeated convolution.Note that equation 2.35 must be modified when working with multi-channel convolution. The total memory required is now a function of height,width, and the number of channels in an image at that layer:M ≈∑l={0,1,2,...,L}hl · wl · cl. (2.37)Since the number of channels does not necessarily change by a propor-tional constant, the equation can no longer be solved as a geometric series.382.3. Training using Supervised Machine Learning2.3.8.3 DepthDepth has an important impact on the performance of neural networks.Recall from Section 2.2.1.1 that non-linearity in the network is added bynon-linear activation functions. The activation function is applied as theoutput to each layer, so additional layers allow for additional applicationsof non-linear functions. However, depth also expands the network size veryquickly, as shown in Equations 2.36 and 2.37. A larger network means alarger memory requirement and more floating point operations. Additionallayers also mean additional convolution steps which increases the number oftrainable parameters.2.3.8.4 EnsemblingEnsembling relies on multiple, uncorrelated predictions to produce amore reliable final prediction. Given multiple models which produce nor-mally distributed error, the average error of all predictions is given by:E[(1k∑ii)2] =1k2[∑i(2i +∑j 6=iij)] =1kv +k − 1kc, (2.38)where v is the variance of the error E[2i ]and c is is the covariance of errors E[ij ].When the models are perfectly uncorrelated, c = 0 and the error reducesto 1k . When the models are perfectly correlated, c = v and the error reducesto v, which is no improvement over a single predictor. Thus, averaging canhelp to reduce the error of multiple models, so long as the models producedifferent predictions [22].2.3.8.5 Gradient BoostingGradient Boosting is a method for combining weak learners such thatthe overall loss is minimized. This usually means that subsequently trainedlearners are encouraged to correct mistakes made by their predecessors. Gra-dient boosting begins with Equation 2.1, which is extended to include anadditional learner which is combined with the firstYˆm = f(X) + Yˆm−1. (2.39)392.3. Training using Supervised Machine LearningInserting the modified prediction into loss function from Equation 2.25yieldsL = L(Y, Yˆm−1)= L(Y, f(X) + Yˆm−1). (2.40)Loss is minimized when Yˆm = Y. Inserting this equality and solving forthe current prediction in Equation 2.39 yieldsf(X) = Y − Yˆm−1. (2.41)Thus, the current learner is learning the residual loss of the previouslearner. This can be repeated for many iterations, with each new learnercorrecting the mistakes of past learners [17, 18].Boosting is commonly applied to binary classification problems throughdecision trees. Decision trees produce a single scalar output, where the loss iseasily calculable. More recent work by Bardirli et al. applied boosting tech-niques to neural networks for the tasks of binary classification, regression,and ranking. With neural networks, they achieved results which surpassedthe baseline in all cases [2].40Chapter 3CompetitionsCompetitions are a popular way for the computer vision community totackle complicated problems with machine learning. Usually, they evaluatesubmissions using statistical methods and there is little statistical differencebetween the top submissions. For the researcher, they offer value throughthe availability of data sets and evaluated techniques for solving popularproblems. Competitions are also not peer-reviewed, with the advantagethat they usually have a faster turn-around time on results; however, thisalso means that winning techniques are not fully vetted by the community,and so should be approached by the researcher with cautious optimism.Many competitions have developed around medical applications andComputer Aided Diagnosis (CAD), a term coined in 1966. CAD relatedto cancer detection consists of two main stages: candidate detection andfalse-positive reduction [19].CT Scans Candidate Detection Feature Extraction Classification CAD MarksFalse-Positive ReductionFigure 3.1: Computer Aided Diagnosis. Adapted from [19]In the candidate detection stage, the priority is to ensure that all truepositives are included. In chest imaging, for example, many objects areeasily distinguished and removed; however, some unwanted objects, partic-ularly airways, are difficult to distinguish from cancerous nodules. This isespecially true in two-dimensions, where the cross section of a cylinder anda sphere look the same.False-positive reduction is a classification task, with each candidate be-ing assigned a probability of being cancerous. Including low probabilitycandidates includes more true positives, whereas including only high prob-ability candidates reduces the number of false positives. The output of thisprocess is a collection of CAD marks, which mark the centroid, size, and413.1. U-Netcancerous probability for nodules.The candidate detection, feature extraction, and classification tasks canbe performed using a variety of techniques, notably human observation,traditional computer vision, or neural networks. They can also be completedin 2D, or in 3D.These competitions have fueled recent research; the same general ap-proach used to win the Kaggle Data Science Bowl 2017 (DSB17) was fea-tured recently in a paper on lung cancer screening; they achieved an AUCof 95.9% which exceeded performance of radiologists [1].3.1 U-NetU-Net was developed as a solution to the 2015 ISBI cell tracking challengewith promising results: an IOU score of 92% on the PhC-U373 dataset withsecond place scoring 83%; and an IOU score of 77.5% on the DIC-HeLadataset with second place scoring only 46% [55].In 2016, 3D U-Net was developed, which replaced all operations with3D equivalents and added batch normalization [10]. It does not appear tobe widely used initially; 3D U-Net would likely have succeeded in the 2017Kaggle Data Science Bowl, but appears untried [34, 41].In 2018, Isensee et. al took a novel approach to medical image segmen-tation with nnU-Net [31]. Where much of the research had been focusedon architectural improvements of the underlying neural network, Isensee et.al hypothesized that the theory behind U-Net was fundamentally sound,and was advanced enough to perform well in a variety of tasks. Their workfocused instead on the proper training and implementation of U-Net.We argue the strong case for taking away superfluous bells andwhistles of many proposed network designs and instead focuson the remaining aspects that make out the performance andgeneralizability of a method.nnU-Net, Isensee et. al (2018) [31]We hypothesize that some of the architectural modifications pre-sented recently are in part overfitted to specific problems or couldsuffer from imperfect validation that results from sub-optimalreimplementations of the state-of-the-art.nnU-Net, Isensee et. al (2018) [31]423.1. U-NetUsing the 2D and 3D versions of U-Net as the underlying theory, nnU-Net has achieved significant practical success in medical imaging semanticsegmentation tasks. Also of import, nnU-Net is easy for researchers to adoptas there is a fully functioning implementation [29]; this implementation self-adapts to various data sets with differing properties.In a short time, nnU-Net has achieved great success. The Medical ImageSegmentation Decathalon was devised to address the possibility of overfit-ting and specialization that previous tasks had enabled. Rather than presenta single task, 10 unique tasks in Medical Image Segmentation were devised.nnU-Net won the challenge overall, showing that the method and archi-tecture were capable of generalizing well to a variety of medical imagingsegmentation tasks. The author of nnU-Net also won the KiTS19 Challenge[25].3.1.1 ArchitectureU-Net is named for the characteristic U shape of the neural networkarchitecture. It has proven to be incredibly useful in image sematic segmen-tation. U-Net consists of two main paths: contraction, or dimensionalityreduction to a feature map; and expansion, or feature extraction from thefeature map to a final segmentation map. The architecture of U-Net isconfigurable in many aspects, including depth, width, and activation func-tion; for the sake of simplicity, the winning solution created by the originalauthors is described here.The input image is passed in the left, and convolution is used to createmultiple feature channels. Max pooling is used as the main form of dimen-sionality reduction. Here, a 2x2 kernel is passed over the image with a strideof 2 (stride refers to skipping pixels while sliding) so the resulting image ishalf the size. The maximum value under the kernel is used as the outputvalue. This allows for small translational shifts in the image to have noeffect on the output, and has been shown to work very well in practice.The expansive path produces the desired segmentations from the featuremap. If one were to simply resize the resulting image, then the expansivepath would be determinstic and force the feature map to produce a scaleddown version of the final segmentation; instead, a learnable process is pre-ferred, which allows the resulting segmentation to be dependent on both thecontractive and expansive paths.The main operation used in U-Net for learnable dimensionality expan-sion is what the authors call up-convolution; in actuality, this is transposedconvolution with a stride of 2. This causes the number of pixels per channel433.2. Lung Nodule Competitionsto double in each up convolution step. Convolution is again employed toproduce a number of feature channels. There is one more important op-eration in the expansive path, which is cropping and copying the featuremap from the same layer in the contractive path to the expansive path.This allows features from each layer to contribute to the final segmentation.Cropping is required because the dimensions on the layers of the contractiveand expansive path do not necessarily match.The 3D version of U-Net is much the same as the 2D version; even theauthors of the paper acknowledge that it is the 2D version with all operationsreplaced by their 3D counterpart, and batch normalization added [10].Figure 3.2: U-Net architecture. From [79]3.2 Lung Nodule CompetitionsThree popular competitions on lung cancer nodule detection are Au-tomatic Nodule Detection 2009 (ANODE09), Lung Nodule Analysis 2016(LUNA16), and the DSB17. The available data set size increased dramat-ically between 2009 and 2016. This was likely the largest contributor tobetter results. Another important factor was the development of U-Net; no-tably, the 2nd and 3rd place teams in 2016 competing in LUNA16 employedU-Net or a modified U-Net in their solution.The DSB17 posed a different question than LUNA16. Rather than iden-tifying and classifying all nodes, the task was to produce a single probability443.2. Lung Nodule Competitionsthat the patient is diagnosed with cancer in the next year. Log Loss wasemployed for evaluation, which is a common method of evaluating singleprobability predictions.Table 3.1: Lung Nodule Detection CompetitionsANODE09 [71] LUNA16 [60] DSB17 [34]Year 2009 2016 2017Goal Diagnostic1 Predictive2# of Scans 55 1018 > 10003Evaluation AUC Log LossBest Score 0.632 0.9514 0.399753.2.1 Z-NetZ-Net was the 2nd place entry for LUNA16. It is a machine learningsolution which achieved an AUC score of 0.811. Z-Net relied on U-Net forsegmentation. After segmentation with U-Net, thresholding and erosionwere used to further segment the candidate nodules. Nodules across imageswere combined to produce a 3D shape; candidates were deemed to be thesame in neighbouring Computed Tomography (CT) scans and their centroidswere within 50% of a radius of each other to each other. Three cross-sectionswere taken from each 3D shape (x-y, x-z, and y-z), and these cross sectionswere fed into a classification neural network [5].Z-Net is notable as it shows that a pseudo-3D solution can be reasonablysuccessful at lung nodule detection.3.2.2 Multi-Opening and Threshold (MOT M5Lv1)Multi-Opening and Threshold (MOT M5Lv1) was the 3rd place entryfor LUNA16 with an AUC score of 0.742. Nodule detection was performedin 3D using a region-growing method. The overall region-growing method isnot well defined, but the authors report that it is based on [49]; it also uses1Identify locations of possible nodules, and assign a probability for being a nodule to eachlocation.2Will the patient be diagnosed with lung cancer within one year of the date the scan wastaken?3This data set is no longer public, so exact size cannot be determined.4Achieved in 2018. The highest score in 2016 was 0.856.5The 1st place LUNA16 team in 2016 did not publish a detailed description of their approach.453.2. Lung Nodule Competitionsa morphological technique for separating nodules from blood vessels [36].The following features are extracted from each candidate nodule:Geometrical Features Intensity-related featuresRadius (mm) AverageSphericity Average outside maskSkewness of distance from center Standard deviationKurtosis of distance from center Standard deviation outsideFraction of voxel connected to lung cage EntropyDistance from lung top Entropy outside maskDistance from HiluxClassification is performed by a feed-forward neural network with 15 in-put neurons (one for each feature), 1 hidden layer of 31 neurons, and a 1neuron output layer [66, 67]. Generally, the process of extracting featuressuch as those listed and passing them into a classifier is in the field of Ra-diomics. Radiomics seeks to extract large amounts of quantitative featuresfrom medical images using data-characterisation algorithms. It is an alter-native to deep learning, breaking the overall task into steps with, potentially,a machine learning classifier at the end.This solutions shows that a radiomics-based solution can be successful.This is significant as radiomics is potentially easier to interpret and explain,which is a key requirement missing from these competitions.Image Acquisition & ReconstructionFalse-Positive ReductionImage SegmentationImage QuantizationPreprocessingHistogram Generation Texture Matrix DesignTexture Feature ExtractionQuantificationStatistical Analysis and/or Machine learningAnalysisFigure 3.3: Radiomics Workflow. Adapted from [19]463.2. Lung Nodule Competitions3.2.3 grt123The grt123 team won the DSB17 with a Log Loss score of 0.39975. Thismethod was a significant departure from the 2016 solutions shown above.Notably, a 3D Deep Convolutional Neural Network was used for candidatedetection. Candidate detection in 3D has the advantage of being able todistinguish nodule from blood vessel and airways more easily. Nodules arespherical, whereas blood vessels and airways are cylindrical; in 2 dimensionsthe cross sections look similar and are hard to distinguish.The solution first creates a 3-dimensional model by combining CT scanslices. This procedure is done without machine learning.1. Mask Extraction(1) Gaussian Filter (σ = 1px)(2) Binarize (-600 Hounsfield Units (HU) threshold)(3) Connected Component Filtering2. Re-Introduce Edge Nodules(1) Separate lung into left and right(2) Convex hull of each is combined(3) This step is ignored if the size of the convex hull is too large3. Intensity Normalization(1) Convert HU → UINT8(2) Clip intensity to [-1200 600] (air is ≈-1000, lung is ≈-700, boneis ≈400)(3) Linear Transform intensity to [0, 255](4) Masked with the result of (2)(5) Set area outside mask 170(6) Set intensities > 210 to 170(7) Set intensity of bones to 170The grt123 team has made their source code available online [40]. Theystate that it takes 3-4 days to train with 8 Titan Xp GPUs. Figure 3.4shows rendered pre-processing results after converting from numpy voxelsinto mesh objects using PyMesh [80] and rendering with the Blender Python473.2. Lung Nodule CompetitionsAPI [65]. Pre-processed results are chunked into 128x128x128 cubes and fedinto the candidate detection neural network, which the authors called N-Net.The output of this neural network is a sequence of 32x32x32x3 features (thedimensions of the yellow cube) where each feature is a 5-dimensional vector(oˆ,dˆx,dˆy,dˆz,dˆr). oˆ represents confidence, and the remaining values representthe centroid and radius of the nodule.Figure 3.4: Example grt123 Pre-Processing Results.Once proposed nodules are identified, a method of combining them tocalculate a single probability of the patient being diagnosed with cancer isrequired. First, the top 5 candidates are extracted, then one of four methodsis used for combination:− Feature Combining - Concatenate all 5 feature sets and feed into aNeural Net. The downside is that this process is difficult to interpretand explain. It is unclear why the grt123 team felt this was the pointwhere the process became difficult to explain, but it is reassuring thatinterpretability was a concern.− MaxP - Each feature set fed into a separate neural net to get a prob-ability, then take the max. However, since the probabilities are notindependent (the probability of a second nodule being cancerous giventhe patient already has a cancerous nodule is much higher) this methodis inaccurate.− Noisy-or - Independent probabilities are combined using P = 1 −∏i(1− Pi).− Leaky Noisy-or - Independent probabilities are combined using P =1− (1− Pd)∏i(1− Pi), where Pd is the probability of dummy noduleand is learned.483.3. Other Competitions3.3 Other CompetitionsThere are numerous other competitions, both within and outside of med-ical imaging. Many of the latter are part of the Grand Challenges in MedicalImaging Analysis [73]; while these challenges span a variety of tasks, manyof them have segmentation either as the main goal or as a critical step in thepipeline. Recently, KiTS19 was issued as part of MICCAI 2019 [15], withthe winner and most of the leading submissions employing a 3D U-Net [26].There are also many competitions outside of the domain of medical imag-ing analysis, many of which are hosted on Kaggle [33]. A recent exampleof a non-medical image segmentation task is the Severstal Steel Defect De-tection challenge [27]. 2,431 teams competed, with 126 entries achievinga score between 0.90002 and 0.90883, which shows how competitive thesecompetitions have become.3.4 Trustworthiness of Competition ResultsThe LUNA16 challenge was inundated with submissions that scored wellwithout actually solving the problem presented. The ground truth to alldata, including the test set, was released publicly. It is, therefore, easyto overfit a solution which achieves great results. More maliciously, it ispossible to submit the correct results exactly, without even attempting thetask.It became obvious to the organizers that some teams did not, or didnot properly, use cross-validation and intentionally overfit their model. TheLUNA16 team acknowledged that some solutions were “too good to be true”and tried to identify edge cases which would have been difficult for a gener-alized solution to identify; they then tried to filter submissions which out-performed on these edge cases.The competition was ultimately limited to submissions with supportingdocumentation, with a note: “It would be a good idea to repeat the LUNA16challenge with a new test data set. Preferably we do not release this testdata publicly” [72].DSB17 released only a portion of their expected results publicly; theothers were held back to ensure generalizability. However, they suffered fromanother exploit, which allowed teams to divine expected results throughquerying the log loss results [75]. This method does not appear to havebeen used in competition, but it is impossible to know for sure. In the end,the top-placing solutions produced good descriptions of their solutions, and493.4. Trustworthiness of Competition Resultsthe prize money likely helped provide motivation to root out questionableapproaches.The grt123 team used an optimization which could potentially limit itsgeneralizability. It is called out here, but it is left to each reader to passjudgment. The grt123 realized that the LUNA16 dataset had many smallnodules, and that the DSB17 dataset had many larger nodules. Before train-ing, they removed nodules smaller than 6mm from the LUNA16 annotationsand manually labelled some nodules in the DSB17 dataset. To their credit,the team identified this deficiency publicly and called out their approach intheir solution; they also justified it by pointing out that very small noduleswere less likely to be cancerous [41].Many, if not most, winning contest submissions employ heavy hyperpa-rameter tuning and model ensembling. There is a high computational costassociated with hyperparameter tuning, so most submissions appear to beginwith the current state-of-the-art methods and focus much of their attentionon this step. This presents an element of bias, as the best approaches havethe most time spent on optimizing parameters, and so are likely to performbest. KiTS19, for exmaple, saw leading solutions overwhelmingly employ3D U-Net [15].50Chapter 4MethodologyThe experiment explores the hypothesis that many weak learners canmeet or exceed the performance of a single large learner. It is very temptingin deep learning to try and produce the best overall result. However, theweak learner hypothesis can be efficiently explored if the problem is reducedto its essence: a comparison of weak learners to a a single, strong learner. Insuch a case, many hyperparameters can be fixed to good-enough values usedin state of the art solutions; a full hyperparamter search was not performedin this work. Similarly, data augmentation was kept to a minimum withonly flipping employed.Four models were devised to explore the hypothesis:1. A baseline learner inspired by state-of-the-art solutions.2. A weak learner trained in multiple folds.3. A boosted learner consisting of many weak learners.4. A strong learner with the same width as the boosted learners.4.1 DatasetsThe Severstal and KiTS19 datasets were selected as the same deep learn-ing approach could be used for both while their underlying properties woulddiffer enough to evaluate generalizability. The Severstal dataset has a largeclass imbalance; notably, Class 2 is represented in only 247 images whileclass 3 is in 5,150. Both the kidney and tumor classes are well representedin the dataset. The Severstal dataset appears to be crudely annotated, withlarge areas around defects also being included in the mask; annotations inthe KiTS19 dataset, by contrast, appear considered and accurate [25].The datasets were split randomly for each class such that 90% of ex-amples were in the training set and 10% were in the evaluation set. Whencreating 2D slices from 3D cases in the KiTS19 dataset, entire cases werekept in either the training or evaluation set; slices from the same 3D case514.1. Datasetswere never present in both. Similarly, when creating folds, all slices fromthe same case were guaranteed to be in either the training or validation set.For the KiTS19 dataset, images were resized to a constant value of (512,512). For both datasets, contrast stretching with percentiles of 2% and 98%was employed to maximize the intensity range.(a) Severstal (b) KiTS19Figure 4.1: Class Instance Counts.The Severstal dataset has a large class imbalance, shown in figure 4.1(a).In a multi-class segmentation model, this would need to be considered; themodel would prefer to learn the dominant class at the expense of less frequentclasses. Class 2 shows only a very small number of instances. Training thisclass will demonstrate how the training process handles a limited exampleset.Components in classes 1 and 2 generally occupy only a small portion ofthe image, whereas in classes 3 and 4 they occupy a larger section of theimage, as shown in table 4.1.Table 4.1: Component size statistics for the Severstal classes.Mean (%) Median (%) Standard DeviationClass 1 0.31 0.21 0.31Class 2 0.63 0.56 0.39Class 3 2.19 1.03 4.67Class 4 3.52 1.88 4.63The kits dataset shows a moderate class imbalance, but both classes arewell represented. In the kidney class coverage histogram, there is a large524.2. Model Designspike near 0%; this is likely the result of how the 2D images were createdby slices from 3D examples; even a single pixel example for a slice would beincluded in that class. The tumor class shows an even larger spike at 0%.(a) Kidney Component Count (b) Kidney Component Size(c) Tumor Component Count (d) Tumor Component SizeFigure 4.2: Components in the KiTS19 Dataset.By observation, the severstal dataset appears to be crudely annotatedwith large areas around defects also being included in the mask. By contrast,the KiTS19 dataset appears to be very accurate, which is expected fromradiologist-annotated examples.4.2 Model DesignEfficientNetB0 was selected for the baseline as it is a state-of-the-artsize-optimized model [64]. Since the baseline is made up of a single model,it does not employ cross-validation; instead, the evaluation set is used fortraining validation. This leaves no unseen evaluation set to be used on the534.2. Model Designfully trained model. The baseline model, therefore, has an advantage at theevaluation stage, as the weights selected in training minimize the loss onthe evaluation set. Other models used cross validation of the training set;10 folds were used to increase performance of poorly represented classes, asdata augmentation was limited to horizontal and vertical flipping.A modified U-Net, configurable in the scaling dimensions and activationfunction, was employed for the experimental models. Similar to [10], batchnormalization was used. Through a grid search, it was determined that amodel could perform well with 6 layers and a constant 4 channels per layer;this is a marked change from the original U-Net design, which called fordoubling the number of channels at each layer [55]. Predictions from allfolds were averaged before applying the sigmoid function.Strong learners were created in the same fashion as the weak learners,but with 20 channels per layer. The total increase in trainable parametersis given by 20242= 25. The additional trainable parameters are entirely madeup of convolutional kernels, requiring roughly 25 times more floating pointoperations (FLOPS). The boosted learners were crafted from the weak learn-ers, with gradient boosting applied 4 times. The final boosted learner modelis comparable to the strong learner model in number of neurons; however,trainable parameters increased by approximately 5·4242= 5. While additionalbatch normalization must be performed, convolution is the dominant oper-ation and the boosted network requires approximately 20% of the FLOPSrequired for the strong learner network.Based on the evaluation done by [53], the activation functioncos(x) − x was deemed a good candidate, but was negated to produce anactivation function increasing in x. Binary focal loss (BFL) was used, whichdownplays the significance of well-classified examples when compared withbinary cross-entropy. BFL promotes correction of misclassified examplesin sparse datasets and discourages greatly misclassified predictions [42]; thisfacilitates aggregation, as many predictions near the margin can be correctedby a few well-classified predictions.4.2.1 AggregationThe model used for training and evaluation is a construction of one ormore smaller sub-models. These smaller models are consistent in architec-ture, including size and connections; however, they have different weightsbetween connections. They are arranged in such a way that one or moresub-models make up a single fold through gradient boosting. One or morecomplete folds make up a final model. For a final model with n folds and m544.2. Model DesignTable 4.2: A comparison of the configurations evaluated.Baseline Weak Learner Boosted Learners Strong LearnersTrainable Parameters ≈10M ≈5K × 10 folds ≈ 50K ≈ 250K ≈ 1.25MSub-Model efficientnetb0 1 Modified U-NetFolds 1 10Boost Folds 0 0 4 0Depth - 3 6Width (constant) - 3 4 20Activation Function - 3 x− cos(x)Optimizer AdamBatch Size 4 10Loss Dice (β=1) + Binary Cross-Entropy 1,2 Binary Focal (γ = 2) 1Learning Rate 0.001, reduced by factor of 0.1 after 20 epochs, minimum 10−8Early Stopping 60 epochsNormalization BatchInitialization He Normal 2L1 / L2 Regularization 5× 10−3 2 0Data Augmentation 25% Horizontal Flip 25% Vertical Flip 25% Both Flipped 25% NoneData Pre-Processing Contrast Stretching Contrast Stretching and Recenter Mean at 01Implemented in segmentation models [78]2As recommended by nnU-Net [32]3Defined by efficientnetb0 [64]boosts per fold, a total of n×m sub-models are used.Theoretically, any sub-model architecture can be employed in such asetup, subject to memory and other system limitations. In practice, manycommon models employ a sigmoid model as their final layer. Since thesigmoid function has a non-zero mean and a clipped range, fold boost-ing through summation can only increase the predictions after the sigmoidfunction is applied; this would lead to a biased system. The summation,therefore, must be performed without the logistic function applied; anotherpossible implementation is to invert the logistic function before summationby applying the logits functiony = log(x1− x). (4.1)The aggregator responsible for combining predictions from folds into afinal prediction. It consists of the activation function after each fold f1,the aggregation function A, and the activation function after aggregationf2. Since application of the sigmoid function twice always results in positivepredictions, only one of f1 and f2 can be a sigmoid; the other must be linear.554.2. Model DesignA f2 Yˆ∑Fold 0f0 b0f0 b1f0 bm...f1...∑Fold 1f1 b0f1 b1f1 bm...f1∑Fold nfn b0fn b1fn bm...f1Figure 4.3: Model Construction. One or more model is produced for eachfold and summed. The output of the summed functions is then passedthrough activation function f1. The output of the folds are aggregatedusing function A. The final aggregated results are passed through the finalactivation function f2 to produce YˆA f2 YˆYˆ0 f1...Yˆ1 f1Yˆn f1AggregatorFigure 4.4: Aggregation. Predictions from multiple folds are combined toproduce a final prediction.564.3. TrainingTable 4.3: A comparison of the four aggregation methods evaluated.f1 A f2Average After Sigmoid sigmoid 1nn∑i=0yˆi linearAverage Before Sigmoid linear 1nn∑i=0yˆi sigmoidNoisy Or sigmoid 1−n∏i=0(1− yˆi) linearVote sigmoid 1nn∑i=0round(yˆi) linear4.3 TrainingFolds without gradient boosting share no dependencies when training.As such, multiple folds can be trained on independent hardware and thencombined into the final model. While this does not reduce total computetime required for training, it can reduce calendar time when sufficient hard-ware is available.Folds trained through gradient boosting, however, share a serial depen-dency which requires the first model to be trained for a fold before thesecond, and so on. Since each boost fold must be trained completely beforethe next, each boost fold increases both compute time and calendar time ofthe training process.Gradient boosting also increases the memory requirement of a model lin-early, as previous folds must be loaded to determine their loss when training.When training, the total approximate memory requirement is the productof the batch size, the number of gradient boosted folds, and the size of eachsubmodel. Some overhead must be kept for the optimizer and aggregationfunctions; however, this overhead is usually much smaller than the aboveproduct.∑f0 b0f0 b1f0 bm...f1 YˆFigure 4.5: Boosted Model Training. Previous folds contribute to the loss.The gradient is backpropagated to the final fold only.574.4. FacilitiesWhen training gradient boosted folds, the previously trained folds serveonly to modify the output values and resulting loss for backpropagation.Thus, all previous boost folds have their weights fixed and the gradient ispropagated only into the current fold being trained. Similarly, the weightssaved are for the current fold being trained only.4.4 Facilities4.4.1 Software System OverviewA system for processing, training, evaluating, and visualizing 2D seman-tic segmentation datasets was developed.Data ProcessingSource DataProcess DataData SetTrainingTraining DataTrainingWeightsEvaluationEvaluation Data Evaluator Evaluation ResultsOptimizerJob ConfigurationCallbacksLossMetricsModelVisualizationVisualizer Visualization ResultsFigure 4.6: System Overview.Execution of tasks is controlled by a launcher created as a Python pack-age. The package installs a script called launch into the system path;this script searches recursively through the current directory for files namedtasks.py which contains both the arguments and execution required to com-plete the task. Tasks include processing datasets, training, evaluating, andvisualizing metrics.584.4. FacilitiesOn top of being a convenient method for defining and launching taskswith varied parameters, the tool also allows launching tasks inside Singular-ity or Docker containers. When launching inside a container, the requiredsource code, data, and output directories are first mounted in the container.This provides a means for isolating dependencies, which is particularly usefulwhen using data centers such as Compute Canada.4.4.2 Computational ResourcesInitial development and experimentation was performed on a desktopPC with a single GTX 1070 graphics card. This served well in quicklyassessing the viability of particular approaches; however, training all foldsand all classes for a full dataset became prohibitively long. Compute Canadaresources were employed and the training of folds performed in parallel forfull evaluation.Since Compute Canada is a shared resource that prefers short-runningjobs, tasks were executed in chunks with an aggressively small time limit; thetasks re-scheduled themselves if not completed. This allowed the training totake up small chunks of time on the shared resources as they came available,and release them to others periodically.Once fully trained, the weights were transferred back to the desktop PCand evaluation was done on the GTX 1070 graphics card. Evaluation washeavily dependent on the size of the dataset, but took several days to runto completion.4.4.3 Evaluation and VisualizationEvaluators load a model and produce numerical metrics which can beused to measure performance. An evaluator is either threshold aware, or it isfold aware. A threshold aware evaluator accepts a full model and evaluatesmetrics at varying thresholds. A fold aware evaluator processes each foldindividually without building the complete model.A visualizer is responsible for displaying evaluation results in a human-readable format, usually a graph. There does not need to be a one-to-onerelation between evaluators and visualizers, but often there is. Visualiza-tion was separated from evaluation because evaluation requires computingthe model, which requires a GPU and is usually slow. Iterating on visual-izations is much faster when the overhead of evaluation is not included; andwithout a gpu requirement, the visualizations can be produced on virtuallyany machine.594.4. Facilities4.4.4 Data ProcessingA processed dataset consists of: a compressed numpy file for each case,with entries for a 2D image and an array of K instances of segmentationmaps of the same shape as the image, where K is the number of trainingclasses; a mapping of cases to class, split into training and evaluation sets,with a list of length K describing the order of classes in the mask array; anda separation of training and validation cases for various numbers of folds.Datasets are loaded using Tensorflow Datasets. A Python class wascreated which creates a generator of filenames for the required fold, class,and train or evaluation data. Parameters exist to shuffle the dataset (fortraining), and the loaded files are cached in memory to lower the probabilityof the process being bottlenecked by disk I/O operations.Data augmentations are specified in the job configuration and performedonline by the albumentations library [7].4.4.5 Job ConfigurationConfigurations are produced which define the full parameters for an al-gorithm. Examples of commonly used parameters are the number of foldsto train and which model to use. These configuration files are persisted todisk and are partitioned by a unique key; this key is generated by a hash ofthe configuration. This partitioning allows for separation of the job artifactsbetween configurations; for example, the weights produced by training arenecessarily different between job configurations and thus need to be keptseparately.Many training parameters are defined in the job configuration, includingthe optimizer, patience, batch size and learning rate. The loss function isa configurable linear combination of dice loss, binary focal loss, and binarycross-entropy. The weights are configurable in the job configuration, as arethe underlying loss parameters beta and gamma.4.4.6 CheckpointingTraining is a long-running process. Some training jobs can take hours oreven days. The concept of checkpointing was incorporated into the systemto allow breaking such jobs into smaller slots of time. This also allowsresumption of the job should it terminate for an undesired reason such asa power failure. A checkpoint is saved after each training epoch, whichconsists of saving: the current weights of the model; the optimizer state;and the state of any stateful callbacks.60Chapter 5Results and DiscussionThe proposed method was evaluated on six classes across the KiTS19 andSeverstal Steel Defect Detection datasets. The primary metric to evaluateoverall performance was the F1 score. For each of the aggregation methods,a sweep of thresholds was performed and the threshold producing the bestmean F1 score used for full analysis. The sweep of thresholds was alsoused to produce the PRC for the model, and the AUC was calculated usingthe trapezoid rule to approximate the integral. For the boosted network,gradient boosting was performed a fixed number of 4 times per fold; thereduction in loss relative to the unboosted model for the fold was evaluated.The average after sigmoid, average before sigmoid, and noisy or aggrega-tions methods were all found to produce very similar F1 score distributionsat their best thresholds, and the vote method was found to underperform.Because of this similarity in performance, and to promote brevity in theanalysis, discussion for all sections except for the F1 score will be presentedfor only average before sigmoid aggregation method. Full results for all ag-gregators, including a view of metrics by threshold, is available in AppendixB.5.1 F1 ScoreThe network of aggregated weak learners is able to achieve a mean F1score within 93% of the EfficientNetB0 baseline. A similar network of ag-gregated strong learners matches the mean F1 score of the baseline on theSeverstal dataset. Gradient boosting applied to the weak learners achievesa mean F1 score within 98% of the strong learner network on the Severstaldataset. The method also shows good results on the Kidney class in theKiTS19 dataset.In the tumor class, the baseline vastly outperforms all smaller networks;however, the box bottom for the baseline model is near 0, which is an indi-cation that the tumor class is difficult to isolate. In all other classes except2, the weak learners underperform; in class 2, the extremely limited datasetlikely limits possible improvement through scaling. It is also interesting to615.1. F1 Scoresee that each of the boosted, strong, and baseline models has a slight edgein certain classes. Moreover, the boosted and strong learners generally havesmaller whiskers than the baseline, indicating that they are moderately morerobust.Figure 5.1: Average After Sigmoid F1 Score by Model and Class.Figure 5.2: Average Before Sigmoid F1 Score by Model and Class.Figure 5.3: Noisy Or F1 Score by Model and Class.625.2. BoostingFigure 5.4: Vote F1 Score by Model and Class.5.2 BoostingGradient boosting was performed a fixed number of 4 times per fold; thereduction in loss relative to the unboosted model for the fold is presentedbelow. Classes 1 and 3 saw excellent loss reduction through boosting, withmedian improvement near 25% and 20%, respectively. Class 4 and thekidney class saw median improvement near 10% and 15%, respectively. Class2 and the tumor class saw improvement to the median loss, but also sawpoorer performance for some examples after boosting. This suggests thatthe residual loss is more complex than the initial loss, and the weak learnersare not able to properly model the increasingly complex function. In thecase of class 2, this could be caused by the very limited dataset promotingoverfitting.(a) KiTS19 Kidney (b) KiTS19 TumorFigure 5.5: KiTS19 Boosting Improvement.635.3. Area Under the Curve Evaluation(a) Severstal Class 1 (b) Severstal Class 2(c) Severstal Class 3 (d) Severstal Class 4Figure 5.6: Severstal Boosting Improvement.5.3 Area Under the Curve EvaluationSince the datasets are sparse, this analysis uses the area under the PRC.The curve is generated by varying the final threshold during inferencing,and area is calculated using the trapezoid rule. The below plots also showthe precision-recall points of baseline model; as can be seen, changing thethreshold of the baseline model does very little to move the points on thecurve. This shows that the aggregated models have an advantage in be-ing able to prefer one of precision or recall over the other; this can haveimplications for specific applications where false-positive or false-negativepredictions may have more significant practical impact.The weak learners underperform the baseline in all cases, but the curve645.3. Area Under the Curve Evaluationmoves towards the top left corner through boosting and increasing the net-work size; on all four of the Severstal classes, they come very close to, orslightly surpass, the baseline on the PRC. On the KiTS19 dataset, however,they are outperformed by the baseline. Note that the curve in the tumorclass has an inflection point which the other curves do not; this is unusualand is a sign that the model is having issues isolating the class.(a) Weak Learners (b) Boosted Learners (c) Strong LearnersFigure 5.7: Severstal Class 1 AUC.(a) Weak Learners (b) Boosted Learners (c) Strong LearnersFigure 5.8: Severstal Class 2 AUC.(a) Weak Learners (b) Boosted Learners (c) Strong LearnersFigure 5.9: Severstal Class 3 AUC.655.4. Layer Outputs and Class Confusion(a) Weak Learners (b) Boosted Learners (c) Strong LearnersFigure 5.10: Severstal Class 4 AUC.(a) Weak Learners (b) Boosted Learners (c) Strong LearnersFigure 5.11: KiTS19 Kidney AUC.(a) Weak Learners (b) Boosted Learners (c) Strong LearnersFigure 5.12: KiTS19 Tumor AUC.5.4 Layer Outputs and Class ConfusionThe activation layer outputs of the full-domain models utilize a widthroughly double that of the baseline; this is expected as rectification clipshalf of the domain. The increased output domain is also observed in theconvolutional layers, where the utilized width increased from 14.61 to 19.47.665.4. Layer Outputs and Class ConfusionThe standard deviation of the output values also increased for both theactivation and convolutional layers, from 0.41 to 1.25 and 7.18 to 10.47respectively.(a) Baseline Convolution (b) Full-Domain ConvolutionFigure 5.13: Convolution Layer Output Histograms.(a) Baseline Activation (b) Full-Domain ActivationFigure 5.14: Activation Layer Output Histograms.The peak in full-domain activation layer outputs is only 26.3% as highas the rectified counterpart, and is shifted from 0.0 to -1.57. The shift islikely caused by the negative y-intercept of the x − cos(x) function. Thereduction in the peak is even more pronounced in the convolutional layeroutputs, with the height of the full-domain peak only 15.6% of the baseline.675.4. Layer Outputs and Class ConfusionInterestingly, the peaks in both convolutional histograms are at 0.0, a resultof normalization and the network correcting for the negative y-intercept ofthe activation function. Values of 0 create sparsity in the network, whichhas value in disentangling changes in one section of the image from another;however, in fully convolution neural networks, sparsity is enforced throughthe nature of the connections, and forcing further sparsity appears to providelittle value.The confusion matrices presented are a prediction-normalized summa-tion of the confusions for each class, where the class confusion matricesconsider all other classes to be part of the background. Most predictionerrors are false-positive predictions from the background. The notable ex-ception is the tumor class, which shows significant confusion with both thebackground and the kidney class. This double-confusion leads to a poorlyperforming model, particularly for the extremely small networks. It wouldlikely be beneficial to use a multi-class segmentation to isolate the tumorclass, which would encourage the network to select between one of the twoclasses.(a) Severstal Baseline (b) Severstal Strong (c) Severstal BoostedFigure 5.15: Severstal Confusion matrices.(a) KiTS19 Baseline (b) KiTS19 Strong (c) KiTS19 BoostedFigure 5.16: KiTS19 Confusion matrices.685.5. Selected Predictions5.5 Selected Predictions(a) Baseline (b) Weak(c) Boosted (d) StrongFigure 5.17: KiTS19 Kidney - Case 00003, Slice 62.A pair of examples demonstrating the confusion between the tumor andkidney classes in the KiTS19 dataset are shown in figures 5.17 and 5.18.Notably, the baseline performs very well, with only small issues aroundthe boundaries. All other methods have the tumor included in the kidney695.5. Selected Predictionssegmentation, as well as large parts of the kidneys included in the tumorsegmentation.(a) Baseline (b) Weak(c) Boosted (d) StrongFigure 5.18: KiTS19 Tumor - Case 00003, Slice 62.Severstal Class 1 is also notable, as there are many small components.In the example shown in figure 5.19, the baseline misses one of these compo-nents completely, but has no false positive components. The weak network705.5. Selected Predictionsis able to find the component missed by the baseline, but introduces threefalse-positive components; boosting is able to remove one of these false pos-itives, and the strong network is able to remove two.(a) Baseline(b) Weak(c) Boosted(d) StrongFigure 5.19: Severstal Class 1 - 3fdc95c42.71Chapter 6Conclusion and Future WorkScalability of neural networks used for image segmentation is explored.Where previous scalability studies focused mainly on network depth, width,and image size, this study also considered ensembling and gradient boosting.An extremely efficient modified U-Net was created by limiting the numberof feature channels and using a constant number of channels per layer. Thisis a significant departure from previous work, as convolution is O(n2) innetwork width. A grid search on model depth and width was used to pro-duce a miniature network using only 0.5% of the trainable parameters whencompared to the EfficientNetB0 baseline; this network is able to achieve F1scores exceeding 90% of the baseline in 5 of the 6 classes taken from theSeverstal and KiTS19 datasets.The smaller networks employ a full-domain activation function, x −cos(x); this is novel, as most image processing networks use rectified lin-ear functions. Compared to the baseline model, the full-domain activationnetworks exhibit dramatic reductions in sparsity-promoting, 0-value peaksin layer output histograms; activation layer peaks are reduced from 19.39%to 5.10%, and convolution layer peaks are reduced from 15.69% to 2.45%.Similarly, the utilized width in outputs is increased from 14.61 to 19.47,and 3.17 to 6.58 for the convolution and activation layers, respectively. Thesuccess of this activation function in promoting network density has impli-cations for all types of memory-constrained neural network applications.Gradient boosting is a particularly interesting technique which has seenlimited adoption in neural network applications. This is significant, as boost-ing is more efficient than the O(n2) computational complexity in networksby width. Gradient boosting of the miniature network improved medianloss in all 6 classes, but did produce underperforming outliers in 2 of the 6classes. It is likely that the limited dataset for Severstal Class 2 constrainedscalability and promoted overfitting. For the tumor class, it is likely thatan increasingly complex residual loss is too hard for the small networks tomodel. In fact, detecting tumors in the KiTS19 dataset was found to bethe most difficult task, and the all of the small networks significantly underperformed the baseline.72Chapter 6. Conclusion and Future WorkScaling through ensembling and boosting could be combined with recentadvances in scaling networks by depth and width to produce even more ef-ficient networks. The success of the approach has significant implicationsfor deployed applications; not only is the overall network much smaller,the miniature sub-networks can each be calculated independently and ag-gregated with simple operations. This introduces a real possibility for ef-ficient distributed inferencing, decoupling performance from the maximumsize calculable by a single piece of hardware. In a distributed computingenvironment, however, one must be aware of latency and reliability of eachcomponent. Such problems necessitate the evaluation of the aggregated pre-diction approach through the random exclusion of predictions from the finalaggregation. Such exclusions would simulate remote system failures, givinga sense of how robust the approach would be in practice.A 3D segmentation would likely do better in tumor segmentation; thethird dimension allows better distinction between tumors, which are gener-ally spherical, and organs, which are generally not. The generation of the2D dataset also likely contributed to poor results in the tumor class, as asingle pixel was sufficient for a slice to be considered part of that class. Thetumor class had approximately 12% of examples with 0.1% class coverage orless; in contrast, the kidney class had approximately 3.5%. Extremely smallcomponents are hard to isolate, and errors in sparse predictions are partic-ularly bad for metrics; these examples tend to generate extreme F1 scores.Moreover, the aggregation of 2D F1 scores is not necessarily equivalent tothe 3D F1 score, even for the same predictions. Finally, the best case tobe made for the improved performance of 3D segmentation are the KiTS19results; the winner used a 3D U-Net and achieved mean F1 scores of 0.97and 0.85 for the kidney and tumor class, respectively [26, 30].The move to 3D also unlocks many interesting avenues of research inHuman-Computer Interaction (HCI). In medical applications, the trend to-wards less invasive surgical procedures has necessitated additional tools toassist the surgeon. Operative information imaging systems are employed toprovide information which is obscured because of reduced operative access.Common examples of such systems include endoscopic video and Intraopera-tive Ultrasounds (IOUS). Information from these systems is often combinedwith pre-operative information images such as CT scans.In 2015, Wang et. al provided a 3D model for pre-operative planningand intra-operative assistance, and found that mean operation time wasshortened and patient blood loss was decreased. In 2019, Shirk et al. createda 3D model where the surgeon could virtually orbit around the model bymoving their head, and zoom in and out using a small button on the headset.73Chapter 6. Conclusion and Future WorkIn their experiment, operating room time was reduced by an hour, clamptime by 4.2 minutes, blood loss nearly halved, and the length of stay greatlyreduced. This is a great result for a rudimentary 3D model and a relativelycheap Virtual Reality (VR) headset.Wake et. al endeavoured to compare three possible Augmented Reality(AR) and VR tools to assist in patient education of kidney stone procedures[74]. In the study, a 3D printed (physical) model, a 3D computer model, anda 3D AR model were provided to patients before their procedure. Patientsreported the all of 3D models helped to learn about their anatomy, cancer,the surgery they would undergo, and possible complications of the surgery.As part of this thesis, work has begun on an AR solution for displaying3D models using a smartphone. The user is able to translate, scale, androtate the model, as well as insert text annotations. It is possible to hideparts of the model; for example, the user can hide the outer shell of the lungsto see the airways and candidate nodules. Candidate nodules are highlightedin red. Screenshots of this work are shown in figure 6.1.(a) Lung Model(b) Lung InteriorFigure 6.1: Augmented Reality Smartphone Application.74Chapter 6. Conclusion and Future WorkParticularly lacking in this area of research is evaluation of tools in clin-ical settings. Of the 52 papers selected for review in a survey paper on VRand AR applications in renal interventions, only 20 had been evaluated inclinical settings at all [12]. Most of these had a small sample size (n <10). The study found only paper which evaluated their system’s effect onpatient outcome metrics such as time in the operating room and length ofthe patient’s stay in the hospital.75Bibliography[1] Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher,J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G.,Naidich, D. P., and Shetty, S. End-to-end lung cancer screen-ing with three-dimensional deep learning on low-dose chest computedtomography. Nature Medicine 25, 6 (Jun 2019), 954–961. → pages 42[2] Badirli, S., Liu, X., Xing, Z., Bhowmik, A., and Keerthi,S. S. Gradient boosting neural networks: Grownet. arXiv preprintarXiv:2002.07971 (2020). → pages 2, 40[3] Badrinarayanan, V., Kendall, A., and Cipolla, R. Segnet: Adeep convolutional encoder-decoder architecture for image segmenta-tion. IEEE transactions on pattern analysis and machine intelligence39, 12 (2017), 2481–2495. → pages 1[4] Bengio, Y., Simard, P., and Frasconi, P. Learning long-termdependencies with gradient descent is difficult. IEEE transactions onneural networks 5, 2 (1994), 157–166. → pages 32[5] Berens, M., et al. Znet. http://luna16.grand-challenge.org/media/luna16/public_html/pdfs/ZNET_NDET_160831.pdf/, 2016.[Online; accessed 3-April-2019]. → pages 45[6] Boyd, S., Boyd, S. P., and Vandenberghe, L. Convex optimiza-tion. Cambridge university press, 2004. → pages 24[7] Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A.,Druzhinin, M., and Kalinin, A. A. Albumentations: Fast andflexible image augmentations. Information 11, 2 (2020). → pages 60[8] Caruana, R., Lawrence, S., and Giles, C. L. Overfitting inneural nets: Backpropagation, conjugate gradient, and early stopping.In Advances in neural information processing systems (2001), pp. 402–408. → pages 2876Bibliography[9] Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam,H. Encoder-decoder with atrous separable convolution for semanticimage segmentation. In Computer Vision – ECCV 2018 (Cham, 2018),V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., SpringerInternational Publishing, pp. 833–851. → pages 1[10] C¸ic¸ek, O¨., Abdulkadir, A., Lienkamp, S. S., Brox, T., andRonneberger, O. 3d u-net: learning dense volumetric segmenta-tion from sparse annotation. In International conference on medicalimage computing and computer-assisted intervention (2016), Springer,pp. 424–432. → pages 1, 42, 44, 54[11] Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fastand accurate deep network learning by exponential linear units (elus).CoRR abs/1511.07289 (2015). → pages xxx[12] Detmer, F. J., Hettig, J., Schindele, D., Schostak, M., andHansen, C. Virtual and augmented reality systems for renal interven-tions: A systematic review. IEEE Reviews in Biomedical Engineering10 (2017), 78–94. → pages 75[13] Dice, L. R. Measures of the amount of ecologic association betweenspecies. Ecology 26, 3 (1945), 297–302. → pages 15[14] Dumoulin, V., and Visin, F. A guide to convolution arithmetic fordeep learning. arXiv preprint arXiv:1603.07285 (2016). → pages 9[15] et. al, C. W. 2019 kidney tumor segmentation challenge. https://kits19.grand-challenge.org/, 2019. [Online; accessed 4-June-2019].→ pages 2, 49, 50[16] Frankle, J., Schwab, D. J., and Morcos, A. S. Training batch-norm and only batchnorm: On the expressive power of random featuresin cnns, 2020. → pages 35[17] Freund, Y., Schapire, R., and Abe, N. A short introduction toboosting. Journal-Japanese Society For Artificial Intelligence 14, 771-780 (1999), 1612. → pages 2, 40[18] Freund, Y., and Schapire, R. E. A desicion-theoretic generalizationof on-line learning and an application to boosting. In ComputationalLearning Theory (Berlin, Heidelberg, 1995), P. Vita´nyi, Ed., SpringerBerlin Heidelberg, pp. 23–37. → pages 2, 4077Bibliography[19] Ginneken, B. Fifty years of computer analysis in chest imaging: rule-based, machine learning, deep learning. Radiological Physics and Tech-nology 1, 10 (2017), 23–32. → pages 41, 46[20] Glorot, X., and Bengio, Y. Understanding the difficulty of train-ing deep feedforward neural networks. In Proceedings of the thirteenthinternational conference on artificial intelligence and statistics (2010),pp. 249–256. → pages xxxi, 32, 33[21] Glorot, X., Bordes, A., and Bengio, Y. Deep Sparse RectifierNeural Networks. In 14th International Conference on Artificial In-telligence and Statistics (Fort Lauderdale, United States, Apr. 2011),vol. 15, pp. 315–323. → pages 19[22] Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning.MIT Press, 2016. http://www.deeplearningbook.org. → pages xxxi,2, 7, 17, 27, 32, 34, 39[23] He, K., Zhang, X., Ren, S., and Sun, J. Spatial pyramid pooling indeep convolutional networks for visual recognition. IEEE transactionson pattern analysis and machine intelligence 37, 9 (2015), 1904–1916.→ pages 1[24] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning forimage recognition. In The IEEE Conference on Computer Vision andPattern Recognition (CVPR) (June 2016). → pages 33[25] Heller, N., Sathianathen, N., Kalapara, A., Walczak, E.,Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Rengel,Z., Oestreich, M., et al. The kits19 challenge data: 300 kidneytumor cases with clinical context, ct semantic segmentations, and sur-gical outcomes. arXiv preprint arXiv:1904.00445 (2019). → pages 1,43, 51[26] Heller, N., Sathianathen, N., Kalapara, A., Walczak, E.,Moore, K., Kaluzniak, H., Rosenberg, J., Blake, P., Ren-gel, Z., Oestreich, M., et al. Kits19 results. http://results.kits-challenge.org/miccai2019/, 2019. [Online; accessed 29-July-2019]. → pages 1, 49, 73[27] Inc., K. Severstal: Steel defect detection. https://www.kaggle.com/c/severstal-steel-defect-detection/, 2019. [Online; accessed 9-June-2020]. → pages 2, 4978Bibliography[28] Ioffe, S., and Szegedy, C. Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift. arXiv preprintarXiv:1502.03167 (2015). → pages 35[29] Isensee, F. nn-unet source code. https://github.com/MIC-DKFZ/nnUNet, 2019. [Online; accessed 4-June-2019]. → pages 43[30] Isensee, F., and Maier-Hein, K. H. An attempt at beatingthe 3d u-net. http://results.kits-challenge.org/miccai2019/manuscripts/Isensee_1e.pdf, 2019. [Online; accessed 29-July-2019].→ pages 1, 73[31] Isensee, F., Petersen, J., Klein, A., Zimmerer, D., Jaeger,P. F., Kohl, S., Wasserthal, J., Koehler, G., Norajitra, T.,Wirkert, S., and Maier-Hein, K. H. nnu-net: Self-adapting frame-work for u-net-based medical image segmentation, 2018. → pages xxx,42[32] Isensee, F., Petersen, J., Kohl, S. A., Ja¨ger, P. F., andMaier-Hein, K. H. nnu-net: Breaking the spell on successful medicalimage segmentation. arXiv preprint arXiv:1904.08128 (2019). → pages34, 55[33] Kaggle. Kaggle: Your machine learning and data science community.https://www.kaggle.com/. [Online; accessed 9-June-2020]. → pages49[34] Kaggle. 2017 kaggle data science bowl. https://www.kaggle.com/c/data-science-bowl-2017, 2017. [Online; accessed 7-April-2019]. →pages 42, 45[35] Kingma, D. P., and Ba, J. Adam: A method for stochastic opti-mization. arXiv preprint arXiv:1412.6980 (2014). → pages 31, 32[36] Kuhnigk, J.-M., Dicken, V., Bornemann, L., Bakai, A., Wor-manns, D., Krass, S., and Peitgen, H.-O. Morphological segmen-tation and partial volume analysis for volumetry of solid pulmonarylesions in thoracic ct scans. IEEE Transactions on Medical Imaging 25,4 (2006), 417–434. → pages 46[37] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard,R. E., Hubbard, W., and Jackel, L. D. Backpropagation appliedto handwritten zip code recognition. Neural computation 1, 4 (1989),541–551. → pages 2679Bibliography[38] LeCun, Y., et al. Generalization and network design strategies. InConnectionism in perspective, Steels, Ed., vol. 19. Elsevier, 1989. →pages 22, 26[39] LeCun, Y. A., Bottou, L., Orr, G. B., and Mu¨ller, K.-R.Efficient BackProp. Springer Berlin Heidelberg, Berlin, Heidelberg,2012, pp. 9–48. → pages 21[40] lfz, and zakizhou. The solution of team ’grt123’ in dsb2017. https://github.com/lfz/DSB2017, 2019. [Online; accessed 8-April-2019]. →pages 47[41] Liao, F., Liang, M., Li, Z., Hu, X., and Song, S. Evaluate themalignancy of pulmonary nodules using the 3-d deep leaky noisy-ornetwork. IEEE Transactions on Neural Networks and Learning Systems(2019), 1–12. → pages 42, 50[42] Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dolla´r, P.Focal loss for dense object detection. In Proceedings of the IEEE in-ternational conference on computer vision (2017), pp. 2980–2988. →pages 25, 54[43] Liu, W., Rabinovich, A., and Berg, A. C. Parsenet: Lookingwider to see better. CoRR (2015). → pages 1[44] Luo, L., Xiong, Y., Liu, Y., and Sun, X. Adaptive gradient meth-ods with dynamic bound of learning rate, 2019. → pages 32[45] Luo, W., Li, Y., Urtasun, R., and Zemel, R. Understandingthe effective receptive field in deep convolutional neural networks. InAdvances in neural information processing systems (2016), pp. 4898–4906. → pages xxxvii, 10[46] MANI-LEVITSKA, P. Chapter 1.1 - characterizations of convex sets.In Handbook of Convex Geometry, P. GRUBER and J. WILLS, Eds.North-Holland, Amsterdam, 1993, pp. 19 – 41. → pages xxxiv, 24[47] Marr, D., Hildreth, E., and Brenner, S. Theory of edge detec-tion. Proceedings of the Royal Society of London. Series B. BiologicalSciences 207, 1167 (1980), 187–217. → pages 12[48] Masnadi-Shirazi, H., Mahadevan, V., and Vasconcelos, N. Onthe design of robust classifiers for computer vision. In 2010 IEEE Com-80Bibliographyputer Society Conference on Computer Vision and Pattern Recognition(2010), IEEE, pp. 779–786. → pages 24[49] Messay, T., Hardie, R. C., and Rogers, S. K. A new compu-tationally efficient cad system for pulmonary nodule detection in ctimagery. Medical image analysis 14, 3 (2010), 390–406. → pages 45[50] Minsky, M., and Papert, S. A. Perceptrons: An Introduction toComputational Geometry. MIT Press, 1969. → pages 17[51] Pinheiro, P. O., Collobert, R., and Dolla´r, P. Learning to seg-ment object candidates. In Advances in Neural Information ProcessingSystems (2015), pp. 1990–1998. → pages 1[52] Qian, N. On the momentum term in gradient descent learning al-gorithms. Neural Networks 12, 1 (1999), 145 – 151. → pages xxxvi,30[53] Ramachandran, P., Zoph, B., and Le, Q. V. Searching for acti-vation functions, 2017. → pages xxx, 54[54] Reddi, S., Kale, S., and Kumar, S. On the convergence of adamand beyond. In International Conference on Learning Representations(2018). → pages 32[55] Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu-tional networks for biomedical image segmentation. In InternationalConference on Medical image computing and computer-assisted inter-vention (2015), Springer, pp. 234–241. → pages 1, 34, 42, 54[56] Rosenblatt, F. The perceptron: a probabilistic model for informationstorage and organization in the brain. Psychological review 65, 6 (1958),386. → pages 17, 18[57] Ruder, S. An overview of gradient descent optimization algorithms.arXiv preprint arXiv:1609.04747 (2016). → pages 32[58] Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. Howdoes batch normalization help optimization? In Advances in NeuralInformation Processing Systems (2018), pp. 2483–2493. → pages 35[59] Schwenk, H., and Bengio, Y. Training methods for adaptive boost-ing of neural networks. In Advances in neural information processingsystems (1998), pp. 647–653. → pages 281Bibliography[60] Setio, A. A. A., et al. Validation, comparison, and combination ofalgorithms for automatic detection of pulmonary nodules in computedtomography images: the luna16 challenge. Medical image analysis 42(2017), 1–13. → pages 45[61] Simpson, A. L., et al. Medical segmentation decathlon. http://medicaldecathlon.com/, 2019. [Online; accessed 12-June-2019]. →pages 1[62] Sorensen, T. A. A method of establishing groups of equal amplitudein plant sociology based on similarity of species content and its appli-cation to analyses of the vegetation on danish commons. Biol. Skar. 5(1948), 1–34. → pages 15[63] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M.,Howard, A., and Le, Q. V. Mnasnet: Platform-aware neural ar-chitecture search for mobile. In The IEEE Conference on ComputerVision and Pattern Recognition (CVPR) (June 2019). → pages 2, 36[64] Tan, M., and Le, Q. V. Efficientnet: Rethinking model scalingfor convolutional neural networks. arXiv preprint arXiv:1905.11946(2019). → pages 2, 36, 53, 55[65] Team, B. The blender python api reference. https://docs.blender.org/api/249PythonDoc/index.html, 2019. [Online; accessed 8-April-2019]. → pages 48[66] Torres, E. L., et al. Multi opening and threshold cad(mot m5l). http://luna16.grand-challenge.org/media/luna16/public_html/pdfs/20161018_161506_MOT_M5Lv1_NDET.pdf/, 2016.[Online; accessed 3-April-2019]. → pages 46[67] Torres, E. L., Fiorina, E., Pennazio, F., Peroni, C., Saletta,M., Camarlinghi, N., Fantacci, M. E., and Cerello, P. Largescale validation of the m5l lung cad on heterogeneous ct datasets. Med-ical physics 42, 4 (2015), 1477–1489. → pages 46[68] Ulku, I., and Akagunduz, E. A survey on deep learning-basedarchitectures for semantic segmentation on 2d images. arXiv preprintarXiv:1912.10230 (2019). → pages 1, 16[69] User:MatrinThoma, W. C. Semantic segmenta-tion. https://commons.wikimedia.org/wiki/File:82BibliographyEmiMa-099-semantic-segmentation.png, 2017. [Online; accessed22-June-2020] Used under CC0 1.0 Public Domain Dedication. →pages 6[70] User:MTheiler, W. Objects detected with opencv’s deep neuralnetwork module (dnn). https://en.wikipedia.org/wiki/File:Detected-with-YOLO--Schreibtisch-mit-Objekten.jpg, 2019.[Online; accessed 22-June-2020] Used under CC BY-SA 4.0 license. →pages 5[71] van Ginneken, B., et al. Comparing and combining algorithms forcomputer-aided detection of pulmonary nodules in computed tomogra-phy scans: The anode09 study. Medical Image Analysis 14, 6 (2010),707 – 722. → pages 45[72] van Ginneken, B., et al. Luna16 announcement (sept 1, 2017).https://luna16.grand-challenge.org/announcements/, 2017. [On-line; accessed 8-April-2019]. → pages 49[73] van Ginneken, B., Kerkstra, S., and Meakin, J. Grand chal-lenges in biomedical image analysis. https://grand-challenge.org/,2019. [Online; accessed 29-July-2019]. → pages 49[74] Wake, N., Rosenkrantz, A. B., Huang, R., Park, K. U.,Wysock, J. S., Taneja, S. S., Huang, W. C., Sodickson, D. K.,and Chandarana, H. Patient-specific 3d printed and augmented re-ality kidney and prostate cancer models: impact on patient education.3D Printing in Medicine 5, 1 (Feb 2019), 4. → pages 74[75] Whitehill, J. Climbing the kaggle leaderboard by exploiting the log-loss oracle. In Workshops at the Thirty-Second AAAI Conference onArtificial Intelligence (2018). → pages 49[76] Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters,J., and Schmidhuber, J. Natural evolution strategies. Journal ofMachine Learning Research 15, 27 (2014), 949–980. → pages 32[77] Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., andRecht, B. The marginal value of adaptive gradient methods in ma-chine learning. In Advances in Neural Information Processing Sys-tems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus,S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017,pp. 4148–4158. → pages 3283Bibliography[78] Yakubovskiy, P., and others. Segmentation models with pre-trained backbones. keras and tensorflow keras. https://github.com/qubvel/segmentation_models/, 2020. [Online; accessed 5-May-2020;git commit 94f624b7029deb463c859efbd92fa26f512b52b8]. → pages 55[79] Yazdani, M. Example architecture of u-net for produc-ing k 256-by-256 image masks for a 256-by-256 rgb im-age. https://commons.wikimedia.org/wiki/File:Example_architecture_of_U-Net_for_producing_k_256-by-256_image_masks_for_a_256-by-256_RGB_image.png, 2019. [Online; accessed22-June-2020] Used under CC BY-SA 4.0 license. → pages 44[80] Zhou, Q. Pymesh - geometry processing library for python. https://pymesh.readthedocs.io/en/latest/, 2019. [Online; accessed 8-April-2019]. → pages 4784Appendix85Appendix ADataset PropertiesA.1 SeverstalFigure A.1: Class Instance Counts in the Severstal Dataset.86A.1. SeverstalFigure A.2: Class 1 Coverage in the Severstal Dataset.Figure A.3: Class 2 Coverage in the Severstal Dataset.87A.1. SeverstalFigure A.4: Class 3 Coverage in the Severstal Dataset.Figure A.5: Class 4 Coverage in the Severstal Dataset.88A.1. Severstal(a) Class 1 Component Count (b) Class 1 Component SizeFigure A.6: Components in Class 1 of the Severstal Dataset.(a) Class 2 Component Count (b) Class 2 Component SizeFigure A.7: Components in Class 2 of the Severstal Dataset.89A.1. Severstal(a) Class 3 Component Count (b) Class 3 Component SizeFigure A.8: Components in Class 3 of the Severstal Dataset.(a) Class 4 Component Count (b) Class 4 Component SizeFigure A.9: Components in Class 4 of the Severstal Dataset.90A.2. Kits19A.2 Kits19Figure A.10: Class Instance Counts in the KiTS19 Dataset.Figure A.11: Class Kidney Coverage in the KiTS19 Dataset.91A.2. Kits19Figure A.12: Class Tumor Coverage in the KiTS19 Dataset.(a) Class Kidney Component Count (b) Class Kidney Component SizeFigure A.13: Components in Class Kidney of the KiTS19 Dataset.92A.2. Kits19(a) Class Tumor Component Count (b) Class Tumor Component SizeFigure A.14: Components in Class Tumor of the KiTS19 Dataset.93Appendix BFull ResultsTable B.1: Summary of Performance MetricsBaseline Weak Learners1 Boosted Learners1 Strong Learners1µf1 AUC2 µf1 AUC2 µf1 AUC2 µf1 AUC2SeverstalClass 1 0.65 0.70 0.60 0.71 0.66 0.77 0.66 0.78Class 2 0.70 0.74 0.69 0.81 0.69 0.82 0.71 0.85Class 3 0.77 0.84 0.69 0.83 0.74 0.86 0.78 0.90Class 4 0.84 0.89 0.78 0.90 0.81 0.92 0.82 0.93KiTS19 Kidney 0.94 0.95 0.86 0.95 0.87 0.96 0.90 0.98Tumor 0.62 0.74 0.29 0.49 0.31 0.49 0.43 0.631Using the best aggregation method and threshold2Calculated on Recall vs. False-Discovery RateTable B.2: Summary of Activation Layer OutputsBaseline Weak Learners Boosted Learners Strong LearnersPeak Width Peak Width Peak Width Peak WidthSeverstalClass 1 18.51% at 0.00 99.14% in 2.68 4.19% at -1.58 99.07% in 6.55 4.46% at -1.57 98.82% in 6.52 4.67% at -1.57 98.03% in 6.50Class 2 17.85% at 0.00 98.93% in 2.52 3.85% at -1.58 98.81% in 6.67 4.18% at-1.54 99.03% in 6.49 5.50% at -1.54 98.33% in 6.43Class 3 18.84% at 0.00 98.81% in 2.68 4.99% at -1.58 98.87% in 6.75 5.10% at -1.58 98.63% in 6.71 4.84% at -1.58 98.03% in 6.82Class 4 19.60% at 0.00 98.95% in 2.87 4.70% at -1.58 98.97% in 6.62 4.35% at -1.58 99.00% in 6.69 3.52% at -1.58 98.22% in 6.57KiTS19 Kidney 20.01% at 0.00 99.05% in 3.27 5.90% at -1.58 99.23% in 6.58 6.04% at -1.57 98.62% in 6.53 5.83% at -1.57 97.86% in 6.56Tumor 19.40% at 0.00 98.96% at 3.20 7.12% at -1.57 98.94% in 6.52 6.94% at -1.57 98.71% in 6.52 5.65% at -1.57 97.79% in 12.58Mean 19.04% 2.87 5.13% 6.62 5.18% 6.58 5.00% 7.58Table B.3: Summary of Convolution Layer OutputsBaseline Weak Learners Boosted Learners Strong LearnersPeak Width Peak Width Peak Width Peak WidthSeverstalClass 1 14.36% at 0.00 93.79% in 10.23 1.60% at -0.01 97.45% in 16.18 1.59% at -0.01 97.11% in 17.59 1.93% at 0.00 97.28% in 18.56Class 2 13.80% at 0.00 95.51% in 13.37 4.52% at -0.01 97.23% in 12.61 3.83% at 0.00 97.98% in 11.70 3.98% at -0.01 97.79% in 11.87Class 3 13.63% at 0.00 96.89% in 15.37 1.13% at 0.02 97.98% in 18.87 1.18% at -0.01 98.79% in 19.56 2.24% at 0.00 100.00% in 19.99Class 4 14.47% at 0.00 95.53% in 14.21 0.90% at 0.02 97.15% in 12.98 0.70% at 0.49 97.11% in 13.89 0.81% at 0.00 97.18% in 16.61KiTS19 Kidney 17.62% at 0.00 96.15% in 15.66 3.79% at 0.00 98.29% in 18.93 3.36% at 0.00 96.82% in 18.28 3.70% at -0.01 98.55% in 19.47Tumor 16.96% at 0.00 94.93% at 13.26 3.33% at 0.00 97.20% in 16.19 3.04% at -0.01 97.18% in 15.83 3.29% at -0.01 96.62% in 19.02Mean 15.14% 13.68 2.55% 15.96 2.28% 16.14 2.66% 17.5994B.1. SeverstalB.1 SeverstalB.1.1 SummaryTable B.4: Class 1 ResultsBest F1 ScoreFalse-Positive AUC False-Discovery AUCµ σ f˜1Baseline 0.65 0.20 0.69 0.86 0.70WeakLearners Average After 0.60 0.18 0.65 0.99 0.71Average Before 0.60 0.18 0.65 0.99 0.70Noisy Or 0.60 0.18 0.64 0.99 0.70Vote 0.46 0.22 0.48 0.70 0.59BoostedLearners Average After 0.66 0.17 0.71 1.00 0.77Average Before 0.65 0.17 0.71 1.00 0.77Noisy Or 0.65 0.17 0.71 1.00 0.76Vote 0.60 0.20 0.65 0.80 0.69StrongLearners Average After 0.66 0.20 0.73 1.00 0.78Average Before 0.66 0.20 0.73 1.00 0.78Noisy Or 0.66 0.19 0.73 1.00 0.76Vote 0.63 0.19 0.67 0.82 0.71Table B.5: Class 2 ResultsBest F1 ScoreFalse-Positive AUC False-Discovery AUCµ σ f˜1Baseline 0.70 0.19 0.76 0.87 0.74WeakLearners Average After 0.69 0.14 0.69 1.00 0.81Average Before 0.69 0.14 0.72 1.00 0.81Noisy Or 0.68 0.14 0.71 1.00 0.79Vote 0.65 0.18 0.70 0.86 0.73BoostedLearners Average After 0.69 0.16 0.74 1.00 0.82Average Before 0.69 0.16 0.74 1.00 0.82Noisy Or 0.69 0.17 0.74 1.00 0.78Vote 0.67 0.17 0.72 0.86 0.74StrongLearners Average After 0.71 0.13 0.70 1.00 0.85Average Before 0.70 0.13 0.69 1.00 0.85Noisy Or 0.70 0.13 0.72 1.00 0.81Vote 0.69 0.14 0.72 0.88 0.79Table B.6: Class 3 ResultsBest F1 ScoreFalse-Positive AUC False-Discovery AUCµ σ f˜1Baseline 0.77 0.19 0.83 0.93 0.84WeakLearners Average After 0.69 0.24 0.77 0.98 0.83Average Before 0.68 0.24 0.77 0.98 0.83Noisy Or 0.69 0.23 0.77 0.98 0.81Vote 0.64 0.29 0.75 0.80 0.74BoostedLearners Average After 0.74 0.22 0.81 0.99 0.86Average Before 0.74 0.21 0.80 0.99 0.86Noisy Or 0.74 0.21 0.80 0.99 0.83Vote 0.72 0.23 0.81 0.86 0.80StrongLearners Average After 0.78 0.20 0.85 0.99 0.90Average Before 0.78 0.20 0.85 0.99 0.90Noisy Or 0.78 0.19 0.84 0.99 0.87Vote 0.75 0.24 0.83 0.86 0.82Table B.7: Class 4 ResultsBest F1 ScoreFalse-Positive AUC False-Discovery AUCµ σ f˜1Baseline 0.84 0.14 0.89 0.94 0.89WeakLearners Average After 0.78 0.16 0.83 0.99 0.90Average Before 0.78 0.16 0.83 0.99 0.90Noisy Or 0.78 0.16 0.83 0.98 0.87Vote 0.76 0.19 0.81 0.87 0.83BoostedLearners Average After 0.81 0.16 0.85 0.99 0.92Average Before 0.81 0.16 0.85 0.99 0.92Noisy Or 0.81 0.16 0.85 0.99 0.88Vote 0.79 0.17 0.84 0.89 0.85StrongLearners Average After 0.82 0.16 0.87 0.99 0.93Average Before 0.82 0.16 0.86 0.99 0.93Noisy Or 0.82 0.16 0.87 0.99 0.90Vote 0.81 0.17 0.87 0.90 0.8795B.1. SeverstalB.1.2 BaselineB.1.2.1 Baseline Class 1(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.1: Severstal Class 1 Metrics for Baseline Model Dummy.(a) False-Positive AUC (b) False-Discovery AUCFigure B.2: Severstal Class 1 AUC for Baseline Model Dummy.96B.1. Severstal(a) Convolution (b) Normalization(c) ActivationFigure B.3: Severstal Class 1 Layer Outputs for Baseline Model.B.1.2.2 Best Performing Threshold HistogramFigure B.4: Class 1 F1 Score for threshold 0.5 with Baseline Model andSeverstal Dataset.97B.1. SeverstalB.1.2.3 Baseline Class 2(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.5: Severstal Class 2 Metrics for Baseline Model Dummy.(a) False-Positive AUC (b) False-Discovery AUCFigure B.6: Severstal Class 2 AUC for Baseline Model Dummy.98B.1. Severstal(a) Convolution (b) Normalization(c) ActivationFigure B.7: Severstal Class 2 Layer Outputs for Baseline Model.B.1.2.4 Best Performing Threshold HistogramFigure B.8: Class 2 F1 Score for threshold 0.5 with Baseline Model andSeverstal Dataset.99B.1. SeverstalB.1.2.5 Baseline Class 3(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.9: Severstal Class 3 Metrics for Baseline Model Dummy.(a) False-Positive AUC (b) False-Discovery AUCFigure B.10: Severstal Class 3 AUC for Baseline Model Dummy.100B.1. Severstal(a) Convolution (b) Normalization(c) ActivationFigure B.11: Severstal Class 3 Layer Outputs for Baseline Model.B.1.2.6 Best Performing Threshold HistogramFigure B.12: Class 3 F1 Score for threshold 0.5 with Baseline Model andSeverstal Dataset.101B.1. SeverstalB.1.2.7 Baseline Class 4(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.13: Severstal Class 4 Metrics for Baseline Model Dummy.(a) False-Positive AUC (b) False-Discovery AUCFigure B.14: Severstal Class 4 AUC for Baseline Model Dummy.102B.1. Severstal(a) Convolution (b) Normalization(c) ActivationFigure B.15: Severstal Class 4 Layer Outputs for Baseline Model.B.1.2.8 Best Performing Threshold HistogramFigure B.16: Class 4 F1 Score for threshold 0.5 with Baseline Model andSeverstal Dataset.103B.1. SeverstalB.1.3 Weak LearnersB.1.3.1 Weak Learners Class 1 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.17: Severstal Class 1 Metrics for Weak Model and Average AfterAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.18: Severstal Class 1 AUC for Weak Model and Average AfterAggregation.104B.1. SeverstalB.1.3.2 Weak Learners Class 1 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.19: Severstal Class 1 Metrics for Weak Model and Average BeforeAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.20: Severstal Class 1 AUC for Weak Model and Average BeforeAggregation.105B.1. SeverstalB.1.3.3 Weak Learners Class 1 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.21: Severstal Class 1 Metrics for Weak Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.22: Severstal Class 1 AUC for Weak Model and Noisy Or Aggre-gation.106B.1. SeverstalB.1.3.4 Weak Learners Class 1 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.23: Severstal Class 1 Metrics for Weak Model and Vote Aggrega-tion.(a) False-Positive AUC (b) False-Discovery AUCFigure B.24: Severstal Class 1 AUC for Weak Model and Vote Aggregation.107B.1. SeverstalB.1.3.5 Weak Learners Class 1 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.25: Severstal Class 1 Layer Outputs for Weak Model.108B.1. SeverstalB.1.3.6 Best Performing Threshold HistogramsFigure B.26: Class 1 F1 Score for threshold 0.29, Weak Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.27: Class 1 F1 Score for threshold 0.28, Weak Learners, AverageBefore Aggregation, and Severstal Dataset.109B.1. SeverstalFigure B.28: Class 1 F1 Score for threshold, 0.97 Weak Learners, Noisy OrAggregation, and Severstal Dataset.Figure B.29: Class 1 F1 Score for threshold 0.00, Weak Learners, VoteAggregation, and Severstal Dataset.110B.1. SeverstalB.1.3.7 Weak Learners Class 2 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.30: Severstal Class 2 Metrics for Weak Model and Average AfterAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.31: Severstal Class 2 AUC for Weak Model and Average AfterAggregation.111B.1. SeverstalB.1.3.8 Weak Learners Class 2 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.32: Severstal Class 2 Metrics for Weak Model and Average BeforeAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.33: Severstal Class 2 AUC for Weak Model and Average BeforeAggregation.112B.1. SeverstalB.1.3.9 Weak Learners Class 2 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.34: Severstal Class 2 Metrics for Weak Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.35: Severstal Class 2 AUC for Weak Model and Noisy Or Aggre-gation.113B.1. SeverstalB.1.3.10 Weak Learners Class 2 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.36: Severstal Class 2 Metrics for Weak Model and Vote Aggrega-tion.(a) False-Positive AUC (b) False-Discovery AUCFigure B.37: Severstal Class 2 AUC for Weak Model and Vote Aggregation.114B.1. SeverstalB.1.3.11 Weak Learners Class 2 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.38: Severstal Class 2 Layer Outputs for Weak Model.115B.1. SeverstalB.1.3.12 Best Performing Threshold HistogramsFigure B.39: Class 2 F1 Score for threshold 0.34, Weak Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.40: Class 2 F1 Score for threshold 0.33, Weak Learners, AverageBefore Aggregation, and Severstal Dataset.116B.1. SeverstalFigure B.41: Class 2 F1 Score for threshold 0.99, Weak Learners, Noisy OrAggregation, and Severstal Dataset.Figure B.42: Class 2 F1 Score for threshold 0.00, Weak Learners, VoteAggregation, and Severstal Dataset.117B.1. SeverstalB.1.3.13 Weak Learners Class 3 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.43: Severstal Class 3 Metrics for Weak Model and Average AfterAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.44: Severstal Class 3 AUC for Weak Model and Average AfterAggregation.118B.1. SeverstalB.1.3.14 Weak Learners Class 3 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.45: Severstal Class 3 Metrics for Weak Model and Average BeforeAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.46: Severstal Class 3 AUC for Weak Model and Average BeforeAggregation.119B.1. SeverstalB.1.3.15 Weak Learners Class 3 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.47: Severstal Class 3 Metrics for Weak Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.48: Severstal Class 3 AUC for Weak Model and Noisy Or Aggre-gation.120B.1. SeverstalB.1.3.16 Weak Learners Class 3 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.49: Severstal Class 3 Metrics for Weak Model and Vote Aggrega-tion.(a) False-Positive AUC (b) False-Discovery AUCFigure B.50: Severstal Class 3 AUC for Weak Model and Vote Aggregation.121B.1. SeverstalB.1.3.17 Weak Learners Class 3 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.51: Severstal Class 3 Layer Outputs for Weak Model.122B.1. SeverstalB.1.3.18 Best Performing Threshold HistogramsFigure B.52: Class 3 F1 Score for threshold 0.30, Weak Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.53: Class 3 F1 Score for threshold 0.30, Weak Learners, AverageBefore Aggregation, and Severstal Dataset.123B.1. SeverstalFigure B.54: Class 3 F1 Score for threshold 0.97, Weak Learners, Noisy OrAggregation, and Severstal Dataset.Figure B.55: Class 3 F1 Score for threshold 0.00, Weak Learners, VoteAggregation, and Severstal Dataset.124B.1. SeverstalB.1.3.19 Weak Learners Class 4 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.56: Severstal Class 4 Metrics for Weak Model and Average AfterAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.57: Severstal Class 4 AUC for Weak Model and Average AfterAggregation.125B.1. SeverstalB.1.3.20 Weak Learners Class 4 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.58: Severstal Class 4 Metrics for Weak Model and Average BeforeAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.59: Severstal Class 4 AUC for Weak Model and Average BeforeAggregation.126B.1. SeverstalB.1.3.21 Weak Learners Class 4 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.60: Severstal Class 4 Metrics for Weak Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.61: Severstal Class 4 AUC for Weak Model and Noisy Or Aggre-gation.127B.1. SeverstalB.1.3.22 Weak Learners Class 4 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.62: Severstal Class 4 Metrics for Weak Model and Vote Aggrega-tion.(a) False-Positive AUC (b) False-Discovery AUCFigure B.63: Severstal Class 4 AUC for Weak Model and Vote Aggregation.128B.1. SeverstalB.1.3.23 Weak Learners Class 4 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.64: Severstal Class 4 Layer Outputs for Weak Model.129B.1. SeverstalB.1.3.24 Best Performing Threshold HistogramsFigure B.65: Class 4 F1 Score for threshold 0.31, Weak Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.66: Class 4 F1 Score for threshold 0.30, Weak Learners, AverageBefore Aggregation, and Severstal Dataset.130B.1. SeverstalFigure B.67: Class 4 F1 Score for threshold 0.98, Weak Learners, Noisy OrAggregation, and Severstal Dataset.Figure B.68: Class 4 F1 Score for threshold 0.00, Weak Learners, VoteAggregation, and Severstal Dataset.131B.1. SeverstalB.1.4 Boosted LearnersB.1.4.1 Boosted Learners Class 1 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.69: Severstal Class 1 Metrics for Boosted Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.70: Severstal Class 1 AUC for Boosted Model and Average AfterAggregation.132B.1. SeverstalB.1.4.2 Boosted Learners Class 1 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.71: Severstal Class 1 Metrics for Boosted Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.72: Severstal Class 1 AUC for Boosted Model and Average BeforeAggregation.133B.1. SeverstalB.1.4.3 Boosted Learners Class 1 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.73: Severstal Class 1 Metrics for Boosted Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.74: Severstal Class 1 AUC for Boosted Model and Noisy OrAggregation.134B.1. SeverstalB.1.4.4 Boosted Learners Class 1 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.75: Severstal Class 1 Metrics for Boosted Model and Vote Aggre-gation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.76: Severstal Class 1 AUC for Boosted Model and Vote Aggrega-tion.135B.1. SeverstalB.1.4.5 Boosted Learners Class 1 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.77: Severstal Class 1 Layer Outputs for Boosted Model.Figure B.78: Severstal Class 1 Boosting Improvement.136B.1. SeverstalB.1.4.6 Best Performing Threshold HistogramsFigure B.79: Class 1 F1 Score for threshold 0.30, Boosted Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.80: Class 1 F1 Score for threshold 0.30, Boosted Learners, AverageBefore Aggregation, and Severstal Dataset.137B.1. SeverstalFigure B.81: Class 1 F1 Score for threshold, 0.97 Boosted Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.82: Class 1 F1 Score for threshold 0.00, Boosted Learners, VoteAggregation, and Severstal Dataset.138B.1. SeverstalB.1.4.7 Boosted Learners Class 2 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.83: Severstal Class 2 Metrics for Boosted Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.84: Severstal Class 2 AUC for Boosted Model and Average AfterAggregation.139B.1. SeverstalB.1.4.8 Boosted Learners Class 2 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.85: Severstal Class 2 Metrics for Boosted Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.86: Severstal Class 2 AUC for Boosted Model and Average BeforeAggregation.140B.1. SeverstalB.1.4.9 Boosted Learners Class 2 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.87: Severstal Class 2 Metrics for Boosted Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.88: Severstal Class 2 AUC for Boosted Model and Noisy OrAggregation.141B.1. SeverstalB.1.4.10 Boosted Learners Class 2 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.89: Severstal Class 2 Metrics for Boosted Model and Vote Aggre-gation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.90: Severstal Class 2 AUC for Boosted Model and Vote Aggrega-tion.142B.1. SeverstalB.1.4.11 Boosted Learners Class 2 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.91: Severstal Class 2 Layer Outputs for Boosted Model.Figure B.92: Severstal Class 2 Boosting Improvement.143B.1. SeverstalB.1.4.12 Best Performing Threshold HistogramsFigure B.93: Class 2 F1 Score for threshold 0.34, Boosted Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.94: Class 2 F1 Score for threshold 0.33, Boosted Learners, AverageBefore Aggregation, and Severstal Dataset.144B.1. SeverstalFigure B.95: Class 2 F1 Score for threshold 0.98, Boosted Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.96: Class 2 F1 Score for threshold 0.00, Boosted Learners, VoteAggregation, and Severstal Dataset.145B.1. SeverstalB.1.4.13 Boosted Learners Class 3 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.97: Severstal Class 3 Metrics for Boosted Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.98: Severstal Class 3 AUC for Boosted Model and Average AfterAggregation.146B.1. SeverstalB.1.4.14 Boosted Learners Class 3 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.99: Severstal Class 3 Metrics for Boosted Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.100: Severstal Class 3 AUC for Boosted Model and AverageBefore Aggregation.147B.1. SeverstalB.1.4.15 Boosted Learners Class 3 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.101: Severstal Class 3 Metrics for Boosted Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.102: Severstal Class 3 AUC for Boosted Model and Noisy OrAggregation.148B.1. SeverstalB.1.4.16 Boosted Learners Class 3 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.103: Severstal Class 3 Metrics for Boosted Model and Vote Ag-gregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.104: Severstal Class 3 AUC for Boosted Model and Vote Aggre-gation.149B.1. SeverstalB.1.4.17 Boosted Learners Class 3 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.105: Severstal Class 3 Layer Outputs for Boosted Model.Figure B.106: Severstal Class 3 Boosting Improvement.150B.1. SeverstalB.1.4.18 Best Performing Threshold HistogramsFigure B.107: Class 3 F1 Score for threshold 0.33, Boosted Learners, Av-erage After Aggregation, and Severstal Dataset.Figure B.108: Class 3 F1 Score for threshold 0.32, Boosted Learners, Av-erage Before Aggregation, and Severstal Dataset.151B.1. SeverstalFigure B.109: Class 3 F1 Score for threshold 0.98, Boosted Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.110: Class 3 F1 Score for threshold 0.00, Boosted Learners, VoteAggregation, and Severstal Dataset.152B.1. SeverstalB.1.4.19 Boosted Learners Class 4 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.111: Severstal Class 4 Metrics for Boosted Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.112: Severstal Class 4 AUC for Boosted Model and Average AfterAggregation.153B.1. SeverstalB.1.4.20 Boosted Learners Class 4 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.113: Severstal Class 4 Metrics for Boosted Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.114: Severstal Class 4 AUC for Boosted Model and AverageBefore Aggregation.154B.1. SeverstalB.1.4.21 Boosted Learners Class 4 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.115: Severstal Class 4 Metrics for Boosted Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.116: Severstal Class 4 AUC for Boosted Model and Noisy OrAggregation.155B.1. SeverstalB.1.4.22 Boosted Learners Class 4 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.117: Severstal Class 4 Metrics for Boosted Model and Vote Ag-gregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.118: Severstal Class 4 AUC for Boosted Model and Vote Aggre-gation.156B.1. SeverstalB.1.4.23 Boosted Learners Class 4 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.119: Severstal Class 4 Layer Outputs for Boosted Model.Figure B.120: Severstal Class 4 Boosting Improvement.157B.1. SeverstalB.1.4.24 Best Performing Threshold HistogramsFigure B.121: Class 4 F1 Score for threshold 0.32, Boosted Learners, Av-erage After Aggregation, and Severstal Dataset.Figure B.122: Class 4 F1 Score for threshold 0.31, Boosted Learners, Av-erage Before Aggregation, and Severstal Dataset.158B.1. SeverstalFigure B.123: Class 4 F1 Score for threshold 0.98, Boosted Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.124: Class 4 F1 Score for threshold 0.00, Boosted Learners, VoteAggregation, and Severstal Dataset.159B.1. SeverstalB.1.5 Strong LearnersB.1.5.1 Strong Learners Class 1 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.125: Severstal Class 1 Metrics for Strong Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.126: Severstal Class 1 AUC for Strong Model and Average AfterAggregation.160B.1. SeverstalB.1.5.2 Strong Learners Class 1 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.127: Severstal Class 1 Metrics for Strong Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.128: Severstal Class 1 AUC for Strong Model and Average BeforeAggregation.161B.1. SeverstalB.1.5.3 Strong Learners Class 1 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.129: Severstal Class 1 Metrics for Strong Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.130: Severstal Class 1 AUC for Strong Model and Noisy OrAggregation.162B.1. SeverstalB.1.5.4 Strong Learners Class 1 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.131: Severstal Class 1 Metrics for Strong Model and Vote Aggre-gation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.132: Severstal Class 1 AUC for Strong Model and Vote Aggrega-tion.163B.1. SeverstalB.1.5.5 strong Learners Class 1 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.133: Severstal Class 1 Layer Outputs for Strong Model.164B.1. SeverstalB.1.5.6 Best Performing Threshold HistogramsFigure B.134: Class 1 F1 Score for threshold 0.33, Strong Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.135: Class 1 F1 Score for threshold 0.32, Strong Learners, AverageBefore Aggregation, and Severstal Dataset.165B.1. SeverstalFigure B.136: Class 1 F1 Score for threshold, 0.98 Strong Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.137: Class 1 F1 Score for threshold 0.00, Strong Learners, VoteAggregation, and Severstal Dataset.166B.1. SeverstalB.1.5.7 Strong Learners Class 2 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.138: Severstal Class 2 Metrics for Strong Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.139: Severstal Class 2 AUC for Strong Model and Average AfterAggregation.167B.1. SeverstalB.1.5.8 Strong Learners Class 2 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.140: Severstal Class 2 Metrics for Strong Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.141: Severstal Class 2 AUC for Strong Model and Average BeforeAggregation.168B.1. SeverstalB.1.5.9 Strong Learners Class 2 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.142: Severstal Class 2 Metrics for Strong Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.143: Severstal Class 2 AUC for Strong Model and Noisy OrAggregation.169B.1. SeverstalB.1.5.10 Strong Learners Class 2 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.144: Severstal Class 2 Metrics for Strong Model and Vote Aggre-gation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.145: Severstal Class 2 AUC for Strong Model and Vote Aggrega-tion.170B.1. SeverstalB.1.5.11 strong Learners Class 2 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.146: Severstal Class 2 Layer Outputs for Strong Model.171B.1. SeverstalB.1.5.12 Best Performing Threshold HistogramsFigure B.147: Class 2 F1 Score for threshold 0.33, Strong Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.148: Class 2 F1 Score for threshold 0.32, Strong Learners, AverageBefore Aggregation, and Severstal Dataset.172B.1. SeverstalFigure B.149: Class 2 F1 Score for threshold 0.99, Strong Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.150: Class 2 F1 Score for threshold 0.10, Strong Learners, VoteAggregation, and Severstal Dataset.173B.1. SeverstalB.1.5.13 Strong Learners Class 3 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.151: Severstal Class 3 Metrics for Strong Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.152: Severstal Class 3 AUC for Strong Model and Average AfterAggregation.174B.1. SeverstalB.1.5.14 Strong Learners Class 3 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.153: Severstal Class 3 Metrics for Strong Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.154: Severstal Class 3 AUC for Strong Model and Average BeforeAggregation.175B.1. SeverstalB.1.5.15 Strong Learners Class 3 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.155: Severstal Class 3 Metrics for Strong Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.156: Severstal Class 3 AUC for Strong Model and Noisy OrAggregation.176B.1. SeverstalB.1.5.16 Strong Learners Class 3 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.157: Severstal Class 3 Metrics for Strong Model and Vote Aggre-gation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.158: Severstal Class 3 AUC for Strong Model and Vote Aggrega-tion.177B.1. SeverstalB.1.5.17 strong Learners Class 3 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.159: Severstal Class 3 Layer Outputs for Strong Model.178B.1. SeverstalB.1.5.18 Best Performing Threshold HistogramsFigure B.160: Class 3 F1 Score for threshold 0.34, Strong Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.161: Class 3 F1 Score for threshold 0.34, Strong Learners, AverageBefore Aggregation, and Severstal Dataset.179B.1. SeverstalFigure B.162: Class 3 F1 Score for threshold 0.98, Strong Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.163: Class 3 F1 Score for threshold 0.00, Strong Learners, VoteAggregation, and Severstal Dataset.180B.1. SeverstalB.1.5.19 Strong Learners Class 4 - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.164: Severstal Class 4 Metrics for Strong Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.165: Severstal Class 4 AUC for Strong Model and Average AfterAggregation.181B.1. SeverstalB.1.5.20 Strong Learners Class 4 - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.166: Severstal Class 4 Metrics for Strong Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.167: Severstal Class 4 AUC for Strong Model and Average BeforeAggregation.182B.1. SeverstalB.1.5.21 Strong Learners Class 4 - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.168: Severstal Class 4 Metrics for Strong Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.169: Severstal Class 4 AUC for Strong Model and Noisy OrAggregation.183B.1. SeverstalB.1.5.22 Strong Learners Class 4 - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.170: Severstal Class 4 Metrics for Strong Model and Vote Aggre-gation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.171: Severstal Class 4 AUC for Strong Model and Vote Aggrega-tion.184B.1. SeverstalB.1.5.23 strong Learners Class 4 - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.172: Severstal Class 4 Layer Outputs for Strong Model.185B.1. SeverstalB.1.5.24 Best Performing Threshold HistogramsFigure B.173: Class 4 F1 Score for threshold 0.32, Strong Learners, AverageAfter Aggregation, and Severstal Dataset.Figure B.174: Class 4 F1 Score for threshold 0.31, Strong Learners, AverageBefore Aggregation, and Severstal Dataset.186B.1. SeverstalFigure B.175: Class 4 F1 Score for threshold 0.98, Strong Learners, NoisyOr Aggregation, and Severstal Dataset.Figure B.176: Class 4 F1 Score for threshold 0.00, Strong Learners, VoteAggregation, and Severstal Dataset.187B.2. KiTS19B.2 KiTS19B.2.1 SummaryTable B.8: Kidney ResultsBest F1 ScoreFalse-Positive AUC False-Discovery AUCµ σ f˜1Baseline 0.94 0.16 0.98 0.97 0.95WeakLearners Average After 0.86 0.24 0.95 1.00 0.95Average Before 0.86 0.25 0.95 1.00 0.95Noisy Or 0.86 0.24 0.95 1.00 0.93Vote 0.85 0.25 0.95 0.94 0.92BoostedLearners Average After 0.87 0.24 0.96 1.00 0.96Average Before 0.87 0.24 0.96 1.00 0.96Noisy Or 0.87 0.23 0.95 1.00 0.93Vote 0.86 0.24 0.96 0.95 0.93StrongLearners Average After 0.90 0.19 0.97 1.00 0.98Average Before 0.90 0.20 0.97 0.99 0.97Noisy Or 0.90 0.20 0.97 1.00 0.95Vote 0.90 0.20 0.97 0.96 0.95Table B.9: Tumor ResultsBest F1 ScoreFalse-Positive AUC False-Discovery AUCµ σ f˜1Baseline 0.62 0.39 0.85 0.83 0.74WeakLearners Average After 0.28 0.31 0.15 0.97 0.48Average Before 0.29 0.26 0.22 0.97 0.49Noisy Or 0.29 0.27 0.22 0.98 0.47Vote 0.20 0.34 0.00 0.61 0.46BoostedLearners Average After 0.31 0.31 0.22 0.97 0.48Average Before 0.31 0.32 0.20 0.97 0.49Noisy Or 0.31 0.31 0.22 0.98 0.47Vote 0.23 0.33 0.00 0.63 0.38StrongLearners Average After 0.43 0.37 0.42 0.98 0.63Average Before 0.42 0.35 0.41 0.98 0.63Noisy Or 0.43 0.35 0.42 0.99 0.62Vote 0.40 0.37 0.40 0.73 0.53188B.2. KiTS19B.2.2 BaselineB.2.2.1 Baseline Class Kidney(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.177: KiTS19 Class Kidney Metrics for Baseline Model .(a) False-Positive AUC (b) False-Discovery AUCFigure B.178: KiTS19 Class Kidney AUC for Baseline Model .189B.2. KiTS19(a) Convolution (b) Normalization(c) ActivationFigure B.179: KiTS19 Class Kidney Layer Outputs for Baseline Model.B.2.2.2 Best Performing Threshold HistogramFigure B.180: Kidney F1 Score for threshold 0.5 with Baseline Model andKiTS19 Dataset.190B.2. KiTS19B.2.2.3 Baseline Class Tumor(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.181: KiTS19 Class Tumor Metrics for Baseline Model .(a) False-Positive AUC (b) False-Discovery AUCFigure B.182: KiTS19 Class Tumor AUC for Baseline Model .191B.2. KiTS19(a) Convolution (b) Normalization(c) ActivationFigure B.183: KiTS19 Class Tumor Layer Outputs for Baseline Model.B.2.2.4 Best Performing Threshold HistogramFigure B.184: Tumor F1 Score for threshold 0.5 with Baseline Model andKiTS19 Dataset.192B.2. KiTS19B.2.3 Weak LearnersB.2.3.1 Weak Learners Kidney- Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.185: KiTS19 Class Kidney Metrics for Weak Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.186: KiTS19 Class Kidney AUC for Weak Model and AverageAfter Aggregation.193B.2. KiTS19B.2.3.2 Weak Learners Kidney - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.187: KiTS19 Class Kidney Metrics for Weak Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.188: KiTS19 Class Kidney AUC for Weak Model and AverageBefore Aggregation.194B.2. KiTS19B.2.3.3 Weak Learners Kidney - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.189: KiTS19 Class Kidney Metrics for Weak Model and NoisyOr Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.190: KiTS19 Class Kidney AUC for Weak Model and Noisy OrAggregation.195B.2. KiTS19B.2.3.4 Weak Learners Kidney - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.191: KiTS19 Class Kidney Metrics for Weak Model and VoteAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.192: KiTS19 Class Kidney AUC for Weak Model and Vote Ag-gregation.196B.2. KiTS19B.2.3.5 Weak Learners Kidney - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.193: KiTS19 Class Kidney Layer Outputs for Weak Model.197B.2. KiTS19B.2.3.6 Best Performing Threshold HistogramFigure B.194: Kidney F1 Score for threshold 0.32, Boosted Learners, Av-erage After Aggregation, and KiTS19 Dataset.Figure B.195: Kidney F1 Score for threshold 0.31, Boosted Learners, Av-erage Before Aggregation, and KiTS19 Dataset.198B.2. KiTS19Figure B.196: Kidney F1 Score for threshold 0.98, Boosted Learners, NoisyOr Aggregation, and KiTS19 Dataset.Figure B.197: Kidney F1 Score for threshold 0.10, Boosted Learners, VoteAggregation, and KiTS19 Dataset.199B.2. KiTS19B.2.3.7 Weak Learners Tumor- Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.198: KiTS19 Class Tumor Metrics for Weak Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.199: KiTS19 Class Tumor AUC for Weak Model and AverageAfter Aggregation.200B.2. KiTS19B.2.3.8 Weak Learners Tumor - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.200: KiTS19 Class Tumor Metrics for Weak Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.201: KiTS19 Class Tumor AUC for Weak Model and AverageBefore Aggregation.201B.2. KiTS19B.2.3.9 Weak Learners Tumor - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.202: KiTS19 Class Tumor Metrics for Weak Model and Noisy OrAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.203: KiTS19 Class Tumor AUC for Weak Model and Noisy OrAggregation.202B.2. KiTS19B.2.3.10 Weak Learners Tumor - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.204: KiTS19 Class Tumor Metrics for Weak Model and VoteAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.205: KiTS19 Class Tumor AUC for Weak Model and Vote Ag-gregation.203B.2. KiTS19B.2.3.11 Weak Learners Tumor - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.206: KiTS19 Class Tumor Layer Outputs for Weak Model.204B.2. KiTS19B.2.3.12 Best Performing Threshold HistogramFigure B.207: Tumor F1 Score for threshold 0.20, Boosted Learners, Aver-age After Aggregation, and KiTS19 Dataset.Figure B.208: Tumor F1 Score for threshold 0.15, Boosted Learners, Aver-age Before Aggregation, and KiTS19 Dataset.205B.2. KiTS19Figure B.209: Tumor F1 Score for threshold 0.85, Boosted Learners, NoisyOr Aggregation, and KiTS19 Dataset.Figure B.210: Tumor F1 Score for threshold 0.00, Boosted Learners, VoteAggregation, and KiTS19 Dataset.206B.2. KiTS19B.2.4 Boosted LearnersB.2.4.1 Boosted Learners Kidney - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.211: KiTS19 Class Kidney Metrics for Boosted Model and Aver-age After Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.212: KiTS19 Class Kidney AUC for Boosted Model and AverageAfter Aggregation.207B.2. KiTS19B.2.4.2 Boosted Learners Kidney - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.213: KiTS19 Class Kidney Metrics for Boosted Model and Aver-age Before Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.214: KiTS19 Class Kidney AUC for Boosted Model and AverageBefore Aggregation.208B.2. KiTS19B.2.4.3 Boosted Learners Kidney - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.215: KiTS19 Class Kidney Metrics for Boosted Model and NoisyOr Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.216: KiTS19 Class Kidney AUC for Boosted Model and NoisyOr Aggregation.209B.2. KiTS19B.2.4.4 Boosted Learners Kidney - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.217: KiTS19 Class Kidney Metrics for Boosted Model and VoteAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.218: KiTS19 Class Kidney AUC for Boosted Model and VoteAggregation.210B.2. KiTS19B.2.4.5 Boosted Learners Kidney - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.219: KiTS19 Class Kidney Layer Outputs for Boosted Model.Figure B.220: KiTS19 Class Kidney Boosting Improvement.211B.2. KiTS19B.2.4.6 Best Performing Threshold HistogramFigure B.221: Kidney F1 Score for threshold 0.33, Boosted Learners, Av-erage After Aggregation, and KiTS19 Dataset.Figure B.222: Kidney F1 Score for threshold 0.31, Boosted Learners, Av-erage Before Aggregation, and KiTS19 Dataset.212B.2. KiTS19Figure B.223: Kidney F1 Score for threshold 0.98, Boosted Learners, NoisyOr Aggregation, and KiTS19 Dataset.Figure B.224: Kidney F1 Score for threshold 0.10, Boosted Learners, VoteAggregation, and KiTS19 Dataset.213B.2. KiTS19B.2.4.7 Boosted Learners Tumor - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.225: KiTS19 Class Tumor Metrics for Boosted Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.226: KiTS19 Class Tumor AUC for Boosted Model and AverageAfter Aggregation.214B.2. KiTS19B.2.4.8 Boosted Learners Tumor - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.227: KiTS19 Class Tumor Metrics for Boosted Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.228: KiTS19 Class Tumor AUC for Boosted Model and AverageBefore Aggregation.215B.2. KiTS19B.2.4.9 Boosted Learners Tumor - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.229: KiTS19 Class Tumor Metrics for Boosted Model and NoisyOr Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.230: KiTS19 Class Tumor AUC for Boosted Model and Noisy OrAggregation.216B.2. KiTS19B.2.4.10 Boosted Learners Tumor - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.231: KiTS19 Class Tumor Metrics for Boosted Model and VoteAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.232: KiTS19 Class Tumor AUC for Boosted Model and VoteAggregation.217B.2. KiTS19B.2.4.11 Boosted Learners Tumor - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.233: KiTS19 Class Tumor Layer Outputs for Boosted Model.Figure B.234: KiTS19 Class Tumor Boosting Improvement.218B.2. KiTS19B.2.4.12 Best Performing Threshold HistogramFigure B.235: Tumor F1 Score for threshold 0.20, Boosted Learners, Aver-age After Aggregation, and KiTS19 Dataset.Figure B.236: Tumor F1 Score for threshold 0.20, Boosted Learners, Aver-age Before Aggregation, and KiTS19 Dataset.219B.2. KiTS19Figure B.237: Tumor F1 Score for threshold 0.90, Boosted Learners, NoisyOr Aggregation, and KiTS19 Dataset.Figure B.238: Tumor F1 Score for threshold 0.00, Boosted Learners, VoteAggregation, and KiTS19 Dataset.220B.2. KiTS19B.2.5 Strong LearnersB.2.5.1 Strong Learners Kidney - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.239: KiTS19 Class Kidney Metrics for Strong Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.240: KiTS19 Class Kidney AUC for Strong Model and AverageAfter Aggregation.221B.2. KiTS19B.2.5.2 Strong Learners Kidney - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.241: KiTS19 Class Kidney Metrics for Strong Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.242: KiTS19 Class Kidney AUC for Strong Model and AverageBefore Aggregation.222B.2. KiTS19B.2.5.3 Strong Learners Kidney - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.243: KiTS19 Class Kidney Metrics for Strong Model and NoisyOr Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.244: KiTS19 Class Kidney AUC for Strong Model and Noisy OrAggregation.223B.2. KiTS19B.2.5.4 Strong Learners Kidney - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.245: KiTS19 Class Kidney Metrics for Strong Model and VoteAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.246: KiTS19 Class Kidney AUC for Strong Model and VoteAggregation.224B.2. KiTS19B.2.5.5 Strong Learners Kidney - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.247: KiTS19 Class Kidney Layer Outputs for Strong Model.225B.2. KiTS19B.2.5.6 Best Performing Threshold HistogramFigure B.248: Kidney F1 Score for threshold 0.33, Strong Learners, AverageAfter Aggregation, and KiTS19 Dataset.Figure B.249: Kidney F1 Score for threshold 0.32, Strong Learners, AverageBefore Aggregation, and KiTS19 Dataset.226B.2. KiTS19Figure B.250: Kidney F1 Score for threshold 0.99, Strong Learners, NoisyOr Aggregation, and KiTS19 Dataset.Figure B.251: Kidney F1 Score for threshold 0.10, Strong Learners, VoteAggregation, and KiTS19 Dataset.227B.2. KiTS19B.2.5.7 Strong Learners Tumor - Average After Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.252: KiTS19 Class Tumor Metrics for Strong Model and AverageAfter Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.253: KiTS19 Class Tumor AUC for Strong Model and AverageAfter Aggregation.228B.2. KiTS19B.2.5.8 Strong Learners Tumor - Average Before Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.254: KiTS19 Class Tumor Metrics for Strong Model and AverageBefore Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.255: KiTS19 Class Tumor AUC for Strong Model and AverageBefore Aggregation.229B.2. KiTS19B.2.5.9 Strong Learners Tumor - Noisy Or Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.256: KiTS19 Class Tumor Metrics for Strong Model and NoisyOr Aggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.257: KiTS19 Class Tumor AUC for Strong Model and Noisy OrAggregation.230B.2. KiTS19B.2.5.10 Strong Learners Tumor - Vote Aggregator(a) Metric Mean vs. Threshold (b) F1 Score Boxplot vs. ThresholdFigure B.258: KiTS19 Class Tumor Metrics for Strong Model and VoteAggregation.(a) False-Positive AUC (b) False-Discovery AUCFigure B.259: KiTS19 Class Tumor AUC for Strong Model and VoteAggregation.231B.2. KiTS19B.2.5.11 Strong Learners Tumor - Aggregator Independent(a) Convolution (b) Normalization(c) ActivationFigure B.260: KiTS19 Class Tumor Layer Outputs for Strong Model.232B.2. KiTS19B.2.5.12 Best Performing Threshold HistogramFigure B.261: Tumor F1 Score for threshold 0.25, Strong Learners, AverageAfter Aggregation, and KiTS19 Dataset.Figure B.262: Tumor F1 Score for threshold 0.20, Strong Learners, AverageBefore Aggregation, and KiTS19 Dataset.233B.2. KiTS19Figure B.263: Tumor F1 Score for threshold 0.93, Strong Learners, NoisyOr Aggregation, and KiTS19 Dataset.Figure B.264: Tumor F1 Score for threshold 0.00, Strong Learners, VoteAggregation, and KiTS19 Dataset.234B.3. Selected PredictionsB.3 Selected Predictions(a) Baseline (b) Weak(c) Boosted (d) StrongFigure B.265: KiTS19 Kidney - Case 00000, Slice 28.235B.3. Selected Predictions(a) Baseline (b) Weak(c) Boosted (d) StrongFigure B.266: KiTS19 Tumor - Case 00012, Slice 40.236B.3. Selected Predictions(a) Baseline (b) Weak(c) Boosted (d) StrongFigure B.267: KiTS19 Kidney - Case 00003, Slice 62.237B.3. Selected Predictions(a) Baseline (b) Weak(c) Boosted (d) StrongFigure B.268: KiTS19 Tumor - Case 00003, Slice 62.238B.3. Selected Predictions(a) Baseline(b) Weak(c) Boosted(d) StrongFigure B.269: Severstal Class 1 - 3fdc95c42.239B.3. Selected Predictions(a) Baseline(b) Weak(c) Boosted(d) StrongFigure B.270: Severstal Class 2 - 17ac2163c.240B.3. Selected Predictions(a) Baseline(b) Weak(c) Boosted(d) StrongFigure B.271: Severstal Class 3 - 0ebb34d89.241B.3. Selected Predictions(a) Baseline(b) Weak(c) Boosted(d) StrongFigure B.272: Severstal Class 4 - 5c8104408.242
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-ShareAlike 4.0 International
Graduate
Efficient 2D image segmentation
Text