Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Borden, Bailey
Master of Science - MSc
University of British Columbia
Purpose: Artificial intelligence (AI) autocontouring tools are becoming readily available, and can also be created using published deep neural network (DNN) frameworks. With the labour demand required to peer review cases, contouring peer review may not occur before the beginning of treatment. This work investigates the use of AI for autocontouring to aid within the breast radiotherapy contouring peer review process. The use of AI within contour peer review may aid with labour demands and allow patient cases to be reviewed earlier in treatment. Methods: Using open-source deep learning libraries, DNNs were trained to segment the heart, the left lung, and the right lung. The model architectures explored in this work are U-Net and LinkNet. The trained architectures were compared to the commercial algorithm Limbus AI. The trained U-Net model architecture was selected to demonstrate uncertainty quantification using conformal prediction, while Limbus AI was used to show an approach to model uncertainty using test time augmentation (TTA). Results: For multiclass segmentation explored, U-net and Linknet achieved a mean intersection over union (IoU) of 0.924 and 0.910 respectively. When the trained algorithms were compared to the commercial algorithm, U-Net and LinkNet achieved mean IoUs of 0.914 and 0.911, while Limbus AI achieved a mean IoU of 0.883. Using the conformal prediction methodology on U-Net, uncertain pixels within the model's predictions were found to be on the boundaries of structures. When testing the uncertainty of the Limbus AI using TTA, a standard error of 4.95% was observed on the heart predictions, 3.54% on the left lung predictions, and 3.35% on the right lung predictions. Conclusion: Based on the literature, evaluating contours using both area and distance based metrics is recommended for peer review quality assurance. Testing of a selected model using a sample of data representing the potential patient demographic is recommended. Additionally, testing the uncertainty of a selected model using unseen data is recommended, to provide model insight. The appropriate use of these AI algorithms could aid in the contouring peer review process.
A Machine Learning Approach toComputed Tomography OrganSegmentation for Quality AssurancebyBailey BordenB.Sc. Physics (Honours), The University of Northern British Columbia, 2022A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Medical Physics)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)November 2024© Bailey Borden 2024The following individuals certify that they have read, and recommend to the Col-lege of Graduate Studies for acceptance, the thesis entitled:A Machine Learning Approach to Computed Tomography Organ Segmentationfor Quality Assurancesubmitted by Bailey Borden in partial fulfillment of the requirements for the de-gree of Master of Science in Medical PhysicsExamining Committee:Dr. Rasika Rajapakshe, Irving K Barber Faculty of ScienceSupervisorDr. Rebecca Feldman, Irving K Barber Faculty of ScienceCo-supervisorDr. Patricia Lasserre, Irving K Barber Faculty of ScienceSupervisory Committee MemberDr. Sarah Lucas, Department of SurgerySupervisory Committee MemberDr. Jonathan Holzman, School of EngineeringUniversity ExamineriiAbstractPurpose: Artificial intelligence (AI) autocontouring tools are becoming readilyavailable, and can also be created using published deep neural network (DNN)frameworks. With the labour demand required to peer review cases, contouringpeer review may not occur before the beginning of treatment. This work investi-gates the use of AI for autocontouring to aid within the breast radiotherapy con-touring peer review process. The use of AI within contour peer review may aidwith labour demands and allow patient cases to be reviewed earlier in treatment.Methods: Using open-source deep learning libraries, DNNs were trained to seg-ment the heart, the left lung, and the right lung. The model architectures exploredin this work are U-Net and LinkNet. The trained architectures were comparedto the commercial algorithm Limbus AI. The trained U-Net model architecturewas selected to demonstrate uncertainty quantification using conformal predic-tion, while Limbus AI was used to show an approach to model uncertainty usingtest time augmentation (TTA).Results: For multiclass segmentation explored, U-net and Linknet achieved amean intersection over union (IoU) of 0.924 and 0.910 respectively. When thetrained algorithms were compared to the commercial algorithm, U-Net and LinkNetachieved mean IoUs of 0.914 and 0.911, while Limbus AI achieved a mean IoUof 0.883. Using the conformal prediction methodology on U-Net, uncertain pixelswithin the model’s predictions were found to be on the boundaries of structures.When testing the uncertainty of the Limbus AI using TTA, a standard error of4.95% was observed on the heart predictions, 3.54% on the left lung predictions,iiAbstractand 3.35% on the right lung predictions.Conclusion: Based on the literature, evaluating contours using both area and dis-tance based metrics is recommended for peer review quality assurance. Testing ofa selected model using a sample of data representing the potential patient demo-graphic is recommended. Additionally, testing the uncertainty of a selected modelusing unseen data is recommended, to provide model insight. The appropriate useof these AI algorithms could aid in the contouring peer review process.iiiLay SummaryBreast cancer is common. The labour required to peer review organ delineationfor breast cancer treatment is too much to allow each patient’s case to be reviewedprior to the beginning of treatment. The use of a trained and tested AI model toaid in the review of cases will reduce delay. Deep learning models were trainedwithin this work. Comparisons were made between trained models and a commer-cially available system. The training and testing of these systems allowed for adiscussion of implementing AI systems in practice to perform aid in radiotherapypeer review. Considerations for the implementation of AI include fairness, eq-uity, appropriate testing, and understanding of AI models. Implementation of thesuggestions made within this work should be done by a multidisciplinary team.ivPrefaceThe experiments performed in Chapters 2 through 5 were covered by the REBapproval H18-01183.The conformal prediction and test-time augmentation experiments in Chapter 5were performed in collaboration with Brandon Graham-Knight.The use of generative artificial intelligence within this work was limited to trou-bleshooting LATEXcode, and aiding with corrections to grammar.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 Radiotherapy Workflow . . . . . . . . . . . . . . . . . . 31.1.2 Emerging AI Applications in Healthcare . . . . . . . . . 51.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . 61.3 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131.4.1 Project Description . . . . . . . . . . . . . . . . . . . . . 13viTable of Contents2 Methods: Segmentation Tasks . . . . . . . . . . . . . . . . . . . . . 152.1 Deep Learning Pipeline . . . . . . . . . . . . . . . . . . . . . . 152.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.2.1 Overview of available data . . . . . . . . . . . . . . . . . 162.2.2 Data Availability: Segmentation Experiments . . . . . . . 202.3 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.4 Methods: Binary Segmentation . . . . . . . . . . . . . . . . . . 222.4.1 Training Metrics and Loss Functions . . . . . . . . . . . 222.4.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . 242.4.3 Structures for Binary Tasks . . . . . . . . . . . . . . . . 252.5 Methods: Multiclass Segmentation . . . . . . . . . . . . . . . . 252.5.1 Training Metrics and Loss Functions . . . . . . . . . . . 262.5.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . 282.6 Methods: Comparison of Trained DNNs to Commercial Algo-rithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Results: Segmentation Tasks . . . . . . . . . . . . . . . . . . . . . . 303.1 Results: Binary Segmentation . . . . . . . . . . . . . . . . . . . 303.1.1 Results: Left Lung . . . . . . . . . . . . . . . . . . . . . 303.1.2 Results: Heart . . . . . . . . . . . . . . . . . . . . . . . 313.2 Results: Multiclass Segmentation . . . . . . . . . . . . . . . . . 353.3 Results: Comparison of Semantic Segmentation to a CommercialAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 Discussion: Segmentation Tasks . . . . . . . . . . . . . . . . . . . . 404.1 Binary Segmentation Tasks . . . . . . . . . . . . . . . . . . . . 404.2 Multiclass Segmentation . . . . . . . . . . . . . . . . . . . . . . 414.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 Suggestions for the use of AI to Aid in Breast Cancer Contour PeerReview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46viiTable of Contents5.1 Selecting Empirical Methods for Determining Closeness of Pre-dictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465.2 Variability in Contouring . . . . . . . . . . . . . . . . . . . . . . 495.3 Quantifying Error in AI: Using Conformal Prediction for Uncer-tainty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.4 Application of Conformal Prediction for Segmentation Uncertainty515.4.1 Methods: Conformal Prediction . . . . . . . . . . . . . . 535.4.2 Results: Conformal Prediction . . . . . . . . . . . . . . . 545.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 565.4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 565.5 Experiment: Quantifying Error in a Black Box Commercial Al-gorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.5.1 Materials and Methods . . . . . . . . . . . . . . . . . . . 575.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 615.6 Discussion: Suggestions for the use of AI in Peer Review . . . . 625.7 Summary of Recommendations . . . . . . . . . . . . . . . . . . 636 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68AppendicesA Additional Data Annotation Trends . . . . . . . . . . . . . . . . . . 85B Additional Results From Segmentation Experiments . . . . . . . . 87B.1 Binary Segmentation . . . . . . . . . . . . . . . . . . . . . . . . 87viiiTable of ContentsB.1.1 Training Parameters . . . . . . . . . . . . . . . . . . . . 87B.1.2 IoU Training and Validation Curves . . . . . . . . . . . . 88B.1.3 Sample Segmentations . . . . . . . . . . . . . . . . . . . 89B.2 Multi class Segmentation . . . . . . . . . . . . . . . . . . . . . . 98B.2.1 Training Parameters . . . . . . . . . . . . . . . . . . . . 98B.2.2 IoU Training and Validation Curves . . . . . . . . . . . . 98B.2.3 Sample Segmentations . . . . . . . . . . . . . . . . . . . 98C Hardware and Timing Considerations . . . . . . . . . . . . . . . . 101C.1 Timing Comparisons . . . . . . . . . . . . . . . . . . . . . . . . 101C.2 Methods: Performance Comparison Between Machines . . . . . 101C.2.1 Results: Performance Comparison Between Machines . . 102C.3 Methods: Comparison of CNN Speeds on the Same Computer . . 102C.3.1 Results: Comparison of Network Speeds on the SameComputer . . . . . . . . . . . . . . . . . . . . . . . . . 102C.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103D Additional Data for Test Time Augmentation Experiment . . . . . 104ixList of Tables2.1 Number of occurrences within CT studies of all six lung annotations. 203.1 Results of all binary segmentation experiments. . . . . . . . . . . 313.2 Time required to train models. . . . . . . . . . . . . . . . . . . . 313.3 Mean Performance of U-Net and LinkNet over the full test Set. . 363.4 Mean performance of segmentation algorithms on 138 test CTs. . 385.1 Calculated IoU values for the total prediction, the conformal pre-diction with 90% threshold, and the percentage difference be-tween the two values. . . . . . . . . . . . . . . . . . . . . . . . . 555.2 Mean performance of Limbus AI with poor quality data excluded. 60B.1 Training parameters for the binary segmentation of the left lung. . 87B.2 Training parameters for the binary segmentation of the heart. . . . 87B.3 Training parameters for the multiclass segmentation experiment. . 98C.1 Time required for the 2D CNNs to predict over 6 full CTs on bothcomputers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102C.2 Time required for all four CNNs to predict over 6 full CTs oncomputer 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103D.1 Mean structure IoUs for the for the first 35 groups of augmentedCT slices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106D.2 Mean structure IoUs for the for the second 35 groups of aug-mented CT slices. . . . . . . . . . . . . . . . . . . . . . . . . . 107xList of Figures1.1 An illustration of the basic radiation therapy treatment workflow.The radiotherapy treatment planning workflow begins with CTsimulation. This CT simulation is a scan of the patient in theposition they will be treated. The CT scan produced will providea basis for treatment planning allowing contouring and dose pre-scriptions to be applied to the treatment plan. The treatment planwill be peer reviewed by radiation oncologists and then will pro-ceed to treatment. . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 U-Net architecture (adapted from [51]). An input image is passedthrough the network where it will pass through the various en-coder blocks (A). Within these encoder blocks features will beextracted from the images. At each layer of the encoder a skipconnection (B) passes information from the encoder to the de-coder (C). This allows for the prediction to be constructed withoutthe loss of spatial information, as upsampling is performed alongthe decoder path. . . . . . . . . . . . . . . . . . . . . . . . . . . 10xiList of Figures1.3 The ResNet34 architecture (adapted from [56]). An image is in-put into the ResNet34 architecture, convolution (A) and poolingoperations are applied to extract information from the image. Theinformation is then passed through a series of residual blocks (B)consisting of 2 convolutions and a skip connection (C) from thestart of the block to the end. Once the information has passedthrough all residual blocks pooling is performed, followed by afully connected layer (D) corresponding to all possible outputclasses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.4 LinkNet architecture (adapted from [52]). The encoder blocks(A) consist of residual blocks (B) for feature extraction. The in-formation passes through the encoder and is also passed from theencoder to the decoder using skip connections (C). The passingof information directly from the encoder blocks to the decoderblocks (D) allows for predictions to be constructed without theloss of spatial information in the decoder. . . . . . . . . . . . . . 142.1 The deep learning pipeline used in this thesis began with obtain-ing CT data. The CT data for each experiment was then split intotrain, validation, and test sets. The model was then trained usingthe training set of data. At the end of each epoch, the model’sprogress was tracked using the validation set. The trained modelwas then evaluated using the test data. The final step after testingthe deep learning model was to then visually inspect segmenta-tions produced by the models. . . . . . . . . . . . . . . . . . . . 162.2 Cohort age distribution. . . . . . . . . . . . . . . . . . . . . . . . 172.3 Changes in annotation trends over the time span of the dataset forthe ’LT Lung’, ’RT Lung’, and ’Heart’ contours within the samecase (a), and any of the three axilla lymph nodes within a patientcase (b). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19xiiList of Figures2.4 Data filtering for the left lung (a), heart (b), and three structuresegmentation experiments (c). All trials begin with filtering outthe non-breast cancer cases from the data. The next step was todetermine the available cases to be used for training in each ex-periment. From there, a sufficient amount of cases are extractedfrom each data set. . . . . . . . . . . . . . . . . . . . . . . . . . 212.5 Intersection over union of sets A and B. In the context of the prob-lem let A be the reference standard contour, and let B represent thepredicted mask from a DNN. . . . . . . . . . . . . . . . . . . . . 233.1 Training and validation losses for 2D U-Net (a), 2D LinkNet (b),3D U-Net (c), and 3D LinkNet (d) for the left lung trials. Overfit-ting was minimized in all cases, which is evident because of thesimilarities between the training and validation loss curves. . . . . 323.2 Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the prediction made for thisslice by U-Net (c). The difference between the prediction andreference standard contour is around the boundary of the structure(d). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.3 Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the prediction made for thisslice by 3D U-Net (c). The algorithm has predicted the mask be-ing both lungs, rather than only segmenting the left lung. . . . . . 343.4 Training and validation losses for 2D U-Net (a), 2D LinkNet (b),3D U-Net (c), and 3D LinkNet (d) for the heart trials. Overfit-ting was minimized in all cases, which is evident because of thesimilarities between the training and validation loss curves. . . . . 34xiiiList of Figures3.5 Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the prediction made for thisslice by U-Net (c). The difference between the prediction andreference standard contour is around the boundary of the structure(d). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.6 Training and validation losses for 2D U-Net (a), and 2D LinkNet(b) for the multiclass trials. Overfitting was minimized in allcases, which is evident because of the similarities between thetraining and validation loss curves. . . . . . . . . . . . . . . . . . 363.7 Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the multiclass prediction madefor this slice by 2D U-Net (c). The differences between the pre-diction and reference standard contour are on the boundaries ofthe structures (d). . . . . . . . . . . . . . . . . . . . . . . . . . . 373.8 Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the multiclass prediction madefor this slice by 2D U-Net (c). 2D U-Net is predicting a heartcontour where none is present. . . . . . . . . . . . . . . . . . . . 383.9 Visualization of original CT slice(a), reference standard contours(b), commercial AI prediction (c), and 2D U-Net prediction (d).The difference between the prediction from Limbus AI and thereference standard contours is visible along the boundaries of thestructures (e). The differences between the 2D U-Net predictionand the reference standard contours is also along the boundariesof the structures (f). . . . . . . . . . . . . . . . . . . . . . . . . 395.1 Visual representation of minimum distances between each pointon set A (green) and set B (black) with the HD highlighted in red. 48xivList of Figures5.2 Visual example of conformal prediction using the APS method, asdescribed in the text (adapted from [98]). First the softmax scoresare sorted from most to least probable. Followed by determiningthe cumulative softmax score of the true class (orange) (a). Thisprocess is repeated for the entire calibration set to create a distri-bution of the nonconformity scores. From there the quantile ofthese scores qˆ is determined (b). Using the cumulative softmaxscore corresponding to qˆ conformal prediction sets can be made(c). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.3 Visualization of reference standard contours (a) and conformalprediction sets (b) overlayed on the original CT image. Solid col-ors represent predictions within the 95% coverage requirement,and orange pixels represent uncertain pixels. This sample showsuncertain regions within this slice for all three structures. . . . . . 555.4 Visualization of reference standard contours (a) and conformalprediction sets (b) overlayed on the original CT image. Solid col-ors represent predictions within the 95% coverage requirement,and orange pixels represent uncertain pixels. This sample high-lights the uncertain region of the heart prediction, as the heart isnot shown in the reference standard contours. . . . . . . . . . . . 555.5 Four types of augmentations were used within the TTA experi-ment. Grid dropout was applied to a CT slice. Limbus AI recog-nized the lungs within this image (a). Blurring was applied to aCT slice which resulted in the segmenting of both lungs and theheart (b). The application of an elastic deformation on an imageresulted in segmentation of all three desired structures by LimbusAI (c). The insertion of noise into an images resulted in LimbusAI segmenting all three desired structures (d). . . . . . . . . . . . 59xvList of Figures5.6 Example of segmented lung appearing in an image when noisewas inserted (a), and no longer appearing when an elastic defor-mation was applied to the same image (b). . . . . . . . . . . . . . 605.7 Sample TTA segmentation for the left lung (top), right lung (mid-dle), and the heart (bottom). The images show a comparison ofthe input CT slice (left), reference standard contour (center), andthe prediction from the commercial AI (right). . . . . . . . . . . . 61A.1 Appearance of the seroma annotation within dataset. Prior to2006, no seroma cases were present in the dataset, and after thetotal number of seroma cases was low compared to the total breastcancer radiotherapy treatments. The data available for 2019 only16 breast cancer cases contained the seroma annotation. . . . . . . 85A.2 Appearance of the left lung and right lung annotations within thesame case. Within each year of the available data, cases contain-ing both the left lung and right lung annotations were far fewerthan the total number of breast cancer treatments. . . . . . . . . . 86B.1 Training and validation IoU curves for 2D U-Net (a), 2D LinkNet(b), 3D U-Net (c), and 3D LinkNet (d) for the left lung trials. . . . 88B.2 Training and validation IoU for 2D U-Net (a), 2D LinkNet (b), 3DU-Net (c), and 3D LinkNet (d) for the binary heart segmentationtrials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89B.3 2D U-Net sample segmentations of the left Lung. Each row rep-resents a CT slice in which 2D U-Net was used to predict theleft lung. The comparison shows CT slices (left), reference stan-dard annotations (middle), and 2D U-Net predictions (right). Thewhite arrows in the reference standard annotations and the 2D U-Net predictions point to a difference between the two images. . . . 90xviList of FiguresB.4 2D LinkNet sample segmentations of the left Lung. Each row rep-resents a CT slice in which 2D LinkNet was used to predict theleft lung. The comparison shows CT slices (left), reference stan-dard annotations (middle), and 2D LinkNet predictions (right).The white arrows in the reference standard annotations and the2D LinkNet predictions point to a difference between the two im-ages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91B.5 3D U-Net sample segmentations of the left Lung. Each row rep-resents a CT slice in which 3D U-Net was used to predict theleft lung. The comparison shows CT slices (left), reference stan-dard annotations (middle), and 3D U-Net predictions (right). Thewhite arrows in the reference standard annotations and the 3D U-Net predictions point to a difference between the two images. . . . 92B.6 3D LinkNet sample segmentations of the left Lung. Each row rep-resents a CT slice in which 3D LinkNet was used to predict the leftlung. The comparison shows CT slices (left), reference standardannotations (middle), and 3D LinkNet predictions (right).The whitearrows in the reference standard annotations and the 3D LinkNetpredictions point to a difference between the two images. . . . . . 93B.7 2D U-Net sample segmentations of the heart. Each row representsa CT slice in which 2D U-Net was used to predict the heart. Thecomparison shows CT slices (left), reference standard annotations(middle), and 2D U-Net predictions (right). The white arrows inthe reference standard annotations and the 2D U-Net predictionspoint to a difference between the two images. . . . . . . . . . . . 94xviiList of FiguresB.8 2D LinkNet sample segmentations of the heart. Each row rep-resents a CT slice in which 2D LinkNet was used to predict theheart. The comparison shows CT slices (left), reference standardannotations (middle), and 2D LinkNet predictions (right). Thewhite arrows in the reference standard annotations and the 2DLinkNet predictions point to a difference between the two images. 95B.9 3D U-Net sample segmentations of the heart. Each row representsa CT slice in which 3D U-Net was used to predict the heart. Thecomparison shows CT slices (left), reference standard annotations(middle), and 3D U-Net predictions (right). The white arrows inthe reference standard annotations and the 3D U-Net predictionspoint to a difference between the two images. . . . . . . . . . . . 96B.10 3D LinkNet sample segmentations of the heart. Each row rep-resents a CT slice in which 3D LinkNet was used to predict theheart. The comparison shows CT slices (left), reference standardannotations (middle), and 3D LinkNet predictions (right). Thewhite arrows in the reference standard annotations and the 3DLinkNet predictions point to a difference between the two images. 97B.11 Training and validation IoU curves for 2D U-Net (a), and 2DLinkNet (b) for the multiclass trials. . . . . . . . . . . . . . . . . 98B.12 2D U-Net sample multiclass segmentations. Each row representsa CT slice in which 2D U-Net was used to predict the three struc-tures. The comparison shows CT slices (left), reference standardannotations (middle), and 2D U-Net predictions (right). The whitearrows in the reference standard annotations and the 2D U-Netpredictions point to a difference between the two images. . . . . . 99xviiiList of FiguresB.13 2D LinkNet sample multiclass segmentations. Each row repre-sents a CT slice in which 2D LinkNet was used to predict thethree structures. The comparison shows CT slices (left), refer-ence standard annotations (middle), and 2D LinkNet predictions(right). The white arrows in the reference standard annotationsand the 2D LinkNet predictions point to a difference between thetwo images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100D.1 Augmented image in which augmentation resulted in both lungsnot being recognized by Limbus AI. . . . . . . . . . . . . . . . . 104D.2 Box plot analyzing the changes in the performance of the com-mercial AI when segmenting the left lung for the first 30 sets ofaugmented CT slices. Slices 1, 4, and 6 are examples of slicesexcluded from the TTA experiment. These slices have near zeromedian values, and the majority of points are zero or near zero foreach respective set of 200 augmented slices. . . . . . . . . . . . 105xixList of AcronymsAI Artificial IntelligenceANN Artificial Neural NetworkAPS Adaptive Prediction SetBCE Binary Cross EntropyCCE Categorical Cross EntropyCNN Convolutional Neural NetworkCPQR Canadian Partnership for Quality RadiotherapyCT Computed TomographyDICOM Digital Imaging and Communications in MedicineDNN Deep Neural NetworkIoU Intersection Over UnionGPU Graphics Processing UnitHD hausdorff DistanceHD95 Hausdorff Distance 95%MAPIE Model Agnostic Prediction Interval EstimatorReLu Rectified Linear UnitxxList of AcronymsQA Quality AssuranceTTA Test Time AugmentationxxiAcknowledgementsI would like to thank the many people who supported me during this project. Iwould like to thank Dr. Rajapakshe for mentoring me through the project,allowing me to develop skills independently, while also providing guidance whenneeded. Additionally, I would like to thank Brandon Graham-Knight: thank youfor allowing me to bounce ideas off of you; your advice was always very helpful.Thank you to my committee members, Dr. Patricia Lasserre, Dr. Sarah Lucas,and Dr. Rebecca Feldman for your continued support and advice throughout thisproject.I would also like to thank my peers, Mitchell Weibe, Seger Nelson, and ErinMcCurry for your advice and making the program enjoyable for me.Last, but certainly not least, I would like to thank my fiance´e, Anna, forcontinuously supporting me through this venture.xxiiChapter 1Introduction1.1 BackgroundBreast cancer is a prevalent type of cancer, with one in eight Canadian women ex-pected to develop this cancer throughout their lifetime. Breast cancer is expectedto account 13% of cancer related deaths in women during 2024 [1]. Treatmentsfor breast cancer include a combination of systemic therapy, surgery, and radio-therapy [2,3]. For example, breast conserving surgery with adjuvant radiotherapyhas been shown to improve patient outcomes and lower rates of recurrence, as op-posed to surgery alone [4–6]. Radiotherapy is important and is further discussedin this thesis.Radiotherapy is the process of using ionizing radiation to kill cancer cells, butit also induces cell death in healthy cells. An effective scheduled dose deliverysystem will balance tumour cell death while allowing the healthy cells time torecover. For example, a conventional radiation delivery schedule consists of twoGray fractions, five times a week until a desired dose is reached [7–9]. However,breast cancer treatments may occur five times a week at two Gray per dose, orthey may be delivered in fractions larger than two Gray on a modified schedule.Regardless of the selected fractionation schedule the desired tumour control needsto be reached [7–9].Patient quality of life and overall treatment efficacy are important considerationsfor radiotherapy treatments [2, 3]. Tumour control is the goal of a radiotherapytreatment. However, if totally eradicating the tumour will cause severe damageto the patient, it is not a viable goal [8, 10]. Detrimental effects could include11.1. Backgroundirreversible organ damage that could seriously affect a patient’s quality of life.This leads to a balance between tumour control and the risk of adverse effects tothe patient [2, 3, 10].Healthy tissue damage can affect a patient’s quality of life. Sensitive structureswithin the radiotherapy treatment region are at risk of being damaged during treat-ment. The lungs and the heart are two structures at risk when breast radiotherapytreatment is performed [7, 11–13]. For example, the lungs are at risk for late dis-eases such as fibrosis if the dose within the region exceeds a given amount. Theheart is at risk for diseases such as ischemic heart disease [11, 14]. Radiotherapyplanning needs to be performed with organs at risk in mind, limiting healthy tissuedamage.In non-breast radiotherapy treatment sites, such as late stage lung cancer [15], amacroscopic tumour volume is the target of treatment. This differs from typicalbreast cancer specific radiotherapy, as the goal is not to eliminate a macroscopictumour, as the tumour has been removed in the form of surgery [6, 15, 16]. How-ever, microscopic tumour cells may still remain within the breast. The goal of thetreatment is then, to eliminate the microscopic spread of the tumour, which maylater result in local recurrence, or be a source of metastatic cancer [5,6]. Treatmentregions include a margin surrounding the tumour position prior to surgery, as wellas treatment to adjacent lymph nodes. Treatment of lymph nodes must be carriedout to achieve the proper biological effect on the tumour cells, as cancer cells firstspread to the closest lymph nodes [17]. The radiotherapy process can be brokeninto four stages: (1) computed tomography (CT) simulation, (2) contouring anddose prescriptions, (3) peer review of contouring and dose prescriptions, and (4)patient treatment, as illustrated by Figure 1.1.21.1. BackgroundFigure 1.1: An illustration of the basic radiation therapy treatment workflow. Theradiotherapy treatment planning workflow begins with CT simulation. This CTsimulation is a scan of the patient in the position they will be treated. The CT scanproduced will provide a basis for treatment planning allowing contouring and doseprescriptions to be applied to the treatment plan. The treatment plan will be peerreviewed by radiation oncologists and then will proceed to treatment.1.1.1 Radiotherapy WorkflowA patient specific CT simulation is the beginning of the radiotherapy treatmentprocess. During the CT simulation process, patients are to be scanned in theposition in which they will be treated [18–20]. A common approach to the CTsimulation is to take a 4D CT while the patient is asked to perform a breath hold,creating a volumetric video of the organs. The resulting scan is the basis for theremainder of the radiotherapy treatment workflow. The patient’s CT scan acts asa virtual patient for planning the patient’s treatment, providing 3D volumes forcontouring for dose calculations [18–21]. Scanning using breath hold techniquesallows the treatment plan to target specific anatomy of interest while minimizingrespiratory motion. The practice of positioning the patient in the treatment posi-tion during the scan ensures that the patient is in the desired location at each timethe beam is on [19, 20].Once the CT simulation has been completed, the dose prescriptions are performed,providing the therapeutic goals to treatment volumes and dose limits to organs atrisk. The contouring of treatment volumes and various organs at risk is then per-formed. Once the contouring is completed, the dosimetrist will generate the bestpossible plan based on the dose prescriptions. The radiotherapy treatment plan isgenerated using the inverse treatment planning methodology, which will attemptto create an optimal treatment plan based on the dose prescriptions and delineatedvolumes [20–23]. Over- and under-contouring of structures of interest could ad-31.1. Backgroundversely affect the treatment plan. Contouring errors could negatively influencethe optimization of the treatment plan, as contouring is used to provide targetsfor the optimization algorithm. Although breast cancer treatments are not typi-cally as complex as plans for other treatment regions, contouring errors could stillinfluence treatment parameters like field size, total radiation delivered, and dy-namic dose modulation, creating a less than ideal plan [7, 21, 24]. A difference incontouring accuracy could cause dose spillage outside of a target volume, or re-duce the coverage of a target. Reduction of target coverage could potentially limitthe therapeutic goal. Under- or over-contouring could create hot or cold radia-tion dose regions within the treated region and could lead to lower tumour controlprobability, and potential local recurrence [6,24]. Providing high quality contourswith precise delineation of organ boundaries and treatment volumes is crucial inradiotherapy planning for delivering treatments of high efficacy [21, 24, 25].Accurate contouring within the treatment planning process is critical not only fortumour control, but also for patient quality of life. Organs at risk are contouredto ensure they receive a minimal amount of radiation while still achieving thecurative goal of the treatment. Both contouring of treatment volumes and organs atrisk play a key role in beam optimization, and are crucial in the treatment planningprocess [21, 22]. The doses to contoured regions are calculated by the treatmentplanning system based on beam geometry and the specific dosage goals withineach contoured volume.Dose spillage is a serious concern for organs at risk and is mitigated by precisecontouring. Peer review is important as inter- and intra-contouring variability hasbeen observed in the contouring for radiotherapy treatments, and allows consis-tency between treatments [24, 26–29]. The peer review process aids in ensuringthe contouring is accurate and increases overall confidence in the plan. However,peer review is labour intensive, requiring the attention of multiple radiation oncol-ogists to review the contours and dose prescriptions [30, 31]. Given large clinicalworkloads, peer review may not occur prior to the beginning of a patient’s treat-ment. As per the guidelines provided by the Canadian Partnership for Quality41.1. BackgroundRadiotherapy (CPQR), patients will have their cases peer reviewed prior to 25%of the treatment being delivered. Nevertheless, being able to ensure cases are peerreviewed as early as possible in the treatment is ideal [25, 30, 31].Peer review of contouring and dose prescriptions ensures that a plan is of highquality. Peer review has been shown to improve patient care [28, 30]. Once theplan passes peer review the patient will receive their treatment based on the pre-scribed treatment plan. However, if the plan does not pass the peer review processit will return to an earlier stage of the radiotherapy workflow to be modified. Oncemodifications have been made, the plan will once again be peer reviewed. Peerreview is a critical check in the radiotherapy treatment process. Contouring anddose prescription peer review is a final quality assurance (QA) step to ensure thatthe patient’s treatment plan is designed for the desired treatment outcome.1.1.2 Emerging AI Applications in HealthcareArtificial intelligence (AI) is a rapidly evolving field, with a variety of applicationsacross many disciplines. In healthcare, AI-enhanced algorithms are being inves-tigated to aid in diagnoses, image segmentation, and risk analysis [32–37]. Therapidly growing applications of AI algorithms within healthcare raises concernsabout their limitations and the extent of their use within these spaces. Algorithmaccuracy and potential biases need to be given careful consideration within health-care applications [38–40]. Due to the limited understanding of AI algorithms byhealthcare professionals, and the potential deployment of AI models within clin-ical decision making processes [32, 41–44], QA processes need to be developedfor these algorithms. QA processes will be used to ensure that AI tools can beused safely in day-to-day clinical operations. In this thesis, QA is discussed in thecontext of using AI in the contouring peer review process, which is discussed indetail in Chapter 5.51.2. Problem Statement1.2 Problem StatementPatient contouring is a critical part of the treatment planning process, thus peerreview of contours is very important within the radiotherapy workflow. However,with labour demands, the actual time of occurrence of the peer review processwithin the treatment planning process could lead to the verification of contoursoccurring after treatment commences [25, 28, 30]. The use of AI algorithms toaid in the peer review process could greatly reduce the labour required to reviewthese cases. This reduction in labour will ideally allow the peer reviewing of casesearlier within the treatment timeline.Deep learning is a popular research area within AI due to its ability to solve com-plex problems by learning from large datasets. This thesis explores the use of AIto aid in the breast cancer contour peer review process, specifically capitalizingon the capabilities of deep learning [45–48]. This thesis also explores the useof commercially available algorithms for this task with comparisons to the deeplearning models explored in this thesis. Commercially available AI algorithms arebecoming increasingly prominent in clinical applications [42, 49, 50].1.3 Machine LearningDeep learning is a powerful tool with a variety of applications. Deep learning is aform of machine learning, which is a branch of AI which leverages artificial neu-ral networks (ANNs) to perform predictions for a task. The presence of multiplelayers between the input and output of the ANN structures is what coined the termdeep learning [45]. Deep learning has been successful in a variety of image anal-ysis tasks, such as biomedical image segmentation [37, 47, 51–56]. Four typicalcomputer vision tasks are: image classification, object detection, semantic seg-mentation, and instance segmentation [57, 58]. All four of these computer visiontasks are interrelated but differ in goal and task complexity. A type of deep neural61.3. Machine Learningnetwork (DNN) known as a convolutional neural network (CNN) is a commonapproach to these computer vision tasks [57, 59, 60].This thesis will focus on semantic image segmentation. Segmentation is the pro-cess of delineating regions of interest in an image. Building on segmentation,instance segmentation is the process of separately segmenting all instances ofthe same object type. Semantic segmentation is similar to instance segmenta-tion. However, semantic segmentation labels all occurrences of the same objectas part of the same mask [58]. An example of semantic segmentation would belabelling all of the occurrences of people within an image as part of the same seg-mentation mask [61]. Two types of semantic segmentation are performed in thisthesis: binary segmentation, and multiclass segmentation. Binary segmentation isthe process of separating an image into a background and a predicted label class.Multiclass segmentation is the process of delineating the background, and all ob-ject classes of interest in an image. Deep learning has been applied to a variety ofimage segmentation tasks, from more natural cases of image segmentation such assegmenting cars and people on a street to segmenting various structures or pointsof interest in medical imaging [54–56].Automated biomedical image segmentation has been a challenge to perform indeep learning research. Patch-wise CNNs have been explored for this type ofsegmentation [48]. However, these CNNs required extensive redundancy, mak-ing them impractical in many segmentation applications. Encoder-decoder net-works were one particularly useful innovation that improved the speed and per-formance of CNNs designed for image segmentation [51, 52, 62, 63]. Encoder-decoder networks leverage the use of both convolution and deconvolution oper-ations [51, 52, 62]. The deconvolution operations of these networks rely on skipconnections, which allow layers to pass information to layers of the decoder with-out passing through the subsequent encoder layers. These later layers can up-sample data to regain lost information. These network architectures allow imagesto be processed through many layers of blocks consisting of rectified linear units71.3. Machine Learning(ReLu), max pooling, and convolutional blocks all while passing spatial informa-tion from the encoder path to the decoder path through skip connections [51, 52].For the purpose of this research, the topic is explored using DNN architectures inthe form of encoder-decoder networks.Convolutional layers are neural network layers that perform, and optimize thediscrete mathematical convolution related to the features being extracted. A formof such convolution could be an edge extraction kernal. This output informationfrom the convolution operation is called a feature map [59].Max pooling is the process of taking the maximum value within a set of numbers.In deep learning this operation is used on the rectangular output of CNN layers,taking the maximum value within each rectangle before moving through the re-mainder of the data performing additional pooling [59].Activation functions are important in deep learning as they can provide nonlin-earity, and allow networks to learn features in images [64] The ReLu activationfunction applies a nonlinearity to data passing through it as described by equation1.1, where x is an output of a deep learning layer, and ReLu(x) is the output valueof the ReLu function [64].ReLu(x) =0 if x≤ 0x if x> 0 (1.1)The activation function of the output layer influences the prediction output values.For a binary problem, the sigmoid function can be used. The sigmoid activationfunction is described by equation 1.2, where x is an output of a deep learninglayer, and σ (x) is the output value of the sigmoid function in the range {0,1} [64].σ(x) =11+ e−x(1.2)The softmax activation function is an extension of the sigmoid function to multi-class problems. Equation 1.3 [64] describes the softmax function σi(xi) where anoutput xi corresponding to class i is input and divided by the sum of the exponen-tial over all classes k. This function outputs a value for each class i.81.3. Machine Learningσi(xi) =xi∑kj=0 e−x j(1.3)U-Net is a deep learning network that was designed specifically for biomedical im-age segmentation [51]. The U-Net architecture consists of an encoder and decoderpath, as illustrated in Figure 1.2 [51]. U-Net leverages upsampling to make pre-dictions using the information extracted at various network levels. As informationis passed down the encoder path of U-Net, downsampling is performed. Down-sampling using max pooling reduces the overall size of the data along the path, asit reduces the spatial dimensions. To regain the spatial information lost in down-sampling, the network performs upsampling in conjunction with skip connectionsalong the decoder path. The skip connections pass the spatial information directlyfrom the encoder path to the decoder path allowing the information extracted onthe various levels of the network to be used to make the final prediction [51]. Anapproach to extracting features in these encoder-decoder networks is to providebackbones pre-trained on natural images to provide a basis for extracting featuremaps relevant to structures [46, 65, 66].ImageNet is a large-scale publicly available dataset of natural images. Exam-ples of these images include cars, trucks, dogs, cats, and aeroplanes. These im-ages have annotations providing class labels to each image, which is sufficient fortraining models for classification tasks, such as classification of objects in naturalimages [67]. Since the backbones used in encoder-decoder networks can be pre-trained on ImageNet data, this provides a starting point for the overall architec-tures used in this thesis to extract feature maps [68]. This algorithm pre-trainingis later referred to as pre-trained weights, as it is used to set the model weightsbefore training on a given segmentation task.Many variations of U-Net have emerged [36,46,47,65], and it has become widelypopular for medical image segmentation. Therefore, U-Net is a good candidate forthe segmentation tasks explored within this research. In this thesis, the version of91.3. Machine LearningFigure 1.2: U-Net architecture (adapted from [51]). An input image is passedthrough the network where it will pass through the various encoder blocks (A).Within these encoder blocks features will be extracted from the images. At eachlayer of the encoder a skip connection (B) passes information from the encoder tothe decoder (C). This allows for the prediction to be constructed without the lossof spatial information, as upsampling is performed along the decoder path.U-Net available in the Python Segmentation Models library was used [65]. Thisnetwork allows the use of U-Net for 2D segmentation with pre-trained encoderweights. The pre-trained weights are estimated using training of the backboneson the ImageNet dataset, which provides a basis for training the encoder portionof the network [65]. This network architecture can be constructed using a largevariety of backbones, all with pre-trained ImageNet weights [65]. Additionally, alldeep learning model libraries used in this thesis allow the construction of modelswith a variety of different networks in the encoder of the segmentation network.101.3. Machine LearningAll networks used in Chapter 2 leverage ResNet34 pre-trained on ImageNet in theencoder path of the networks.Another variant of U-Net that was of interest for segmentation tasks is the 3D U-Net architecture available from the Segmentation Models 3D library [46]. This 3Dnetwork was originally constructed based on the implementation of U-Net in theSegmentation Models repository [46, 65]. This 3D network architecture behavesvery similarly to the 2D U-Net architecture provided in the Segmentation Modelspackage. However, instead of accepting 2D images, the expected input of thismodel is now 3D volumes.The ResNet architecture addresses a problem within deep learning known as thevanishing gradient. This issue arises in deep learning when attempting to optimizethe gradient descent using backpropagation. As the gradient is calculated throughbackpropagation it will eventually disappear within large networks. This vanish-ing gradient can cause network weights to potentially not be updated [56]. Thisdisruption in weight updates could cause the model to no longer train, leading to aplateau in model performance. ResNet34 uses the various skip connections shownin Figure 1.3 to address this issue. With its complex structure, ResNet has beenused successfully for image classification on the ImageNet dataset [56].ResNet34 is a 34 layer residual neural network architecture. This network consistsof a series of convolutional blocks and skip connections as shown in Figure 1.3[56]. Residual neural networks are capable of performing computer vision taskson their own, such as classification [56, 69]. However, using them in combinationwith another network structure allows these residual networks to extract featuresthat will pass through the additional network [66].The ResNet architecture is built into the encoder pathways of the selected net-works and is used to create feature maps that extract useful information fromthe images. Additionally, the use of ResNet34 allows for transfer learning based111.3. Machine LearningFigure 1.3: The ResNet34 architecture (adapted from [56]). An image is input intothe ResNet34 architecture, convolution (A) and pooling operations are applied toextract information from the image. The information is then passed through aseries of residual blocks (B) consisting of 2 convolutions and a skip connection(C) from the start of the block to the end. Once the information has passed throughall residual blocks pooling is performed, followed by a fully connected layer (D)corresponding to all possible output classes.on training on ImageNet, which can be used as a basis for feature extraction[46, 65, 66]. The purpose of the pre-trained weights in this instance is to reducethe overall training time required for the network [68].Another prominent network architecture for image segmentation is LinkNet. Sim-ilar to U-Net, LinkNet is an encoder-decoder network. The LinkNet architecturecan be seen in Figure 1.4 [52]. LinkNet was designed to perform real-time se-mantic segmentation. Therefore, the time required to perform predictions wasan important consideration in LinkNet’s development [52]. Similar to U-Net, as121.4. Project Scopeinformation passes through LinkNet’s encoder path, downsampling is performed.Upsampling is then performed along the decoder path in conjunction with the skipconnections to restore the spatial information lost in the downsampling, provid-ing a final prediction with the desired output dimensions. LinkNet has become acompetitor for biomedical image segmentation. LinkNet has been used in areassuch as biomedical segmentation, and satellite image segmentation [37, 54, 55].The version of LinkNet used in this project is available in the Segmentation Mod-els package [65]. Similarly, a 3D LinkNet model was also used in this thesis. The3D LinkNet model is available from the Segmentation Models 3D repository [46].1.4 Project Scope1.4.1 Project DescriptionThe purpose of this research was to provide a proof of concept that deep learningbased models can be developed using BC Cancer data to aid in the breast can-cer contouring peer review process; introducing recommendations for the use ofDNNs in QA procedures. The selected network architectures for this project wereU-Net and LinkNet, which are two popular image segmentation algorithms. Twovariants of each architecture were considered in this project, a set of networksdesigned for 2D image inputs and a set of networks designed for 3D volume in-puts. 2D U-Net, 2D LinkNet, and the commercial algorithm Limbus AI [70] weretested on structures relevant to breast radiotherapy treatment. Afterward, recom-mendations for implementing networks such as these into the peer review processfor breast radiotherapy were explored.131.4. Project ScopeFigure 1.4: LinkNet architecture (adapted from [52]). The encoder blocks (A)consist of residual blocks (B) for feature extraction. The information passesthrough the encoder and is also passed from the encoder to the decoder using skipconnections (C). The passing of information directly from the encoder blocks tothe decoder blocks (D) allows for predictions to be constructed without the loss ofspatial information in the decoder. 14Chapter 2Methods: Segmentation Tasks2.1 Deep Learning PipelineFigure 2.1 shows the deep learning pipeline used in this project. The processbegan at the image acquisition stage, where all anonymous images were collectedretrospectively from patients who underwent radiotherapy treatment. This patientcontouring information created the basis for the training, validation, and test setsused in this thesis by providing the image mask pairs required for the segmentationalgorithms. Sets of axial images corresponding to a single CT scan were stored infolders of the same scan ID. All scans for a patient were stored in a folder under thesame anonymous patient ID. To avoid data leakage, folders corresponding to eachanonymous ID were sorted into either the train, validation, or test sets, rather thanseparating them based on 2D planar images alone. Data was separated into setsusing Scikit-Learn’s train test split function [71]. Additionally, once these setswere defined, the axial slices were used within the 2D iteration of this experiment,while subsets of the 3D volumes were used in the 3D portion of this experiment.No augmentation was performed as there was sufficient data in the dataset to trainthe models. The next step was to use the training and validation sets to train thedeep learning model. During each epoch, the model was trained on a portion ofthe training data, at the end of each epoch the training of the deep learning modelwas monitored using the validation set which was not used to update the weightsof the model. This process repeated until it was confirmed that the model hadconverged to a solution. The performance of the model was gauged after trainingusing the test data set by comparing predictions to reference standard contoursover the whole unseen set of data. The final step was to manually inspect the152.2. Datasegmentation results to ensure that they were as expected.Figure 2.1: The deep learning pipeline used in this thesis began with obtainingCT data. The CT data for each experiment was then split into train, validation,and test sets. The model was then trained using the training set of data. At theend of each epoch, the model’s progress was tracked using the validation set. Thetrained model was then evaluated using the test data. The final step after testingthe deep learning model was to then visually inspect segmentations produced bythe models.2.2 Data2.2.1 Overview of available dataThe data used for this work was from a cohort of breast cancer patients who re-ceived their radiation treatments from 2001-2019. Given the large date range, itwas expected that file formats and clinical practices changed during the collec-tion of this data, as illustrated by the changing structure numbers discussed withinthis section. The age of patients in the cohort is summarized in Figure 2.2 . Theoverall size of the cohort was 30006 patients. However, for each of the followingexperiments, this number was filtered based on specific data availability, and howmuch overall data was required to achieve acceptable results for each segmenta-162.2. Datation task. It can be seen from Figure 2.2, that patient age was centered at 60 yearsold which was a key feature of the dataset.Figure 2.2: Cohort age distribution.In the data set, there are a large number of contoured structures available to ana-lyze. However, not all of the contouring was relevant to breast cancer treatment,as some patients may have undergone treatments for other cancers after initiallybeing diagnosed with breast cancer. Additionally, not all available structures hadsufficient contouring data to provide the image segmentation models with suffi-cient data to train them reliably. Therefore, only structures with a large amount ofdata points respective to general structure availability were used [38].Class imbalance appears in image segmentation in a variety of ways. In generalthis term refers to one class appearing more frequently in data than another. Inthe context of the segmentation task being performed this imbalance can happenin two ways. Firstly the total number of annotations of one class may be largerthan others. Another form of class imbalance may occur at the pixel level, wherethere may be considerably more pixels of one class than another present in animage [38].172.2. DataMulticlass semantic segmentation is an important focus in this research, class im-balance was a concern for training the algorithms. Therefore, to perform mul-ticlass image segmentation the desired data sets were filtered to provide equalnumbers of image-mask pairs for each class. Class imbalance is also a concernfor biomedical image segmentation. Structures vary in size, allowing more pixelsof some classes within an image than others, as well as overall images containinga specific class [38–40]. Therefore, strategies for overcoming this additional classimbalance will be discussed in section 2.5.1.Data leakage is an important aspect to consider when training deep learning mod-els. Unexpected leakage of data can create misleading results and incorrect char-acterization of a deep learning model’s performance [38–40]. In order to minimizethese detrimental outcomes, data was sorted to keep data related to the same pa-tient exclusively in a single set. The total number of CT studies available was34032. Some patients have had multiple scans and multiple treatments. Only onescan per patient was used. The exclusion of multiple CTs per patient reduced theavailable number of CT scans to 30006.Within the context of this thesis, the term annotation refers to a reference standardcontour. Annotations within the dataset are contained within the Digital Imagingand Communications in Medicine (DICOM) structure files. From these DICOMfiles, the available annotations within the dataset were analyzed. Furthermore, thechange in practices throughout the years can be observed from the informationin the DICOM structure files. Figure 2.3(a) shows the total number of scans thatcontained all three of the annotations ’LT Lung’, ’RT Lung’, and ’Heart’. It canbe seen from Figure 2.3(a) that the total number of cases that contain all threeannotations is low compared to the total number of cases scanned in a year. How-ever, it is noticeable that the relative number of cases with all three annotationshas increased by the year 2012 in the dataset. The number of occurrences of eachlung annotation label is shown in Table 2.1. The frequency of axilla lymph nodeannotations appearing in the dataset is summarized in Figure 2.3(b). Further visu-alizations of trends within the dataset are available in Appendix A.182.2. Data(a)(b)Figure 2.3: Changes in annotation trends over the time span of the dataset for the’LT Lung’, ’RT Lung’, and ’Heart’ contours within the same case (a), and any ofthe three axilla lymph nodes within a patient case (b).192.2. DataTable 2.1: Number of occurrences within CT studies of all six lung annotations.Label lt lung rt lung ipsi lung both lung lung exclude lung#Occurences 9427 9554 2756 2043 2766 11322.2.2 Data Availability: Segmentation ExperimentsPrior to training the selected deep learning models, data cleaning and curationwere necessary. Palliative treatments were excluded from this work. The exclu-sion of palliative cases brought the total number of patient cases to 29692. Furtherexclusions were made within each experiment, as shown in Figure 2.4. All DNNtraining experiments within this research use an 80:17:3 split for training, testing,and validation sets.The CT data came from six different cancer centers, with different annotationstandards. Therefore, the number of available contours for all three tasks neededto be analyzed to determine the amount of available data for each structure. Uponanalysis, six labels existed describing the lung contour, and only one label wasfound for the heart structure. To segment the left lung, the ’LT Lung’ annota-tion label was determined to be sufficient for training, as it was contained within9427 patient cases, and the left lung is a relatively large structure within the CTscans. Considering the size of the structure and the overall number of scans the9427 scans contain a sufficient number of 2D slices for training purposes. Thisdata cleaning process is summarized in Figure 2.4(a). The ’Heart’ label appearedin 13514 cases. Due to the success of the left lung experiment without using allof the extracted data, fewer cases were extracted for the heart than the left lung.The data cleaning for the heart experiment is shown in Figure 2.4(b). The mul-ticlass experiment was more limited by annotations than the left lung, and heartsegmentation experiments. The right lung, left lung, and heart were selected asstructures of interest due to the frequency of the three contours appearing together.To supplement the dataset and increase the available structures for training, filescontaining the ’ipsil lung’ annotation were sorted into either left lung or rightlung annotations based on the available information for the CT scan. A summary202.3. HardwareabcFigure 2.4: Data filtering for the left lung (a), heart (b), and three structure seg-mentation experiments (c). All trials begin with filtering out the non-breast cancercases from the data. The next step was to determine the available cases to be usedfor training in each experiment. From there, a sufficient amount of cases are ex-tracted from each data set.of the available data for this trial is given in Figure 2.4(c). In all three experi-ments, empty annotation labels existed in the data leading to the final number ofannotations being less than the originally selected amount.2.3 HardwareSeveral computers were used to conduct this research. To train the 2D segmen-tation models, a computer equipped with an Nvidia Quadro P2000 graphics cardwith a memory of five gigabytes, 1024 compute cores, and a memory bandwidthof 140 GB/s was used [72]. The training pipeline for the 2D experiments used212.4. Methods: Binary Segmentationaxial CT slices, 512 by 512 pixels in size. The choice to use these full axial sliceswas made, as the ability to segment based on the full slice is desirable in poten-tial applications. Due to the restrictions placed on the training pipeline and therelatively low amount of video random-access memory (VRAM) available on theP2000 graphics processing unit (GPU) all 2D segmentation tasks were limited tosmall batch training. Small batch training typically leads to the model convergingto a solution. However, they are very sensitive to the learning rate provided tothem in training [73]. With the limited GPU available all deep learning modelshave been trained with a batch size of four. Therefore, model weights are updatedafter predictions are made on four axial slices [73].A different computer was used for training DNNs for 3D segmentation tasks, dueto the limitations of the Quadro P2000’s available VRAM. The computer usedto run the 3D trials contained an NVIDIA RTX 3060 graphics card. The RTX3060 has 12 GB of VRAM, a boost clock of 1.78 GHz, and a base clock of 1.32GHz [74]. The RTX 3060 was capable of handling larger volumes as input for the3D machine learning tasks. For this task providing complete CT scans as inputswas simply not practical, as the size of the network very quickly grows with thesize of the input. To allow adequate GPU resources to train the 3D models 3Dsubvolumes from the CT scans were created using patchify [75]. Due to the GPUlimitations, the 3D segmentation tasks were limited to small batch training using64x64x64 pixel volumes. All 3D segmentation models were trained with a batchsize of four.2.4 Methods: Binary Segmentation2.4.1 Training Metrics and Loss FunctionsThe intersection over union (IoU) between the predicted masks and the referencestandard masks was selected as the metric to evaluate the binary segmentationtask. In the context of biomedical image segmentation the reference standard222.4. Methods: Binary Segmentationmasks are the contoured structures within the images, and the predicted masks arethe contours predicted by the deep learning models in the images. In the 2D case,these are intersecting areas, and in the 3D case, these are intersecting volumes.A visual representation of IoU is shown in Figure 2.5. The IoU is an appropriatetraining metric to consider for segmentation tasks because it represents how wellthe model is able to distinguish the objects it is being trained to segment whilepenalizing the model for including regions outside of the reference standard seg-mentation region [62,76]. The IoU can be modified to fit a loss function known asJaccard loss, which is one of the loss functions used to train the models within thiswork [77]. IoU is given by equation 2.1 and the Jaccard loss is given by equation2.2, where A is a subset of pixels in a mask and B is a subset of pixels in anothermask. [62, 77].IoU =Area/Volume of OverlapArea/Volume of Union=|A∩B||A∪B| (2.1)Jaccardloss= 1− IoU (2.2)Figure 2.5: Intersection over union of sets A and B. In the context of the problemlet A be the reference standard contour, and let B represent the predicted maskfrom a DNN.As with any machine learning model, the correct choice of a loss function is criti-cal to ensure the model weights are updated correctly during training [59]. Binarycross entropy (BCE) loss and Jaccard loss were used in this experiment. Jaccard232.4. Methods: Binary Segmentationloss was selected as it corresponds to the IoU training metric as shown in equation2.2; BCE is characterized by equation 2.3 [78, 79], where N is the total numberof pixels contained in an image, yi is a boolean corresponding to the correct classof the pixel, and p(yi) is the associated probability that the pixel belongs to thepredicted class.BCE =− 1NN∑i=1yi ∗ log(p(yi))+(1− yi)∗ log(1− p(yi)) (2.3)The overall loss function used was a simple addition of BCE and Jaccard lossfunctions. The summation performed in equation 2.4 equally weights the associ-ated loss function between the Jaccard loss and the BCE loss.Total Loss = BCE+ Jaccard Loss (2.4)2.4.2 Data PreprocessingPrior to training the models, preprocessing was performed on each 2D image slice.Initially, the data needed to be normalized. The pixel values were acquired inHounsfield units (HU) [80]. To convert to acceptable inputs for the sigmoid acti-vation function, data was scaled to values between zero and one [64]. Scaling wasdone using Scikit-Learn’s sklearn.preprocessing.MinMaxScaler [71]. In most ofthe cases discussed in this research, a model did not see each data point in thetraining data more than once during the training of the model. However, duringthe 3D heart segmentation training the networks were allowed to see each volumein the training set a maximum of eight times, to ensure that the trials did not runout of data before the networks converged.Additional data preprocessing was required for the 3D networks. To input thevolumes into the network, they needed to be partitioned into slabs which maintainappropriate spatial information about the structures, and are small enough to allo-cate resources for training. To partition the volumes the Python library, patchify242.5. Methods: Multiclass Segmentationwas used [75]. Patchify took the full size CT volumes and create subvolumeswithout changing the resolution of the spatial information. When visualizing thepredictions from 3D networks, the unpatchify function was used to recreate thefull 3D volumes, followed by the visualization of axial slices [75].2.4.3 Structures for Binary TasksInitial testing of algorithms was completed on the binary segmentation tasks. Twostructures were selected to test the efficacy of the algorithms. The left lung wasevaluated first, as there was sufficient data available within the dataset and a largedifference in contrast with the surrounding soft tissue. The selection of the leftlung allowed for a baseline to gauge the initial performance of the algorithms.The second structure evaluated during the binary segmentation tasks was the heart.The heart is a sensitive structure to radiation [12,13], and it does not have as sharpof a contrast with the surrounding soft tissue as the lung.2.5 Methods: Multiclass SegmentationOnce the models succeeded within the binary segmentation tasks discussed insection 2.4, the larger task of performing multiclass semantic segmentation onstructures within the dataset was approached using the 2D network architectures,U-Net and LinkNet. Multiclass segmentation was only performed using the 2Dnetworks, as the IoU for the 3D networks was lower than the 2D networks, andthe 3D networks require more data to train.To explore the utility of DNNs in the QA process networks capable of simulta-neous image segmentation were trained. The trained networks were comparedagainst the commercial segmentation algorithm Limbus AI [70]. An analysis ofthe overall contouring data was performed to determine which structure sets couldbe segmented using a multiclass network. After initial sorting was done to deter-mine structure sets corresponding to each case, the subsets of the data containing252.5. Methods: Multiclass Segmentationcombinations of the desired structures were determined. Multiclass segmentationis a considerably more challenging task than binary segmentation, as class imbal-ance is now a larger concern [38, 39]. Therefore, sufficient data was required toensure the models could be trained.2.5.1 Training Metrics and Loss FunctionsTo address the class imbalance discussed earlier in the context of multiclass seg-mentation individual class weights were used [38, 39]. For the purpose of thismulti-class segmentation task, the Scikit-Learn toolkit was used. Scikit-Learnprovides a function for computing class weights to be used within loss functionsfor training purposes, the sklearn.utils.class weight.compute class weight func-tion [71]. This function computes the class weights as shown in equation 2.5,where wi is a weight for a given class i, N is the total number of pixels within theprovided samples, k is the total number of classes being considered, and ni is thenumber of pixels in the sample corresponding to a given class i.wi =Nk ∗ni (2.5)The calculated class weights were then normalized to be between zero and one.The class weights were calculated to provide a larger impact on the multiclass lossfunctions for classes that were less represented in individual pixels.Similar to the binary experiments, the appropriate metrics and loss functions needto be selected to accurately train the model. As before, the IoU metric was retainedfor this task, as well as the Jaccard loss function. To account for the frequency ofpixels belonging to some classes more frequently than others appropriate weight-ing needed to be applied to the loss contributed by each class. To determine theappropriate class weights the function described by equation 2.5 was used. How-ever, to provide a more general sense of the class imbalance in the regions of theCTs being segmented, eight randomly selected CTs containing the required anno-tations were used to provide a large sample of slices to calculate the appropriate262.5. Methods: Multiclass Segmentationclass weights accounting for the changing shape of the annotations.The weighted IoU, IoUw is now defined as the weighted sum of IoUs for sets Aiand Bi, where i is a class of a given index, wi is the weight associated with class i,and k is the total number of classes being considered for segmentation. The IoUwis described in equation 2.6.IoUw =k∑i=1wi ∗ |Ai∩Bi||Ai∪Bi| (2.6)The IoUw used in the Jaccard loss formula was calculated using weights calculatedby equation 2.5, while the IoUw value using equally weighted classes, IoUmean wasmonitored during the training of the DNNs.In addition to the use of the Jaccard loss, categorical cross entropy (CCE) wasalso used to train the models. CCE is a multiclass loss function, which is anextension of the BCE loss function described in section 2.4 [81]. CCE loss isgiven by equation 2.7, where N is the number of pixels in a given image, i isthe current pixel, w j is the class weight associated with class j, yi j is a binaryvalue corresponding to the correct label for pixel i and class j, and p(xi j) is theassociated probability that the given pixel value belongs to the class j.CCE =− 1NN∑i=14∑j=1w j ∗ yi j ∗ log(p(xi j)) (2.7)In equation 2.7 a summation is performed over all four possible classes: the back-ground, heart, left lung, and right lung. The total loss function for training themulticlass deep learning models is given by equation 2.8.Loss=CCE+ Jaccardloss (2.8)CCE was calculated using the calculated class weights just like the Jaccard loss.The total loss function used was a linear combination of these two loss functionsin an attempt to increase the performance of the models being trained.272.6. Methods: Comparison of Trained DNNs to Commercial Algorithm2.5.2 Data PreprocessingImage preprocessing was required on the axial images prior to inputting data intothe multiclass networks. The image preprocessing is similar to that of the binarysegmentation already discussed. However, there are two key differences. The firstdifference is the range of the scaled input values. To input the images into the net-works the pixel values were scaled to be in the range {0,255}, which correspondedto the values expected by the pre-trained ImageNet weights [67]. Additionally, toinput the images into the networks the annotations were separated into separatepixel masks. [65]2.6 Methods: Comparison of Trained DNNs toCommercial AlgorithmIn Chapter 5, the uncertainty of the commercial AI algorithm Limbus AI [70] wasestimated to investigate the use of commercially available AI algorithms withincontour peer review QA. In order to better understand Limbus AI, as well as itsgeneral performance over the data set, an experiment was performed using 138CT scans. The CTs selected were from the test set of 376 CT scans used withinthe multiclass semantic segmentation experiment. The purpose of this experimentwas to allow a performance comparison between Limbus AI and the algorithmstrained in section 3.3. In order to provide a fair comparison, we only analyzedthe output segmentation masks for the heart, left lung, and right lung, as theseare regions that have been extensively focused on within this work. The IoU foreach structure has been independently calculated, as well as the mean IoU for themasks. Both 2D U-Net and LinkNet were compared to the Limbus segmentationalgorithm.To process the data and make predictions for Limbus AI a server set up withLimbus version 1.7.1 was used. All CTs input were processed according to theLimbus system for the ’BREAST’ structure set. To process this data all axial slices282.6. Methods: Comparison of Trained DNNs to Commercial Algorithmcorresponding to the 138 CT scans were put into a directory visible to the Limbusserver. Once all slices of a given CT were taken a file with contoured structureswas returned. The contoured structures output by Limbus AI were binary maskscorresponding to the structures of interest.29Chapter 3Results: Segmentation Tasks3.1 Results: Binary Segmentation3.1.1 Results: Left LungThe mean IoU of all four deep learning models trained to segment the left lungover the test set is summarized in Table 3.1. The time required to train models issummarized in Table 3.2. The training and validation losses for the 2D variantsof U-Net and LinkNet are shown in Figure 3.1(a) and Figure 3.1(b) respectively.For the 2D left lung trials, the training data was split between 280 epochs. Both ofthe 2D models were trained using early stopping, which monitored the validationloss tracking the number of epochs that passed without improvement. The modelsstopped after 20 epochs of no observed decrease in the validation loss, savingthe weights of the models at the best-observed point. As seen in Figure 3.1(a)2D U-Net only required 50 epochs before the training was terminated by earlystopping, and as shown in Figure 3.1(b) 2D LinkNet only required 70 epochsto train prior to being stopped. Additionally, to minimize plateauing of the lossfunction, the learning rate was decreased by a factor of 10 when 10 epochs withoutimprovement to the validation loss were observed. Similarly, for the 3D models,the models were given 160 epochs to train on the data, with the same values forearly stopping and learning rate decay as the 2D models. Both trials ended early,as illustrated by Figures 3.1(c) and 3.1(d). The calculation for the IoU valuesassociated with the 3D networks was done based on the subvolumes discussedearlier. A sample segmentation from the 2D U-Net variant is shown in Figure3.2. As shown in Figure 3.3, when segmenting the left lung the 3D networks303.1. Results: Binary Segmentationwill also segment the right lung. Training and validation IoU curves are availablein Appendix B.1.2. Additional sample binary segmentations from all networksare available in Appendix B.1.3. Comparisons of DNN speeds are available inAppendix C.Table 3.1: Results of all binary segmentation experiments.Model 2D U-Net 2D LinkNet 3D U-Net 3D LinkNetIoU Left Lung 0.915 0.917 0.856 0.846IoU Heart 0.8506 0.8457 0.7396 0.7267Table 3.2: Time required to train models.Left Lung TrainingModel 2D U-Net 2D LinkNet 3D U-Net 3D LinkNetTime to train (min) 595 742 814 1522.5Average time per epoch (min) 11.9 10.6 11.0 10.5Heart TrainingModel 2D U-Net 2D LinkNet 3D U-Net 3D LinkNetTime to train (min) 550 298 2024 2160Average time per epoch (min) 7.86 7.09 48.2 54.0Multiclass TrainingModel 2D U-Net 2D LinkNet 3D U-Net 3D LinkNetTime to train (min) 427 509Average time per epoch (min) 12.2 11.83.1.2 Results: HeartThe mean IoU for the binary segmentation tasks performed on the heart are sum-marized in Table 3.1. The training and validation loss curves for the segmentationmodels are shown in Figure 3.4. Both 2D models ended prior to the 160 availableepochs, and the 3D models finished training before the 80 available epochs. Thetime required to train all of the models varied and is shown in Table 3.2. Ad-ditionally, the similarities in the loss curves shown in Figure 3.4 suggested thatoverfitting was minimized. A sample segmentation from the 2D U-Net variant isshown in Figure 3.5. Training and validation IoU curves are available in AppendixB.1.2. Additional sample binary segmentations from all networks are available inAppendix B.1.3.313.1. Results: Binary Segmentation(a) (b)(c) (d)Figure 3.1: Training and validation losses for 2D U-Net (a), 2D LinkNet (b), 3DU-Net (c), and 3D LinkNet (d) for the left lung trials. Overfitting was minimizedin all cases, which is evident because of the similarities between the training andvalidation loss curves.323.1. Results: Binary Segmentation(a) (b)(c) (d)Figure 3.2: Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the prediction made for this slice by U-Net (c).The difference between the prediction and reference standard contour is aroundthe boundary of the structure (d).333.1. Results: Binary Segmentation(a) (b) (c)Figure 3.3: Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the prediction made for this slice by 3D U-Net (c). The algorithm has predicted the mask being both lungs, rather than onlysegmenting the left lung.(a) (b)(c) (d)Figure 3.4: Training and validation losses for 2D U-Net (a), 2D LinkNet (b), 3DU-Net (c), and 3D LinkNet (d) for the heart trials. Overfitting was minimized inall cases, which is evident because of the similarities between the training andvalidation loss curves.343.2. Results: Multiclass Segmentation(a) (b)(c) (d)Figure 3.5: Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the prediction made for this slice by U-Net (c).The difference between the prediction and reference standard contour is aroundthe boundary of the structure (d).3.2 Results: Multiclass SegmentationBoth LinkNet and U-Net were able to achieve low losses based on the trainingdata and the test data. Summaries of the training and validation losses for bothnetworks are shown in Figure 3.6. The similarities in both the training and valida-tion curves suggest that overfitting to the training data was minimized. The meanIoU scores achieved also suggest that overfitting was minimized.353.2. Results: Multiclass Segmentation(a) (b)Figure 3.6: Training and validation losses for 2D U-Net (a), and 2D LinkNet (b)for the multiclass trials. Overfitting was minimized in all cases, which is evidentbecause of the similarities between the training and validation loss curves.Once the training of both models was complete, both the IoUmean and the IoUfor each respective class were calculated. Table 3.3 summarizes the relevant IoUvalues. The IoU values for the background masks are omitted from Table 3.3 be-cause the background IoU values are less informative than the other IoU values.However, they are represented within the IoUmean values, as the background val-ues influence the IoUmean values. A sample segmentation from this experiment isshown below in Figure 3.7. Despite the relatively high mean IoU values achievedin this experiment, it is visible that the network has difficulties delineating theboundary of the heart on the superior side of the body. This is shown in Figure3.8, where the algorithm shows three structures in the prediction when there areonly two present in the reference standard annotation. The time required to trainboth networks is summarized in Table 3.2. Training and validation IoU curves areavailable in Appendix B.2.2. Additional sample multiclass segmentations fromall networks are available in Appendix B.2.3.Table 3.3: Mean Performance of U-Net and LinkNet over the full test Set.Model IoUmean Heart IoU Left Lung IoU Right Lung IoU2D U-Net 0.924 0.877 0.911 0.9142D LinkNet 0.910 0.857 0.887 0.892363.3. Results: Comparison of Semantic Segmentation to a Commercial Algorithm(a) (b)(c) (d)Figure 3.7: Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the multiclass prediction made for this sliceby 2D U-Net (c). The differences between the prediction and reference standardcontour are on the boundaries of the structures (d).3.3 Results: Comparison of SemanticSegmentation to a Commercial AlgorithmThe Limbus AI algorithm was compared with the 2D U-Net and LinkNet multi-class segmentation algorithms by analyzing 138 full chest CTs from the test set ofsection 3.2. The results of the segmentation algorithms over this subset of the data373.3. Results: Comparison of Semantic Segmentation to a Commercial Algorithm(a) (b) (c)Figure 3.8: Original CT slice (a) and reference standard contour (b) from thepatient treatment plan compared to the multiclass prediction made for this sliceby 2D U-Net (c). 2D U-Net is predicting a heart contour where none is present.are summarized in Table 3.4. The table summarizes the IoU for each structure ofinterest excluding the background, as well as the IoUmean of the image. A sampleof a CT slice segmented by both Limbus AI and 2D U-Net is shown in Figure3.9.Table 3.4: Mean performance of segmentation algorithms on 138 test CTs.Model IoUmean Heart IoU Left Lung IoU Right Lung IoUCommercial 0.883 0.815 0.860 0.8632D U-Net 0.914 0.871 0.894 0.8952D LinkNet 0.911 0.866 0.898 0.888383.3. Results: Comparison of Semantic Segmentation to a Commercial Algorithm(a) (b)(c) (d) t(e) (f)Figure 3.9: Visualization of original CT slice(a), reference standard contours (b),commercial AI prediction (c), and 2D U-Net prediction (d). The difference be-tween the prediction from Limbus AI and the reference standard contours is vis-ible along the boundaries of the structures (e). The differences between the 2DU-Net prediction and the reference standard contours is also along the boundariesof the structures (f). 39Chapter 4Discussion: Segmentation Tasks4.1 Binary Segmentation TasksBased on the IoU scores the 2D algorithms outperformed the 3D algorithms duringtesting. The 2D segmentation models were able to achieve mean IoUs of over 0.9on the left lung, and over 0.8 on the heart. 2D LinkNet achieved the highest IoUduring the left lung task by a marginal amount, and 3D LinkNet achieved thelowest observed IoU of 0.7267 on the heart segmentation task. Visualizations,as shown in Figures 3.2 and 3.3, demonstrate the 2D models performing betterthan the 3D models. Although the number of structures considered for the binaryexperiments was limited, they demonstrate the capability of DNNs to performsuccessful CT image segmentation.Even though the 2D algorithms outperformed the 3D algorithms, it may be diffi-cult to directly compare the 2D and 3D methods. There was a difference in avail-able training data due to the 2D networks taking axial slices, and the 3D networkstaking 64 by 64 by 64 pixel volumes. Additionally, the training process for allalgorithms was dependent on small batch training. Small batch training is sensi-tive to learning rate [73]. The combination of small batch sizes and differences ininput data make the comparison difficult. Through training with additional GPU,the 3D networks may see an increase in performance. An increase in availableVRAM will allow larger volumes and possibly larger batch sizes to be used totrain the algorithms. However, based on the results of the binary experiments the2D networks outperform the 3D networks.Difficulty in segmenting structures with the 2D and 3D methods could be due to a404.2. Multiclass Segmentationvariety of reasons, such as inter- and intra-contouring variability of the structures.The contouring variability may be exaggerated in the 3D case due to the additionalspatial complexity of the problem [26, 27]. Additionally, the variance in structuresize across all images leads to regions in which the structure is small or largecompared to its mean axial size. All of this contributes to a level of variancethat the models will be trained on. This may have impacted the margins anddelineation of the structure on the boundaries.Focusing on the 3D segmentation of the left lung, the 3D variants of U-Net andLinkNet achieved IoU values of 0.8564 and 0.8462 respectively. However, the 3Dnetworks struggle to differentiate between the left lung and the right lung. Figure3.3 shows a case where U-Net segments both lungs, instead of just segmenting theleft lung. To resolve this issue, an attempt was made to create larger pixel volumesfor training the algorithms. However, due to GPU limitations training was notpossible with the larger volumes. The limited 64 by 64 by 64 pixel partitions usedto train the 3D networks were a major limitation in comparing the 2D and 3Dnetworks. The models then did not learn attributes pertaining specifically to theleft lung but rather attributes pertaining to the lung as a whole, causing the modelto not be able to properly distinguish between the two lungs as separate structures.However, due to the success of the 2D networks for the binary segmentation tasksthis did not affect the larger goal of training a multiclass network.4.2 Multiclass SegmentationData availability was more limited for the multiclass experiment than the binaryexperiment. Cases where all three desired structure sets exist were limited. 2DU-Net and LinkNet were both able to achieve similar IoUmean values. However,2D U-Net performed better on the test set with an improvement in the IoUmean of1.5%. Individual IoUs were calculated in this experiment to allow the analysis ofhow the models were performing on each individual structure. Based on IoU, 2DU-Net performed better on the heart, left lung, and right lung with IoUs of 0.877,414.2. Multiclass Segmentation0.911, and 0.914 respectively. However, differences in the performance observedbetween the two networks were minimal with the largest observed difference be-ing 2.7% on the left lung. Therefore, both networks performed comparably basedon their performance on the test set, with 2D U-Net performing slightly betterthan 2D LinkNet. All three structures segmented produce IoUs of over 0.85, withthe lungs achieving higher IoU values than the heart. The networks had difficultysegmenting near the boundary of structures, which is illustrated in Figure 3.8.Difficulty with the delineation of the boundaries of structures is not uncommon insegmentation algorithms [82, 83], and should be considered when selecting algo-rithms for the purpose of QA. The results of these experiments are comparable topublished segmentation results [84–87]. With the successful training of both 2DLinkNet and U-Net for the task of simultaneously segmenting the left lung, rightlung, and the heart the models were ready for the next experiments.When 2D U-Net, 2D LinkNet and Limbus AI were compared in Table 3.4, Lim-bus AI performed worse than the other models across all three of the structurecategories. The models trained in-house include features in the data such as con-touring variability, which may be different from the data used to train the Limbusmodel. The in-house models performing better within this data set does not implythat they generalize better to other data sets than the Limbus AI. This assumptionsimply cannot be made as the data tested on the models in this work has similarcharacteristics to the training data for the models. The similarities in the trainingand testing data sets were due to the data belonging to the same cohort prior tosplitting data into the respective sets. The generalizability of the in-house modelsshould be tested on other datasets prior to deploying them [40, 43].2D U-Net, 2D LinkNet, and Limbus AI achieved IoU values of over 0.8, makingthem reasonable candidates for exploring their utility in peer review QA. It shouldbe noted that the Limbus AI algorithm was created to aid in the contouring portionof the treatment planning workflow, and was not designed for the QA task that isbeing suggested in this work. The Limbus AI algorithm is discussed to illustratehow to evaluate it for use in QA in place of an in-house segmentation algorithm.424.3. LimitationsAdditionally, if an autocontouring system was used in originally creating the con-tours for a plan, it should not be used for QA, as asking a program to review itsown work will not yield additional information.Although the comparison of the trained networks to the Limbus AI algorithm maybe a small part of this work, it is a very important experiment within the collectionof work presented. Testing of the algorithm verifies the performance of Limbus AIprior to the discussion of error in AI systems allowing a meaningful insight intoerror within the model. The comparison of segmentation algorithms validates thatLimbus AI performs well on the patient cohort, which is important to ensure fairand equitable practice among patients [40,43]. Therefore, a sufficient sample sizeto represent the potential patient population should be used for testing candidatealgorithms for potential use in QA processes [40].The breast cancer cases used to train the segmentation models were collected fromBC Cancer centres in Prince George, Kelowna, Vancouver, Victoria, Abbotsford,and Surrey. These regions vary significantly in distributions of age and ethnicity,these centers provided a diverse data sample potentially increasing the general-izability of the model. However, external validation from a separate data cohortshould be considered prior to deploying models trained in this manner to avoidinternal model bias [38, 39, 43]. Verification of the models on an external datasetcould be an important aspect in validating that the performance metrics of eachmodel are comparable among different cohorts [38, 39, 43].4.3 LimitationsThe axilla lymph nodes were not considered for segmentation in this thesis eventhough they are important structures in breast cancer treatment. Early within thedataset, the axilla lymph nodes do not appear at all. However, by 2012 the sets oflymph nodes begin to be considered. Therefore, a dataset that also includes morerecent patient data will likely contain more of these annotations. The combination434.3. Limitationsof limited annotations and the small size of the structures limited the availabledata to train DNNs to segment the axilla lymph nodes. The side in which lymphnodes were contoured within each patient’s treatment plan was context-sensitive,meaning that only lymph nodes on the side of the body corresponding to the pa-tient’s treatment were annotated. To properly use the available information furtherdata curation would be required to separate the annotations to either left breastaxilla lymph nodes or right breast axilla lymph nodes. The separation of singleannotations into additional annotations will further decrease the available data totrain the models to perform this segmentation task. U-Net is known to performworse when segmenting small structures with ambiguous boundaries [88]. Withadditional data, further data cleaning, and an additional approach to focus on thesmall features within the data, the exploration of axilla lymph node segmentationshould be considered in future studies. Therefore, in order to segment these struc-tures exploration of additional variants of U-Net and LinkNet is a topic for futureresearch, such as the cascade U-Net used by Feng et al [89].The seroma was not considered for segmentation in this experiment. The seromais a fluid filled sack, which is present in the location of the removed tumour.However, this structure is not always present in breast cancer treatment cases [90].Through visual inspection of the data, a pattern was searched for within the CTimages corresponding to the seroma. The annotations used in this project werein the axial plane of the images. However, visual inspection of the axial slicesfailed to consistently detect the seroma. The seroma annotations were observedin the sagittal plane in 15 CT scans. A recognizable pattern was observed in the15 CT scans. However, it is unknown if sagittal contouring is standard practicefor delineating the seroma. Therefore, the seroma has not been considered in thiswork.Only segmentation of the left lung and the heart were considered during the bi-nary segmentation experiments. Considering just the two structures limited theanalysis of available structures for segmentation. However, due to the limited an-444.3. Limitationsnotations available for other structures, the tasks that could be considered for thissegmentation experiment were limited.In addition to the lungs and heart, many annotations existed within the dataset.However, not all of these annotations were complete and consistently labeled.Manual classification of labels, while possible, as described for the axilla lymphnodes, would unlikely result in sufficient training data. Furthermore, many of thedata points for additional structures may not be so easily separated into distinctclasses and would require additional data cleaning and curation. For example,the chest wall was partially annotated in some of the CT studies being consideredin this work. The chest wall annotation was on the side of the body undergo-ing radiotherapy treatment, with the limits of the annotation varying from caseto case. Given the limitations discussed some of the structures of interest werenot able to be explored. However, the results of binary experiments establishedthe 2D variants of U-Net and LinkNet as candidates for the additional multiclasssegmentation experiments.One of the limitations of this dataset was that no ethnicity data was available forthe cohort, so the potential network bias towards different ethnic groups could notbe gauged or mitigated via random stratified sampling for the purposes of theseexperiments. Additional background information was also not available, such asphysical fitness. It is possible that various background information could impactthe performance of the model. Therefore, further analysis should be done onthese models using a cohort that has more underlying information to establish anypossible biases within these models. This is a potential topic to explore, as richerdata sets become available.Near the edges of the distribution shown in Figure 2.2 there are few patient cases.The few cases available in these age ranges is a limitation of the data set. There-fore considerations need to be made prior to deployment of a model built onthis dataset to limit any potential bias in the less represented patient popula-tions [38, 40].45Chapter 5Suggestions for the use of AI to Aidin Breast Cancer Contour PeerReviewThe earlier chapters of this work show the use of AI for chest CT image seg-mentation, focusing on the segmentation of the left lung, the right lung, and theheart. The focus of the previous experiments was to train AI models to performmulticlass semantic segmentation and compare their performance with a commer-cially available AI algorithm that segments a variety of structures. All of this workwas performed to use the obtained information to make meaningful suggestionson how AI models can be used for contouring peer review QA. Throughout thischapter recommendations are made based on available literature, previous exper-iments, and the experiments explored within this chapter, to develop suggestionsfor the use of AI based segmentation in QA processes.5.1 Selecting Empirical Methods for DeterminingCloseness of PredictionsMetric selection is an important consideration for biomedical image segmentationcomparison. The selected metrics must allow the users to meaningfully interpretthe results of the segmentation. IoU and Dice score are commonly used area-based metrics for segmentation in practice [62,76,91]. However, as highlighted byReinke et al., selecting metrics based on common practice for image segmentation465.1. Selecting Empirical Methods for Determining Closeness of Predictionsis not sufficient. Therefore, the application and limitations of the specific metricneed to be carefully considered, prior to the selection of the metric [76, 91]. Forexample, a metric may be susceptible to noise and indicate poor results in thepresence of noise. Additionally, a mean reported metric may yield misleadingresults, as individual information about the classes being segmented is absent [76].To address the issue of ambiguous metrics, metrics that are a combination ofarea and distance-based metrics have been suggested within the literature [92].These metrics are sensitive to both changes in segmentation area and distance be-tween segmentation boundaries. One such metric is the symmetric boundary dicescore. [92]. Since input from professionals is an important consideration withinradiotherapy QA, combination metrics may not be desirable due to their complex-ity.The presence of inaccurate contouring could create a less-than-optimal plan, caus-ing differences in the generated dose distribution [21, 24]. If only one metricis considered in peer review QA, it may not represent the accuracy of the seg-mentations completely. This ambiguity may lead to a scenario where a given AIsegmentation either agrees or disagrees in cases in which it should do the oppo-site. Falsely accepting or rejecting a segmentation should be mitigated in practiceTherefore, it is advised to report a combination of individual area-based metricsper structure, and distance-based metrics [76, 93].The Hausdorff distance (HD) is a distance based metric as shown in equation 5.1,where DA is the set of minimum distances from the boundary of contour A tocontour B, and DB is the set of minimum distances from the boundary of contourB to contour A [62].HD= max{max(DA),max(DB)} (5.1)Figure 5.1 visually shows minimum distances from points on contours A and B.The Hausdorff distance 95% (HD95) is an additional distance based metric. TheHD95 is very similar to the HD. The main difference is that the HD95 takes the475.1. Selecting Empirical Methods for Determining Closeness of Predictionsmaximum distance of the 95th percentiles of sets DA and DB, instead of the max-imum value contained in sets DA and DB [76].Figure 5.1: Visual representation of minimum distances between each point onset A (green) and set B (black) with the HD highlighted in red.For comparing contour similarities, it is proposed to use the IoU for each contourindividually, with set thresholds for each organ at risk, along with the Hausdorffdistance 95% as metrics for the task. The Hausdorff distance 95% (HD95) wasselected over the standard Hausdorff distance (HD), as the HD is susceptible tonoise. Therefore, the addition of noise may lead to artificial metric inflation anda large HD when contours are very close making it a less reliable distance metricfor biomedical image segmentation [76, 91]. The use of both the IoU and HD95should provide a general sense of how close the predicted mask is to the desiredcontours outlined for the radiotherapy plan. Additionally, in practice, the com-parison of the reference standard contours and the predicted contours should bevisualized to allow the user the ability to manually inspect the cases alongside anautomated comparison process.The discussion of metrics introduced some of the intricacies of using AI in thecontour peer review process. To be able to properly compare metrics in clinical485.2. Variability in Contouringpractice, threshold values need to be introduced to allow the passing or failing ofcontours based on area of overlap and distance to agreement between the metrics.An acceptable threshold needs to be set for each organ considered in the segmen-tation. Contouring threshold values should be determined by trained professionalsand may need to be adjusted on a case-by-case basis. To implement such thresh-olds, a multidisciplinary team with extensive knowledge in both the medical andtechnical domains should be consulted. Additionally, the need for user input invalidating the predictions requires the availability of visualizations for users ana-lyzing the results. This will ensure ambiguity is limited [43].5.2 Variability in ContouringContouring variability is a concern in clinical practice. Consequently, studieshave been conducted looking into both inter- and intra- contouring variability [26,27, 94]. To motivate the various needs various threshold limits required by anautocontouring tool for QA purposes inter- and intra-contouring variability arebriefly discussed here.Inter-observer contouring variability has been observed in treatment planning.Perna et al. explore this area, specifically focusing on CT contouring for prostateradiotherapy treatment [26]. Their work shows large variances in the overall vol-ume of the contoured structure resulting in a standard deviation of 6.8Gy in thedose recieved to the penile bulb. The observed variance in contouring performedby trained physicians demonstrates how this difference in contoured volume af-fects the dose to the organs. Although the results from this study are not statisti-cally significant, they do provide a basis for the presence of inter-observer contourvariability [26]. A similar study highlighting the significance of inter-observercontouring variability was performed by Das et al [27]. This study focuses moreon target volume contouring variability in lung cancer treatments and observeddifferences in contouring of up to 100% [27]. However, it is important to note thatin both of these examples, some of the volumes being contoured are quite small,495.3. Quantifying Error in AI: Using Conformal Prediction for Uncertaintywhich increases the difficulty of the task. Although these studies were conductedusing small patient cohorts and had a limited number of physicians performingthe contouring, they highlight the potential for significant differences in volumesdelineated between observers.The study from Das et al. also investigated intra-observer variability by allowingphysicians to contour the same patient on different days [27]. This analysis wasalso performed on lung lesions. The physicians performed this contouring on acohort of seven patients. A key highlight of their findings is the large differencesin intra-observer variability with the mean differences in volume reaching up to120%. Although the small cohort size does not yield statistically significant re-sults, it shows the potential impact of intra-observer variability in small structures.The potential inter- and intra-observer contouring variability within the data setwould be captured by the trained deep learning models, as this variation would bepresent within the training data. However, if an external network is being used forQA purposes, the acceptance tests for such a network should handle the presenceof contouring variability. Therefore, informed decision-making should be used inthe design process of protocols relying on AI autocontouring. Additionally, mar-gins on these structures should be placed based on best clinical practices, and withthe consultation of radiation oncologists prior to using AI for QA to ensure clinic-specific practices are being followed. The complexities of contouring variabilityprompt the need for informed decision-making when designing thresholds for AIcontouring peer review QA.5.3 Quantifying Error in AI: Using ConformalPrediction for UncertaintyConformal prediction is a rigorous statistical methodology that can be used to de-termine the error associated with predictions from machine learning models [95].Conformal prediction was originally proposed by Vovk et al., as an approach505.4. Application of Conformal Prediction for Segmentation Uncertaintyto uncertainty, discussing prediction sets constructed using confidence intervals[95, 96]. The process of performing conformal predictions requires setting asidea subset of data not seen by the model during training to establish the baselinefor determining the uncertainty in the predictions. This sequestered data can befrom the test set created during the formation of the training, validation, and testsets. The data selected to perform conformal predictions is grouped into calibra-tion and test sets. The calibration set will be used to establish the measurementfor uncertainty, and the test set is later used to represent uncertainty within the setbased on the results of the calibration set. There are four steps to using conformalprediction: (1) establish a form of heuristic uncertainty, (2) establish a noncon-formity measure, (3) use the calibration data to relate the nonconformity scoresto the desired level of uncertainty, and finally (4) use the first three steps to createconformal prediction sets [97, 98].5.4 Application of Conformal Prediction forSegmentation UncertaintyThe 2D multiclass U-Net architecture discussed earlier in this work was chosento demonstrate conformal prediction in a medical context. U-Net was selected,as it is a popular choice for biomedical image segmentation [47, 48, 99–101]. Asonly the 2D variant of U-net is used within this chapter it will simply be referredto as U-Net. Since image segmentation is closely related to image classification,the method we chose for conformal prediction treats each pixel as an individualclassifier [97, 102]. To perform the conformal predictions, the Model AgnosticPrediction Interval Estimator (MAPIE) API can be used [98]. MAPIE is an open-source Python library designed to perform conformal predictions using four dif-ferent methods: least ambiguous set-valued classifier, top-K, adaptive predictionsets (APS), and regularized adaptive prediction sets.Conformal prediction using the APS [103] method does not result in empty pre-515.4. Application of Conformal Prediction for Segmentation Uncertaintydiction sets, making it more suited for the work here than other methods [97,103].Conformal prediction using APS follows the four steps described earlier. TheAPS method begins with the selection of a heuristic representation of uncertainty,which for segmentation can be the softmax pixel outputs of a model. Next, anonconformity score is established using the cumulative softmax outputs corre-sponding to the pixel class present in the reference standard contour. For eachpixel x j corresponding to element j in the calibration set A there is a set of soft-max values Yj associated with them. To calculate the nonconformity score thesevalues are sorted from greatest to smallest to form a set Z j, where the element z jlis the softmax pixel value of index l in the sorted set Z j. With the sorted set ofsoftmax values defined, the set Y ′j of cumulative softmax scores corresponding topixel j can be defined. The elements y′jm of the set Y ′j are defined by equation 5.2,where m is the index corresponding to a given class in Y ′j .y′jm =m∑l=0z jl (5.2)With the cumulative softmax score defined the nonconformity score is now givenby equation 5.3, where the index k corresponds to the index of the cumulativesoftmax score of the correct class.s(x j,y j) = y′jk (5.3)A histogram can be generated using all of the nonconformity scores s(x j,y j) fromthe calibration set. The desired coverage of predictions D can then be calculated,as shown by equation 5.4, where α ∈ {0,1} is defined by a user, and n is the totalnumber of pixels in the calibration set.D=⌈(n+1)(1−α)⌉n(5.4)From the distribution of nonconformity scores the quantile qˆ can be determinedwhere the fraction of counts behind qˆ meet the user selected coverage require-ments. After qˆ has been calculated, conformal prediction setsC can be made from525.4. Application of Conformal Prediction for Segmentation Uncertaintythe test data using equation 5.5, where yˆ′test jm is the label corresponding to thecumulative softmax value at index m in the sorted set Y ′test j [97, 98, 103].C(X) = {yˆ′test j1, ..., yˆ′test jk}, where k = sup{m :m∑l=0y′test jl < qˆ}+1 (5.5)The APS method is shown visually in Figure 5.2.(a) (b) (c)Figure 5.2: Visual example of conformal prediction using the APS method, asdescribed in the text (adapted from [98]). First the softmax scores are sorted frommost to least probable. Followed by determining the cumulative softmax score ofthe true class (orange) (a). This process is repeated for the entire calibration set tocreate a distribution of the nonconformity scores. From there the quantile of thesescores qˆ is determined (b). Using the cumulative softmax score corresponding toqˆ conformal prediction sets can be made (c).5.4.1 Methods: Conformal PredictionConformal prediction was performed using the MAPIE python library on predic-tions made on 138 CTs from the test set of the multiclass segmentation experi-ment. Storing the softmax outputs of all 138 CTs allows conformal predictionto be performed multiple times without the need to rerun the deep learning al-gorithms. The predictions were separated into either the calibration or test set,reserving 30% for calibration and 70% for testing. The prediction sets were thenused to make conformal predictions using the APS method. These conformal pre-dictions allowed a visualization of the uncertainty within the model for a given α .535.4. Application of Conformal Prediction for Segmentation UncertaintyUsing the data in the saved predictions, conformal prediction sets were mappedback to the corresponding images, and visually compared to the reference stan-dard annotation for those images.After the softmax values were split into calibration and test values, they wereused to obtain prediction sets for various αi values. For this analysis, 95% and90% coverage values were selected to demonstrate the output of the conformalpredictions. The APS technique allowed visualizations of uncertain regions fora user to observe the behavior of the algorithm. These visualizations providetransparency if the conformal prediction methodology is being used within QAprocedures [43]. The IoU has been consistently used to gauge model performancein this work. Therefore, it was also used in the error quantification process. Toanalyze the error in this section the mean IoU was calculated for each structure,along with the IoU of the 90% coverage conformal predictions. The percentagedifference between these two values was then calculated to represent the uncer-tainty. The requirement of 90% coverage of original predictions has been selectedarbitrarily to demonstrate this procedure. However, in practice, the coverage levelshould be selected based on the context and sensitivity of the structures.5.4.2 Results: Conformal PredictionThe mean IoU values for the full predictions and the conformal prediction setsare summarized in Table 5.1. The difference in IoU between the original predic-tions and the conformal predictions is represented in Table 5.1 as a percentagedifference. The percentage difference is larger for a less certain structure, with thelargest difference being for the heart segmentations. Figures 5.3 and 5.4 demon-strate visualizations of conformal prediction sets for 95% coverage on sampleimages, highlighting the uncertain pixels in orange. Visually inspecting the con-formal predictions with the addition of uncertain pixels from the original predic-tions is important to understanding the patterns present in the uncertain regions.Uncertain pixels are typically seen on the boundaries of the predictions.545.4. Application of Conformal Prediction for Segmentation UncertaintyTable 5.1: Calculated IoU values for the total prediction, the conformal predictionwith 90% threshold, and the percentage difference between the two values.Structure Full Prediction IoU 90% coverage IoU %DifferenceHeart 0.874 0.861 1.46Left Lung 0.894 0.885 1.01Right Lung 0.896 0.893 0.894(a) (b)Figure 5.3: Visualization of reference standard contours (a) and conformal predic-tion sets (b) overlayed on the original CT image. Solid colors represent predictionswithin the 95% coverage requirement, and orange pixels represent uncertain pix-els. This sample shows uncertain regions within this slice for all three structures.(a) (b)Figure 5.4: Visualization of reference standard contours (a) and conformal pre-diction sets (b) overlayed on the original CT image. Solid colors represent predic-tions within the 95% coverage requirement, and orange pixels represent uncertainpixels. This sample highlights the uncertain region of the heart prediction, as theheart is not shown in the reference standard contours.555.4. Application of Conformal Prediction for Segmentation Uncertainty5.4.3 DiscussionBased on the visualizations of the conformal prediction sets on the CT scans, itis apparent that most of the uncertain pixels are on the edges of the structures, asshown in Figures 5.3 and 5.4. However, inspecting Figure 5.4(b) demonstratesthat the model may make predictions in locations where reference standard anno-tations do not exist. The presence of false positives is expected when the modelmakes predictions. By decreasing α , the false positives within the image beginto be excluded. Therefore, although not performed in this work implementingconformal predictions with appropriate thresholding could mitigate the error ofpredicting a structure in an image where it does not exist.As seen in Table 5.1, the heart shows the largest percentage difference between thefull prediction and the conformal prediction. This difference is visible in Figure5.3, where the boundaries of the heart are less certain than the boundaries of thelungs. This displays a general trend, that the models are less certain around theboundaries of the heart than other structures. Overall the model struggled with theboundaries of structures and tended to undercontour all regions, as indicated bythe decline in IoU across all three categories between the conformal IoU and fullprediction IoU.5.4.4 ConclusionU-Net reported a high IoUmean value in Chapter 3. However, the results of thisexperiment highlight regions that the model struggles to segment, and thus theuse of uncertainty analysis in this manner allows a further understanding of themodel being used. Quantifying the error in the model and visually inspecting theuncertain pixels, as done here, highlights not only how confident the model is,but also shows how in certain regions of the images the model tends to be lessconfident. In this experiment, the uncertain pixels were typically found on theboundaries of structures. The analysis based on conformal predictions allowed forthe visualization of uncertain pixels, making the overall model trends visible, and565.5. Experiment: Quantifying Error in a Black Box Commercial Algorithmhelping to better understand the predictions. It is recommended that a subset ofinternal data be stored for the purpose of creating conformal predictions, as priorto using an AI model in practice, one should have an understanding of the modeloutput [43]. Additionally, if the network operates as a black box only providingmasks, the approach to conformal prediction described in this experiment is notfeasible.5.5 Experiment: Quantifying Error in a Black BoxCommercial AlgorithmSince commercial AI algorithms are becoming increasingly common in clinicalapplications [42, 49, 50, 70], potential methods to quantify the error associatedwith commercial segmentation algorithms have been explored. The available tech-niques to quantify error were very restrictive because the configuration of LimbusAI operated as a complete black box, with images as the input, and binary masksas the output. One method for quantifying error within the strict requirements ofthe commercial model was test time augmentation (TTA). [100, 104].5.5.1 Materials and MethodsTo perform an error analysis using TTA a subset of the clinical data availablefrom the previous experiments was used. Due to the labour-intensive nature ofthis method, only 172 CT scans from the test set of the multiclass experimentdiscussed in Chapter 2 were selected for this trial. The process of quantifyingthe error for this model involved using a single axial slice from each of the 172CT scans. The individual slices were input into the commercial algorithm 200times with various augmentations. The changes in the AI contouring were used toquantify the error in the model. Four types of augmentations have been selected:(1) grid dropout, (2) blurring, (3) elastic deformations, and (4) noise insertion.The degree of each augmentation was limited and determined through trial and575.5. Experiment: Quantifying Error in a Black Box Commercial Algorithmerror in an attempt to promote useful augmentations in this experiment. A sampleof each augmentation is shown in Figure 5.5. Additionally, the size of the squaresin the grid dropout mask was set to a minimum side length of three pixels. Theminimum size of grid dropout squares was chosen to ensure that the model wouldnot consider the dropout as noise, thereby allowing the excluded regions to beomitted from the reference standard lung contour.To monitor the change in segmentation over a given set of slices, the IoU of eachstructure was observed. The standard error for each structure was calculated fromthe IoUs in each image set to give an estimate of the uncertainty in the system.All image augmentations performed for this task were completed using the Al-bumentations library [105]. Initially, slices were selected for augmentation basedon the constraint that they contained at least one of three structures: the left lung,the right lung, or the heart. The slices were selected at random, which did notaccount for the areas in the scans where the structures are small relative to theirsize in later slices. AI algorithms tend to struggle with images like these where thesize of the object being segmented is very different from the typical segmentationsize of the object, which makes these comparisons potentially not useful [82,106].With large variations in size, the addition of the TTA may not accurately representthe uncertainty for that slice. Small annotations needed to be excluded within thisexperiment, as the performance of the model was poorer in these regions. There-fore, slices where structures were less than 15% of the average structure size wereexcluded from this experiment’s analysis, based on visual inspection of the pro-cessed data.Figure 5.6 demonstrated the algorithm’s loss of ability to provide contours in animage. The left lung was recognized when the slice was augmented with noisebut was no longer recognized when an elastic deformation was applied to the im-age. Similar behaviour was observed in other images. Cases, where predictedstructures would disappear from one augmented slice to another, were excluded585.5. Experiment: Quantifying Error in a Black Box Commercial Algorithm(a) (b)(c) (d)Figure 5.5: Four types of augmentations were used within the TTA experiment.Grid dropout was applied to a CT slice. Limbus AI recognized the lungs withinthis image (a). Blurring was applied to a CT slice which resulted in the segmentingof both lungs and the heart (b). The application of an elastic deformation on animage resulted in segmentation of all three desired structures by Limbus AI (c).The insertion of noise into an images resulted in Limbus AI segmenting all threedesired structures (d).from the results of this experiment, as the validity of including these augmenta-tions was questionable. Therefore, prior to compiling the final results, the results595.5. Experiment: Quantifying Error in a Black Box Commercial Algorithmwere analyzed per set of augmented slices filtering out sets that contained manyzero or near zero IoU values. This allowed for the exclusion of these slices fromthe results of the TTA experiment. Additional information on a selection of theseslices is given in Appendix D.(a) (b)Figure 5.6: Example of segmented lung appearing in an image when noise wasinserted (a), and no longer appearing when an elastic deformation was applied tothe same image (b).5.5.2 ResultsA selection of sample segmentations of the augmented images can be seen inFigure 5.7. The mean IoU along with the standard error for each structure aresummarized in Table 5.2. Due to the exclusions of poor-quality slices, the sets ofslices used in the analysis were 66 for the left lung, 57 for the right lung, and 28for the heart. The number of samples for the heart was lower, as it was not presentin all of the slices.Table 5.2: Mean performance of Limbus AI with poor quality data excluded.Heart IoU Left Lung IoU Right Lung IoUMean IoU 0.724 0.865 0.865%Error 4.95 3.54 3.35605.5. Experiment: Quantifying Error in a Black Box Commercial AlgorithmFigure 5.7: Sample TTA segmentation for the left lung (top), right lung (middle),and the heart (bottom). The images show a comparison of the input CT slice (left),reference standard contour (center), and the prediction from the commercial AI(right).5.5.3 ConclusionAlthough the TTA method was able to establish an error associated with the com-mercial model, this method cannot be recommended, unless more rigorous meth-ods are unable to be performed due to restrictions. The use of TTA was user-dependent, as a variety of decisions had to be made which impacted the the anal-ysis of the results. Examples of these are the extent an image can be augmentedbefore its validity is questionable, and which slices should be considered eligible615.6. Discussion: Suggestions for the use of AI in Peer Reviewfor uncertainty analysis. The recommendation to not use TTA as demonstrated inthis experiment is based on the flexibility and adaptability of the conformal meth-ods to more accurately represent the error in the models without the mentioneduser dependency. The previously described conformal method could not be usedfor the limbus AI algorithm.5.6 Discussion: Suggestions for the use of AI inPeer ReviewFrom the discussion of metric selection, contouring variability, and error analysis,the complexities of establishing the use of AI for QA were highlighted. The de-cisions made when establishing an AI for QA in practice could have a significantimpact on its applicability to the task. Therefore, when AI is used in practice, theaspects of the implementation should be carefully considered to ensure safe andequitable practices [43]. It is suggested that both an area and distance-based met-ric be used to determine the similarity between segmentations [76, 91, 93]. Thereason for suggesting two types of metrics is to ensure that the contours not onlyhave a high overlap, but also that they do not have dissimilarities that may resultin a potential disagreement in a generated dose distribution for that patient [24].Within this chapter, methods for quantifying the error associated with an AI modelwere discussed. Understanding the error and limitations in an AI model beingused clinically is important, as the user should have confidence and understandaspects of the model [43]. Two approaches to quantify error in AI models werediscussed in this chapter. Conformal prediction was the more robust approachused [95–98, 102, 103, 107–109]. However, due to the limitations placed on theoutput of commercial AI tools, conformal prediction may not always be possible.Therefore, to approach uncertainty in a black box manner, in which the user isonly provided with the binary masks, this work has shown how TTA can be usedto provide an estimate of the uncertainty in an AI model. However, the conformal625.7. Summary of Recommendationsprediction method is recommended when possible, due to the results of the TTAexperiment being user dependent. The exploration of additional nonconformityscores in which the commercial algorithms could be used to perform conformalprediction is a topic of interest for future work, as it could remove the user de-pendency when quantifying the error in these algorithms. Performing an erroranalysis on AI systems prior to clinical use is recommended as a preliminary test,to ensure that the selected algorithm performs as desired on the specific patientpopulation. Additionally, these tests will help users gain some level of under-standing of the predictions made by the models. These suggestions were madeunder the context of fair and equitable use of AI systems [43].One important point made within this chapter is the inclusion of qualified profes-sionals, such as radiation oncologists within the implementation of AI for peerreview QA. The inclusion of radiation oncologists is integral to the developmentof QA procedures, as within the peer review process they are the ones decidingon whether or not a patient should proceed with the current treatment plan. Col-laboration between specialized groups, such as radiation oncologists and medicalphysicists, will aid in the design of thresholds based on imaging metrics and thecontoured volumes that encompass the practices of the clinic. [43].5.7 Summary of RecommendationsBased on the work done within this chapter, the following recommendations aremade for the use of AI for contour peer review:1. A combination of area-based and distance-based metrics should be used tocompare predictions to reference standard contours. It is suggested that theIoU and HD95 metrics be used.2. All metrics should be reported for each individual structure, instead of amean metric encompassing all structures.635.7. Summary of Recommendations3. Using a multidisciplinary team, appropriate thresholds for structures shouldbe selected based on the context of the structure.4. The desired algorithm should be tested on a set of data that is representativeof the potential target patient population.5. Uncertainty quantification should be performed to investigate the algorithmprior to deployment.64Chapter 6Conclusion6.1 SummaryThroughout this work, the training and testing of various deep learning algorithmswere explored for the purpose of image segmentation of chest CTs. The algo-rithms provided a basis for the exploration of the use of deep learning algorithmsfor radiotherapy contour peer review QA. Using the methods described withinthis work, the training and testing of algorithms to perform segmentation taskswas demonstrated, along with an error analysis of an in-house trained and a com-mercial model. Associated error in deep learning models is a topic that should befurther explored. We showcased two methods for quantifying error within deeplearning models, conformal prediction and TTA. Conformal prediction allows rig-orous quantification of error in deep learning models, based on the statistical ap-proach discussed in Chapter 5 [95–97, 103]. However, if commercial algorithmsare being used, conformal predictions may not always be possible due to the lim-ited information provided to the end user. The restrictive nature of commerciallyavailable products led to the use of TTA, which did allow uncertainty within themodel to be established. However, the TTA technique demonstrated is less rigor-ous than the conformal prediction method because the method for error analysisfrom Chapter 5 is user-dependent. Therefore, due to the extreme black box natureof commercial models, reliable in-house models may be a better option for QAprocesses, such as contouring peer review, where confidence in the model’s per-formance is important. The error associated with these in-house developed modelscan be quantified using rigorous approaches like conformal prediction. Establish-ing confidence within the model is an extremely important consideration, as we656.2. Future Worklook to move towards other sites where treatments consist of more sophisticatedtechniques and the error in structure delineation could result in dose spillage athigher rates [110]. Due to the risks of inaccurate contouring in these cases, it issuggested that some level of transparency be established between a clinic look-ing to purchase one of these commercial products, and the vendor supplying theproduct. The vendor should either be able to provide some quantification of theuncertainty over a user-provided data set upon request or should provide the insti-tutions using their product the ability to calculate these uncertainties in a rigorousway independently.6.2 Future WorkMore work needs to be done on the in-house segmentation algorithms, and theerror estimation techniques for AI models. Additionally, the error in the in-housemodels could be further encapsulated using conformal risk control, moving awayfrom marginal coverage provided by conformal prediction. As highlighted by An-gelopoulos and Bates, using conformal risk control will adjust the coverage ob-served in Chapter 5 to be stricter [111]. Additionally, methods for gauging uncer-tainty estimation and model performance could be extended beyond routine QAwith the increasing number of commercial AI applications, which may currentlynot be regulated to the standards other tools are within radiation therapy, such astreatment planning systems and linear accelerators. The increasing presence ofAI within healthcare provides an additional need for AI acceptance testing. Withappropriate consideration, the methods described in Chapter 5 could be extendedto acceptance testing for segmentation algorithms and other applications.Based on the structures segmented in this research, further work can be done toallow the segmentation of more structures. Exploration of additional structuresthrough additional data sources and further data curation is a topic of future in-terest. To be able to consider more structures, additional networks should also beexplored, to allow the segmentation of small structures within the CT scans.666.2. Future WorkIn conclusion, the work shown here demonstrates a variety of concepts applicableto deep learning, image segmentation, radiotherapy, breast cancer, and uncertaintyquantification. However, with the limited scope considered within this thesis,there are still a large amount of topics to be explored related to this research.Topics to be explored include research on image segmentation of structures notcurrently supported by commercial algorithms, work to establish a frameworkfor QA purposes, and the extension of this work to establish methodologies foracceptance testing of AI systems prior to clinical use.67Bibliography[1] D. R. Brenner, J. Gillis, A. A. Demers, L. F. Ellison, J. Billette, S. X.Zhang et al., “Projected estimates of cancer in Canada in 2024,” CMAJ,vol. 196, no. 18, pp. E615–E623, 2024, doi: 10.1503/cmaj.240095.[2] V. Arndt, H. Merx, C. Stegmaier, H. Ziegler, and H. Brenner, “Persistenceof restrictions in quality of life from the first to the third year afterdiagnosis in women with breast cancer,” Journal of Clinical Oncology,vol. 23, no. 22, pp. 4945–4953, 2005, doi: 10.1200/JCO.2005.03.475.[3] P. A. Ganz, L. Kwan, A. L. Stanton, J. E. Bower, and T. R. Belin,“Physical and psychosocial recovery in the year after primary treatment ofbreast cancer,” Journal of Clinical Oncology, vol. 29, no. 9, pp.1101–1109, 2011, doi: 10.1200/JCO.2010.28.8043.[4] U. Veronesi, N. Cascinelli, L. Mariani, M. Greco, R. Saccozzi, A. Luiniet al., “Twenty-year follow-up of a randomized study comparingbreast-conserving surgery with radical mastectomy for early breastcancer,” New England Journal of Medicine, vol. 347, no. 16, pp.1227–1232, 2002, doi: 10.1056/NEJMoa020989.[5] S. J. Schnitt, J. Hayman, R. Gelman, T. J. Eberlein, S. M. Love, K. Mayzelet al., “A prospective study of conservative surgery alone in the treatmentof selected patients with stage I breast cancer,” Cancer, vol. 77, no. 6, pp.1094–1100, 1996, doi: 10.1002/(SICI)1097-0142(19960315)77:6<1094::AID-CNCR14>3.0.CO;2-X.68Bibliography[6] S. Darby, P. McGale, C. Correa, C. Taylor, R. Arriagada, M. Clarke et al.,“Effect of radiotherapy after breast-conserving surgery on 10-yearrecurrence and 15-year breast cancer death: meta-analysis of individualpatient data for 10801 women in 17 randomised trials,” The Lancet, vol.378, no. 9804, pp. 1707–1716, 2011, doi:10.1016/S0140-6736(11)61629-2.[7] S. G. McDuff, C. E. Champ, S. Yoo, and R. C. Blitzblau, “Breast cancer,”in Khan’s Treatment Planning in Radiation Oncology, 5th ed. WoltersKluwer, 2022, ch. 8, pp. 134–154.[8] M. C. Joiner and S. M. Bentzen, “Fractionation: The linear-quadraticapproach,” in Basic clinical radiobiology. CRC press, 2018, ch. 9, pp.99–111.[9] M. Baumann, M. Krause, and V. Gre´goire, “Modified fractionation,” inBasic clinical radiobiology. CRC press, 2018, ch. 11, pp. 125–135.[10] M. C. Joiner, A. J. van der Kogel, and G. G. Steel, “Introduction: Thesignificance of radiobiology and radiotherapy for cancer treatment.” inBasic clinical radiobiology. CRC press, 2018, ch. 1, pp. 1–8.[11] S. M. Pierce, A. Recht, T. I. Lingos, A. Abner, F. Vicini, B. Silver et al.,“Long-term radiation complications following conservative surgery (CS)and radiation therapy (RT) in patients with early stage breast cancer,”International Journal of Radiation Oncology*Biology*Physics, vol. 23,no. 5, pp. 915–923, 1992, doi: 10.1016/0360-3016(92)90895-O.[12] U. Goldman, B. Wennberg, G. Svane, H. Bylund, and P. Lind, “Reductionof radiation pneumonitis by V20-constraints in breast cancer,” RadiationOncology, vol. 5, pp. 1–6, October 2010, doi: 10.1186/1748-717X-5-99.69Bibliography[13] C. L. Shapiro and A. Recht, “Side effects of adjuvant treatment of breastcancer,” New England Journal of Medicine, vol. 344, no. 26, pp.1997–2008, 2001, doi: 10.1056/NEJM200106283442607.[14] S. C. Darby, M. Ewertz, P. McGale, A. M. Bennet, U. Blom-Goldman,D. Brønnum et al., “Risk of ischemic heart disease in women afterradiotherapy for breast cancer,” New England Journal of Medicine, vol.368, no. 11, pp. 987–998, 2013, doi: 10.1056/NEJMoa1209825.[15] K. D. Miller, L. Nogueira, T. Devasia, A. B. Mariotto, K. R. Yabroff,A. Jemal et al., “Cancer treatment and survivorship statistics, 2022,” CA:A Cancer Journal for Clinicians, vol. 72, no. 5, pp. 409–436, 2022, doi:10.3322/caac.21731.[16] A. N. Giaquinto, H. Sung, K. D. Miller, J. L. Kramer, L. A. Newman,A. Minihan et al., “Breast cancer statistics, 2022,” CA: A Cancer Journalfor Clinicians, vol. 72, no. 6, pp. 524–541, 2022, doi: 10.3322/caac.21754.[17] W. Budach, K. Kammers, E. Boelke, and C. Matuschek, “Adjuvantradiotherapy of regional lymph nodes in breast cancer—a meta-analysis ofrandomized trials,” Radiation Oncology, vol. 8, pp. 1–7, December 2013,doi: 10.1186/1748-717X-8-267.[18] P. Despre´s and S. Gaede, “COMP report: CPQR technical quality controlguidelines for CT simulators,” Journal of Applied Clinical MedicalPhysics, vol. 19, no. 2, pp. 12–17, 2018, doi: 10.1002/acm2.12213.[19] D. Latty, K. E. Stuart, W. Wang, and V. Ahern, “Review of deepinspiration breath-hold techniques for the treatment of breast cancer,”Journal of Medical Radiation Sciences, vol. 62, no. 1, pp. 74–81, 2015,doi: 10.1002/jmrs.96.70Bibliography[20] D. N. Mihailidis and N. Papanikolaou, “Patient simulation,” inKhan’s Treatment Planning in Radiation Oncology, 5th ed. WoltersKluwer, 2022, ch. 19, pp. 420–436.[21] W. Ba¨r, M. Schwarz, M. Alber, L. J. Bos, B. J. Mijnheer, C. Rasch et al.,“A comparison of forward and inverse treatment planning forintensity-modulated radiotherapy of head and neck cancer,” Radiotherapyand Oncology, vol. 69, no. 3, pp. 251–258, 2003, doi:10.1016/j.radonc.2003.08.002.[22] D. Thorwarth, X. Geets, and M. Paiusco, “Physical radiotherapy treatmentplanning based on functional PET/CT data,” Radiotherapy and Oncology,vol. 96, no. 3, pp. 317–324, 2010, PET in Radiotherapy Planning. doi:10.1016/j.radonc.2010.07.012.[23] F. M. Khan and J. P. Gibbons, “Introduction: Process, equipment, andpersonnel,” in Khan’s Treatment Planning in Radiation Oncology, 5th ed.Wolters Kluwer, 2022, ch. 16, pp. 366–373.[24] X. A. Li, A. Tai, D. W. Arthur, T. A. Buchholz, S. Macdonald, L. B. Markset al., “Variability of target and normal structure delineation for breastcancer radiotherapy: An RTOG multi-institutional and multiobserverstudy,” International Journal of Radiation Oncology*Biology*Physics,vol. 73, no. 3, pp. 944–951, 2009, doi: 10.1016/j.ijrobp.2008.10.034.[25] Canadian Partnership for Quality Radiotherapy, “Quality assuranceguidelines for canadian radiation treatment programs,” December 2019,[Internet]. [cited 2023 May 1]. Available at:https://www.cpqr.ca/wp-content/uploads/2020/03/QRT2019-12-04.pdf.[26] L. Perna, C. Cozzarini, E. Maggiulli et al., “Inter-observer variability incontouring the penile bulb on CT images for prostate cancer treatmentplanning,” Radiation Oncology, vol. 6, p. 123, 2011, doi:10.1186/1748-717X-6-123.71Bibliography[27] I. J. Das, J. J. Compton, A. Bajaj, and P. A. Johnstone, “Intra- andinter-physician variability in target volume delineation in radiationtherapy,” Journal of Radiation Research, vol. 62, no. 6, pp. 1083–1089, 092021, doi: 10.1093/jrr/rrab080.[28] S. Ramasamy, L. Murray, K. Cardale, K. Dyker, P. Murray, M. Sen et al.,“Quality assurance peer review of head and neck contours in a largecancer centre via a weekly meeting approach,” Clinical Oncology, vol. 31,no. 6, pp. 344–351, 2019, doi: 10.1016/j.clon.2019.03.001.[29] B. W. Cox, A. Kapur, A. Sharma, L. Lee, B. Bloom, R. Sharma et al.,“Prospective contouring rounds: A novel, high-impact tool for optimizingquality assurance,” Practical Radiation Oncology, vol. 5, no. 5, pp.e431–e436, 2015, safety Issue. doi: 10.1016/j.prro.2015.05.005.[30] K. Brunskill, T. K. Nguyen, R. G. Boldt, A. V. Louie, A. Warner, L. B.Marks et al., “Does peer review of radiation plans affect clinical care? asystematic review of the literature,” International Journal of RadiationOncology*Biology*Physics, vol. 97, no. 1, pp. 27–34, 2017, doi:10.1016/j.ijrobp.2016.09.015.[31] S. N. Hamilton, H. Hasan, C. Parsons, S. Tyldesley, A. F. Howard, M. A.Bobinski et al., “Canadian radiation oncologists’ opinions regarding peerreview: A national survey,” Practical Radiation Oncology, vol. 5, no. 2,pp. 120–126, 2015, doi: 10.1016/j.prro.2014.06.002.[32] M. May, “Eight ways machine learning is assisting medicine,” NatureMedicine, vol. 27, no. 1, pp. 2–3, January 2021, doi:10.1038/s41591-020-01197-2.[33] Y. Liu, T. Kohlberger, M. Norouzi, G. E. Dahl, J. L. Smith,A. Mohtashamian et al., “Artificial intelligence–based breast cancer nodalmetastasis detection: Insights into the black box for pathologists,”72BibliographyArchives of Pathology Laboratory Medicine, vol. 143, no. 7, pp. 859–868,10 2018, doi: 10.5858/arpa.2018-0147-OA.[34] K. Dembrower, Y. Liu, H. Azizpour, M. Eklund, K. Smith, P. Lindholmet al., “Comparison of a deep learning risk score and standardmammographic density score for breast cancer risk prediction,”Radiology, vol. 294, no. 2, pp. 265–272, 2020, pMID: 31845842. doi:10.1148/radiol.2019190872.[35] A. Yala, C. Lehman, T. Schuster, T. Portnoi, and R. Barzilay, “A deeplearning mammography-based model for improved breast cancer riskprediction,” Radiology, vol. 292, no. 1, pp. 60–66, 2019, pMID:31063083. doi: 10.1148/radiol.2019182716.[36] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++:A nested U-Net architecture for medical image segmentation,” in DeepLearning in Medical Image Analysis and Multimodal Learning forClinical Decision Support, D. Stoyanov, Z. Taylor, G. Carneiro,T. Syeda-Mahmood, A. Martel, L. Maier-Hein et al., Eds. Cham:Springer International Publishing, 2018, pp. 3–11.[37] G. Ramasamy, T. Singh, and X. Yuan, “Multi-modal semanticsegmentation model using encoder based Link-Net architecture for BraTS2020 challenge,” Procedia Computer Science, vol. 218, pp. 732–740,2023, international Conference on Machine Learning and DataEngineering. doi: 10.1016/j.procs.2023.01.053.[38] P. Rouzrokh, B. Khosravi, S. Faghani, M. Moassefi, D. V. Vera Garcia,Y. Singh et al., “Mitigating bias in radiology machine learning: 1. datahandling,” Radiology: Artificial Intelligence, vol. 4, no. 5, p. e210290,2022, pMID: 35652119. doi: 10.1148/ryai.210290.[39] K. Zhang, B. Khosravi, S. Vahdati, S. Faghani, F. Nugen, S. M.Rassoulinejad-Mousavi et al., “Mitigating bias in radiology machine73Bibliographylearning: 2. model development,” Radiology: Artificial Intelligence,vol. 4, no. 5, p. e220010, 2022, doi: 10.1148/ryai.220010.[40] S. Faghani, B. Khosravi, K. Zhang, M. Moassefi, J. M. Jagtap, F. Nugenet al., “Mitigating bias in radiology machine learning: 3. performancemetrics,” Radiology: Artificial Intelligence, vol. 4, no. 5, p. e220061,2022, doi: 10.1148/ryai.220061.[41] A. Tang, R. Tam, A. Cadrin-Cheˆnevert, and et al., “Canadian associationof radiologists white paper on artificial intelligence in radiology,”Canadian Association of Radiologists Journal, vol. 69, no. 2, pp. 120–135,2018, doi: 10.1016/j.carj.2018.02.002.[42] S. Hickman, G. Baxter, and F. Gilbert, “Adoption of artificial intelligencein breast imaging: evaluation, ethical constraints and limitations,” BritishJournal of Cancer, vol. 125, no. 1, pp. 15–22, July 2021, doi:10.1038/s41416-021-01333-w.[43] D. Ueda, T. Kakinuma, S. Fujita et al., “Fairness of artificial intelligencein healthcare: review and recommendations,” Japanese Journal ofRadiology, vol. 42, pp. 3–15, 2024, doi: 10.1007/s11604-023-01474-3.[44] D. Pinto dos Santos, D. Giese, S. Brodehl, S. H. Chon, W. Staab,R. Kleinert et al., “Medical students’ attitude towards artificialintelligence: a multicentre survey,” European Radiology, vol. 29, no. 4,pp. 1640–1646, Apr. 2019, doi: 10.1007/s00330-018-5601-1.[45] I. Goodfellow, Y. Bengio, and A. Courville, “Introduction,” in DeepLearning. Cambridge, MA: MIT Press, 2016, ch. 1, pp. 1–26.[46] R. Solovyev, A. A. Kalinin, and T. Gabruseva, “3D convolutional neuralnetworks for stalled brain capillary detection,” Computers in Biology andMedicine, vol. 141, p. 105089, 2022, doi:10.1016/j.compbiomed.2021.105089.74Bibliography[47] O¨. C¸ic¸ek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger,“3D U-Net: Learning dense volumetric segmentation from sparseannotation,” in Medical Image Computing and Computer-AssistedIntervention – MICCAI 2016, S. Ourselin, L. Joskowicz, M. R. Sabuncu,G. Unal, and W. Wells, Eds. Cham: Springer International Publishing,2016, pp. 424–432.[48] D. Ciresan, A. Giusti, L. Gambardella, and J. Schmidhuber, “Deep neuralnetworks segment neuronal membranes in electron microscopy images,”in Advances in Neural Information Processing Systems, F. Pereira,C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. CurranAssociates, Inc., 2012.[49] P. J. Doolan, S. Charalambous, Y. Roussakis, A. Leczynski, M. Peratikou,M. Benjamin et al., “A clinical evaluation of the performance of fivecommercial artificial intelligence contouring systems for radiotherapy,”Frontiers in Oncology, vol. 13, 2023, doi: 10.3389/fonc.2023.1213068.[50] K. G. van Leeuwen, S. Schalekamp, M. J. C. M. Rutten, M. Huisman,C. M. Schaefer-Prokop, M. de Rooij et al., “Comparison of commercial AIsoftware performance for radiograph lung nodule detection and bone ageprediction,” Radiology, vol. 310, no. 1, p. e230981, 2024, pMID:38193833. doi: 10.1148/radiol.230981.[51] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networksfor biomedical image segmentation,” in Medical Image Computing andComputer-Assisted Intervention – MICCAI 2015, N. Navab, J. Hornegger,W. M. Wells, and A. F. Frangi, Eds. Cham: Springer InternationalPublishing, 2015, pp. 234–241.[52] A. Chaurasia and E. Culurciello, “LinkNet: Exploiting encoderrepresentations for efficient semantic segmentation,” in 2017 IEEE Visual75BibliographyCommunications and Image Processing (VCIP), 2017, pp. 1–4, doi:10.1109/VCIP.2017.8305148.[53] B. Ait Skourt, A. El Hassani, and A. Majda, “Lung CT imagesegmentation using deep neural networks,” Procedia Computer Science,vol. 127, pp. 109–113, 2018, proceedings of the First InternationalConference on Intelligent Computing in Data Sciences, ICDS2017. doi:10.1016/j.procs.2018.01.104.[54] A. A. Kalinin, V. I. Iglovikov, A. Rakhlin, and A. A. Shvets, “Medicalimage segmentation using deep neural networks with pre-trainedencoders,” in Deep Learning Applications, M. A. Wani, M. Kantardzic,and M. Sayed-Mouchaweh, Eds. Singapore: Springer Singapore, 2020,pp. 39–52, doi: 10.1007/978-981-15-1816-4 3.[55] M. Wu, C. Zhang, J. Liu, L. Zhou, and X. Li, “Towards accurate highresolution satellite image semantic segmentation,” IEEE Access, vol. 7,pp. 55 609–55 619, 2019, doi: 10.1109/ACCESS.2019.2913442.[56] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition (CVPR), June 2016.[57] K. Abdelouahab, M. Pelcat, and F. Berry, “Accelerating the CNNInference on FPGAs,” in Deep Learning in Computer Vision, ser.Principles and Applications. CRC Press, 2020, ch. 1, pp. 1–39, doi:10.1201/9781351003827.[58] H. Rashed, S. Yogamani, A. El-Sallab, M. Hassaballah, and M. Elhelw,“Deep semantic segmentation in autonomous driving,” inDeep Learning in Computer Vision, ser. Principles and Applications.CRC Press, 2020, ch. 6, pp. 151–182, doi: 10.1201/9781351003827.76Bibliography[59] I. Goodfellow, Y. Bengio, and A. Courville, “Convolutional networks,” inDeep Learning. MIT Press, 2016, ch. 9, pp. 326–366.[60] T. Lei and A. K. Nandi, “Neural networks,” in Image Segmentation.John Wiley Sons, Ltd, 2022, ch. 4, pp. 63–96, doi:10.1002/9781119859048.ch9.[61] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollar, “Panopticsegmentation,” in Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), June 2019.[62] T. Lei and A. K. Nandi, “Introduction,” in Image Segmentation. JohnWiley Sons, Ltd, 2022, ch. 1, pp. 1–19, doi: 10.1002/9781119859048.ch1.[63] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks forsemantic segmentation,” in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR), June 2015.[64] I. Goodfellow, Y. Bengio, and A. Courville, “Deep feedforward networks,”in Deep Learning. Cambridge, MA: MIT Press, 2016, ch. 6, pp.164–223.[65] P. Iakubovskii, “Segmentation models,” 2019, [Internet]. [cited 2023 July20 ]. Available from: https://github.com/qubvel/segmentation models.[66] M. Goldblum, H. Souri, R. Ni, M. Shu, V. Prabhu, G. Somepalli et al.,“Battle of the backbones: A large-scale comparison of pretrained modelsacross computer vision tasks,” in Advances in Neural InformationProcessing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko,M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp.29 343–29 371.[67] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, “Imagenet: Alarge-scale hierarchical image database,” in 2009 IEEE Conference on77BibliographyComputer Vision and Pattern Recognition, 2009, pp. 248–255, doi:10.1109/CVPR.2009.5206848.[68] K. He, R. Girshick, and P. Dollar, “Rethinking imagenet pre-training,” inProceedings of the IEEE/CVF International Conference on ComputerVision (ICCV), October 2019.[69] I. Bello, W. Fedus, X. Du, E. D. Cubuk, A. Srinivas, T. Lin et al.,“Revisiting ResNets: Improved training and scaling strategies,” inAdvances in Neural Information Processing Systems, M. Ranzato,A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34.Curran Associates, Inc., 2021, pp. 22 614–22 627.[70] Limbus AI, “Automatic contouring for radiation therapy,” [Internet]. [cited2024 March 8]. Available from: https://limbus.ai/.[71] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Griselet al., “Scikit-learn: Machine learning in Python,” Journal of MachineLearning Research, vol. 12, pp. 2825–2830, 2011.[72] NVIDIA, “Technical specifications of NVIDIA® QUADRO® P2000,”2024, [Internet]. [cited 2024 February 10]. Available from:https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/productspage/quadro/quadro-desktop/quadro-pascal-p2000-data-sheet-us-nvidia-704443-r2-web.pdf.[73] I. Goodfellow, Y. Bengio, and A. Courville, “Optimization for trainingdeep models,” in Deep Learning. Cambridge, MA: MIT Press, 2016,ch. 8, pp. 271–325.[74] NVIDIA Corporation, “GeForce RTX 3060 and 3060 Ti graphics cards,”2024, [Internet]. [cited 2024 February 10]. Available from: https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3060-3060ti.78Bibliography[75] W. Wu, D. Verma, W. Yang, and Z. Lan, “Patchify: A simple Pythonlibrary for patchifying 2D and 3D images and their annotations,” 2023,[Internet]. [cited 2023 September 15]. Available from:https://github.com/dovahcrow/patchify.py.[76] A. Reinke, M. Tizabi, C. Sudre, M. Eisenmann, T. Ra¨dsch,M. Baumgartner et al., “Common limitations of image processing metrics:A picture story,” arXiv preprint arXiv:2104.05642, April 2021.[77] M. A. Rahman and Y. Wang, “Optimizing intersection-over-union in deepneural networks for image segmentation,” in Advances in VisualComputing, G. Bebis, R. Boyle, B. Parvin, D. Koracin, F. Porikli, S. Skaffet al., Eds. Cham: Springer International Publishing, 2016, pp. 234–244.[78] T. Lei and A. K. Nandi, “Image segmentation for medical analysis,” inImage Segmentation. John Wiley Sons, Ltd, 2022, ch. 9, pp. 199–227,doi: 10.1002/9781119859048.ch9.[79] S. Jadon, “A survey of loss functions for semantic segmentation,” in 2020IEEE Conference on Computational Intelligence in Bioinformatics andComputational Biology (CIBCB), 2020, pp. 1–7, doi:10.1109/CIBCB48159.2020.9277638.[80] J. Bushberg, J. Seibert, E. J. Leidholdt, and J. Boone, “Computedtomography,” in The Essential Physics of Medical Imaging, 3rd ed.,J. Bushberg, J. Seibert, E. J. Leidholdt, and J. Boone, Eds. Philadelphia:Lippincott Williams & Wilkins, 2012.[81] A. Rusiecki, “Trimmed categorical cross-entropy for deep learning withlabel noise,” Electronics Letters, vol. 55, no. 6, pp. 319–320, 2019, doi:10.1049/el.2018.7980.79Bibliography[82] X. Liu, K. Li, R. Yang, and L. Geng, “Review of deep learning basedautomatic segmentation for lung cancer radiotherapy,” Frontiers inOncology, vol. 11, 2021, doi: 10.3389/fonc.2021.717039.[83] I. Qureshi, J. Yan, Q. Abbas, K. Shaheed, A. B. Riaz, A. Wahid et al.,“Medical image segmentation using deep semantic-based methods: Areview of techniques, applications and emerging trends,” InformationFusion, vol. 90, pp. 316–352, 2023, doi: 10.1016/j.inffus.2022.09.031.[84] X. Feng, K. Qing, N. J. Tustison, C. H. Meyer, and Q. Chen, “Deepconvolutional neural network for segmentation of thoracic organs-at-riskusing cropped 3D images,” Medical Physics, vol. 46, no. 5, pp.2169–2180, 2019, doi: 10.1002/mp.13466.[85] J. Zhu, J. Zhang, B. Qiu, Y. Liu, X. Liu, and L. Chen, “Comparison of theautomatic segmentation of multiple organs at risk in CT images of lungcancer between deep convolutional neural network-based and atlas-basedtechniques,” Acta Oncologica, vol. 58, no. 2, pp. 257–264, 2019, pMID:30398090. doi: 10.1080/0284186X.2018.1529421.[86] X. Dong, Y. Lei, T. Wang, M. Thomas, L. Tang, W. J. Curran et al.,“Automatic multiorgan segmentation in thorax CT images usingU-net-GAN,” Medical Physics, vol. 46, no. 5, pp. 2157–2168, 2019, doi:10.1002/mp.13458.[87] T. Zhang, Y. Yang, J. Wang, K. Men, X. Wang, L. Deng et al.,“Comparison between atlas and convolutional neural network basedautomatic segmentation of multiple organs at risk in non-small cell lungcancer,” Medicine, vol. 99, no. 34, p. e20940, 2020, doi:10.1097/MD.0000000000021800.[88] R. Wang, T. Lei, R. Cui, B. Zhang, H. Meng, and A. K. Nandi, “Medicalimage segmentation using deep learning: A survey,” IET ImageProcessing, vol. 16, no. 5, pp. 1243–1267, 2022, doi: 10.1049/ipr2.12419.80Bibliography[89] X. Feng, C. Wang, S. Cheng, and L. Guo, “Automatic liver and tumorsegmentation of CT based on cascaded U-Net,” in Proceedings of 2018Chinese Intelligent Systems Conference, Y. Jia, J. Du, and W. Zhang, Eds.Singapore: Springer Singapore, 2019, pp. 155–164.[90] A. Agrawal, A. A. Ayantunde, and K. L. Cheung, “Concepts of seromaformation and prevention in breast cancer surgery,” ANZ Journal ofSurgery, vol. 76, no. 12, pp. 1088–1095, 2006, doi:10.1111/j.1445-2197.2006.03949.x.[91] A. Reinke, M. Tizabi, M. Baumgartner, M. Eisenmann,D. Heckmann-No¨tzel, A. Kavur et al., “Understanding metric-relatedpitfalls in image analysis validation,” Nature Methods, vol. 21, no. 2, pp.182–194, February 2024, doi: 10.1038/s41592-023-02150-0.[92] V. Yeghiazaryan and I. D. Voiculescu, “Family of boundary overlapmetrics for the evaluation of medical image segmentation,” Journal ofMedical Imaging, vol. 5, no. 1, p. 015006, 2018, doi:10.1117/1.JMI.5.1.015006.[93] L. Maier-Hein, A. Reinke, P. Godau, M. D. Tizabi, F. Buettner,E. Christodoulou et al., “Metrics reloaded: recommendations for imageanalysis validation,” Nature Methods, vol. 21, no. 2, pp. 195–212, 2024,doi: 10.1038/s41592-023-02151-z.[94] C. Fiorino, M. Reni, A. Bolognesi, G. M. Cattaneo, and R. Calandrino,“Intra- and inter-observer variability in contouring prostate and seminalvesicles: implications for conformal treatment planning,” Radiotherapyand Oncology, vol. 47, no. 3, pp. 285–292, 1998, doi:10.1016/S0167-8140(98)00021-8.[95] G. Shafer and V. Vovk, “A tutorial on conformal prediction.” Journal ofMachine Learning Research, vol. 9, no. 3, 2008.81Bibliography[96] V. Vovk, A. Gammerman, and G. Shafer, Algorithmic learning in arandom world. Springer, 2005, vol. 29.[97] A. N. Angelopoulos and S. Bates, “A gentle introduction to conformalprediction and distribution-free uncertainty quantification,” 2022.[98] T. Cordier, V. Blot, L. Lacombe, T. Morzadec, A. Capitaine, andN. Brunel, “Flexible and systematic uncertainty estimation with conformalprediction via the MAPIE library,” in Conformal and ProbabilisticPrediction with Applications. PMLR, 2023, pp. 549–581, version 0.8.6,released April 30, 2019,[Internet]. [cited 2024 March 15]. Available from:https://github.com/scikit-learn-contrib/MAPIE.[99] J. Wang, X. Yang, H. Cai, W. Tan, C. Jin, and L. Li, “Discrimination ofbreast cancer with microcalcifications on mammography by deeplearning,” Scientific Reports, vol. 6, no. 1, p. 27327, June 2016, doi:10.1038/srep27327.[100] G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren,“Aleatoric uncertainty estimation with test-time augmentation for medicalimage segmentation with convolutional neural networks,”Neurocomputing, vol. 338, pp. 34–45, 2019, doi:10.1016/j.neucom.2019.01.103.[101] A. Rakhlin, A. Tiulpin, A. A. Shvets, A. A. Kalinin, V. I. Iglovikov, andS. Nikolenko, “Breast tumor cellularity assessment using deep neuralnetworks,” in Proceedings of the IEEE/CVF International Conference onComputer Vision (ICCV) Workshops, Oct 2019.[102] H. Wieslander, P. J. Harrison, G. Skogberg, S. Jackson, M. Fride´n,J. Karlsson et al., “Deep learning with conformal prediction forhierarchical analysis of large-scale whole-slide tissue images,” IEEEJournal of Biomedical and Health Informatics, vol. 25, no. 2, pp. 371–380,2021, doi: 10.1109/JBHI.2020.2996300.82Bibliography[103] Y. Romano, M. Sesia, and E. Candes, “Classification with valid andadaptive coverage,” in Advances in Neural Information ProcessingSystems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin,Eds., vol. 33. Curran Associates, Inc., 2020, pp. 3581–3591.[104] M. S. Ayhan and P. Berens, “Test-time data augmentation for estimation ofheteroscedastic aleatoric uncertainty in deep neural networks,” in MedicalImaging with Deep Learning, 2018.[105] A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin,and A. A. Kalinin, “Albumentations: Fast and flexible imageaugmentations,” Information, vol. 11, no. 2, 2020.[106] N. S. Punn and S. Agarwal, “Modality specific U-Net variants forbiomedical image segmentation: a survey,” Artificial Intelligence Review,vol. 55, no. 7, pp. 5845–5889, Oct 2022, doi:10.1007/s10462-022-10152-1.[107] H. Olsson, K. Kartasalo, N. Mulliqi et al., “Estimating diagnosticuncertainty in artificial intelligence assisted pathology using conformalprediction,” Nature Communications, vol. 13, p. 7761, 2022, doi:10.1038/s41467-022-34945-8.[108] S. Ghosh, T. Belkhouja, Y. Yan, and J. R. Doppa, “Improving uncertaintyquantification of deep classifiers via neighborhood conformal prediction:Novel algorithm and theoretical analysis,” Proceedings of the AAAIConference on Artificial Intelligence, vol. 37, no. 6, pp. 7722–7730, Jun.2023, doi: 10.1609/aaai.v37i6.25936.[109] A. Timans, C. Straehle, K. Sakmann, and E. Nalisnick, “Adaptivebounding box uncertainties via two-step conformal prediction,” 2024.83[110] M. Lin, A. Godley, and R. D. Timmerman, “Stereotactic ablativeradiotherapy,” in Khan’s Treatment Planning in Radiation Oncology,5th ed. Wolters Kluwer, 2022, ch. 28, pp. 595–612.[111] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster,“Conformal risk control,” 2023.84Appendix AAdditional Data Annotation TrendsFigure A.1: Appearance of the seroma annotation within dataset. Prior to 2006,no seroma cases were present in the dataset, and after the total number of seromacases was low compared to the total breast cancer radiotherapy treatments. Thedata available for 2019 only 16 breast cancer cases contained the seroma annota-tion.85Appendix A. Additional Data Annotation TrendsFigure A.2: Appearance of the left lung and right lung annotations within the samecase. Within each year of the available data, cases containing both the left lungand right lung annotations were far fewer than the total number of breast cancertreatments.86Appendix BAdditional Results FromSegmentation ExperimentsB.1 Binary SegmentationB.1.1 Training ParametersTable B.1: Training parameters for the binary segmentation of the left lung.Model 2D U-Net 2D LinkNet 3D U-Net 3D LinkNetBatch Size 4 4 4 4Learning Rate 0.00001 0.00001 0.000005 0.00001Learning Rate Scheduler 10 Epochs 10 Epochs 10 Epochs 10 EpochsEarly Stopping 20 20 20 20# of Epochs Available 280 280 160 160Epoch Training Ended 50 79 72 115Table B.2: Training parameters for the binary segmentation of the heart.Model 2D U-Net 2D LinkNet 3D U-Net 3D LinkNetBatch Size 4 4 4 4Learning Rate 0.00002 0.00002 0.0001 0.00008Learning Rate Scheduler 10 Epochs 10 Epochs 5 Epochs 5 EpochsEarly Stopping 15 15 10 10# of Epochs Available 160 160 80 80Epoch Training Ended 70 42 42 3987B.1. Binary SegmentationB.1.2 IoU Training and Validation Curves(a) (b)(c) (d)Figure B.1: Training and validation IoU curves for 2D U-Net (a), 2D LinkNet (b),3D U-Net (c), and 3D LinkNet (d) for the left lung trials.88B.1. Binary Segmentation(a) (b)(c) (d)Figure B.2: Training and validation IoU for 2D U-Net (a), 2D LinkNet (b), 3DU-Net (c), and 3D LinkNet (d) for the binary heart segmentation trials.B.1.3 Sample Segmentations89B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.3: 2D U-Net sample segmentations of the left Lung. Each row representsa CT slice in which 2D U-Net was used to predict the left lung. The comparisonshows CT slices (left), reference standard annotations (middle), and 2D U-Netpredictions (right). The white arrows in the reference standard annotations andthe 2D U-Net predictions point to a difference between the two images.90B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.4: 2D LinkNet sample segmentations of the left Lung. Each row rep-resents a CT slice in which 2D LinkNet was used to predict the left lung. Thecomparison shows CT slices (left), reference standard annotations (middle), and2D LinkNet predictions (right). The white arrows in the reference standard an-notations and the 2D LinkNet predictions point to a difference between the twoimages.91B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.5: 3D U-Net sample segmentations of the left Lung. Each row representsa CT slice in which 3D U-Net was used to predict the left lung. The comparisonshows CT slices (left), reference standard annotations (middle), and 3D U-Netpredictions (right). The white arrows in the reference standard annotations andthe 3D U-Net predictions point to a difference between the two images.92B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.6: 3D LinkNet sample segmentations of the left Lung. Each row rep-resents a CT slice in which 3D LinkNet was used to predict the left lung. Thecomparison shows CT slices (left), reference standard annotations (middle), and3D LinkNet predictions (right).The white arrows in the reference standard an-notations and the 3D LinkNet predictions point to a difference between the twoimages.93B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.7: 2D U-Net sample segmentations of the heart. Each row representsa CT slice in which 2D U-Net was used to predict the heart. The comparisonshows CT slices (left), reference standard annotations (middle), and 2D U-Netpredictions (right). The white arrows in the reference standard annotations andthe 2D U-Net predictions point to a difference between the two images.94B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.8: 2D LinkNet sample segmentations of the heart. Each row representsa CT slice in which 2D LinkNet was used to predict the heart. The comparisonshows CT slices (left), reference standard annotations (middle), and 2D LinkNetpredictions (right). The white arrows in the reference standard annotations andthe 2D LinkNet predictions point to a difference between the two images.95B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.9: 3D U-Net sample segmentations of the heart. Each row representsa CT slice in which 3D U-Net was used to predict the heart. The comparisonshows CT slices (left), reference standard annotations (middle), and 3D U-Netpredictions (right). The white arrows in the reference standard annotations andthe 3D U-Net predictions point to a difference between the two images.96B.1. Binary SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.10: 3D LinkNet sample segmentations of the heart. Each row representsa CT slice in which 3D LinkNet was used to predict the heart. The comparisonshows CT slices (left), reference standard annotations (middle), and 3D LinkNetpredictions (right). The white arrows in the reference standard annotations andthe 3D LinkNet predictions point to a difference between the two images.97B.2. Multi class SegmentationB.2 Multi class SegmentationB.2.1 Training ParametersTable B.3: Training parameters for the multiclass segmentation experiment.Model 2D U-Net 2D LinkNetBatch Size 4 4Learning Rate 0.0001 0.0001Learning Rate Scheduler 5 Epochs 5 EpochsEarly Stopping 10 10# of Epochs Available 160 160Epoch Training Ended 35 43B.2.2 IoU Training and Validation Curves(a) . (b)Figure B.11: Training and validation IoU curves for 2D U-Net (a), and 2DLinkNet (b) for the multiclass trials.B.2.3 Sample Segmentations98B.2. Multi class SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.12: 2D U-Net sample multiclass segmentations. Each row represents aCT slice in which 2D U-Net was used to predict the three structures. The com-parison shows CT slices (left), reference standard annotations (middle), and 2DU-Net predictions (right). The white arrows in the reference standard annotationsand the 2D U-Net predictions point to a difference between the two images.99B.2. Multi class SegmentationCT Slice Reference Standard PredictionContourPatient 1Patient 2Patient 3Figure B.13: 2D LinkNet sample multiclass segmentations. Each row representsa CT slice in which 2D LinkNet was used to predict the three structures. Thecomparison shows CT slices (left), reference standard annotations (middle), and2D LinkNet predictions (right). The white arrows in the reference standard an-notations and the 2D LinkNet predictions point to a difference between the twoimages.100Appendix CHardware and TimingConsiderationsC.1 Timing ComparisonsThe computing requirements for a completed clinical tool leveraging the powerof pre-trained networks are explored within this appendix. The time required forCNNs to perform predictions on two computers was recorded.C.2 Methods: Performance Comparison BetweenMachinesComputer 1 is equipped with 5GB of VRAM. As shown in this work Computer1 is capable of performing the 2D segmentation tasks. However, one thing toconsider is the potential improvement in speed by running the same networks onComputer 2 equipped with the RTX 3060. For this speed comparison, data wasstored on a network drive to ensure neither computer had an advantage in dataaccess speed. Comparisons were made on the task of segmenting the left lung.The programs were timed on each machine for predictions made on the same6 CT scans. The comparison between the devices over the 6 CTs included dataaccess time and evaluation time. Both LinkNet and U-Net are tested for predictionspeed on both computers.101C.3. Methods: Comparison of CNN Speeds on the Same ComputerC.2.1 Results: Performance Comparison Between MachinesOverall, the computer with the RTX 3060 performed better than the computerwith the P2000, as summarized in Table C.1.Table C.1: Time required for the 2D CNNs to predict over 6 full CTs on bothcomputers.Network PC 1 Time (s) PC 2 Time (s)2D U-Net 67 362D LinkNet 65 31C.3 Methods: Comparison of CNN Speeds on theSame ComputerThe overall speed of the CNN should be considered in addition to the accuracyof the model. Since the 2D CNNs and 3D CNNs were not trained on the samemachine they can not be directly compared based on the training results. An ex-periment was performed comparing the 4 left lung models on the same computer.All 4 models were used to predict over 6 full CTs. The networks used for thisanalysis are the left lung segmentation algorithms, as all 4 networks were trainedfor this task. To ensure a fair comparison the batch sizes, and CTs stored withinRAM were the same. The preprocessing time for the 3D and 2D networks wasconsidered to be incorporated in the overall prediction time.C.3.1 Results: Comparison of Network Speeds on the SameComputerOverall, the 2D LinkNet variant performed the fastest. The CNN speeds aresummarized in Table C.2.102C.4. DiscussionTable C.2: Time required for all four CNNs to predict over 6 full CTs on computer2.Network Time (s)2D U-Net 362D LinkNet 313D U-Net 403D LinkNet 41C.4 DiscussionThe difference in speed for the same predictions between the computers is high-lighted in Table C.1.. There was a large difference in the time required to makepredictions on CTs between computer 1 and computer 2. Both CNNs took over1.8 times the time to complete the predictions on computer 1 than it took them tocomplete the same predictions on computer 2. However, the time to predict overa full CT is only 11.2 seconds for 2D U-Net on computer 1. Therefore, if a taskrequired a large number of predictions the performance difference would be verynoticeable between computers, such as the centralized processing of many CTsfrom different centres. Assuming 500 CTs are processed using LinkNet and thesmaller GPU, the images would be processed in approximately 1.5 hours. Usingthe larger GPU this number would be 43 minutes. However, in the context ofpredicting over a single CT both computers make the predictions in under 12 sec-onds. Additionally, the time required to make the predictions over one CT took amaximum of 6.83 seconds on computer 2. Therefore, a larger GPU is only a con-cern when either the method requires a larger network or the patient throughput ishigh.103Appendix DAdditional Data for Test TimeAugmentation ExperimentAs discussed in Chapter 5, potentially not useful slices were removed from theTTA experiment. These cases either contained small pieces of a given structureor had augmentation beyond what could be determined to be useful to the erroranalysis performed. Figure D.1 is an example of such a slice. When lookingat box plots these slices can be identified by boxes with a low IoU and a tightdistribution of IoUs. This is because, with the addition of the augmentations, themodel often does not predict the structure in these locations. Slice 1 in Table D.1is an example of an excluded slice. An example of such box plots for the left lungis shown in Figure D.2. Additionally, the mean IoUs for the first 70 groups of CTslice augmentations are summarized in Tables D.1 and D.2.Figure D.1: Augmented image in which augmentation resulted in both lungs notbeing recognized by Limbus AI.104Appendix D. Additional Data for Test Time Augmentation ExperimentFigure D.2: Box plot analyzing the changes in the performance of the commercialAI when segmenting the left lung for the first 30 sets of augmented CT slices.Slices 1, 4, and 6 are examples of slices excluded from the TTA experiment. Theseslices have near zero median values, and the majority of points are zero or nearzero for each respective set of 200 augmented slices.105Appendix D. Additional Data for Test Time Augmentation ExperimentTable D.1: Mean structure IoUs for the for the first 35 groups of augmented CTslices.Slice # Heart IoU Left Lung IoU Right Lung IoUSlice 1 0.0594 2.53∗10−9 0.0846Slice 2 N/A 0.286 0.158Slice 3 N/A 0.377 0.389Slice 4 N/A 0.0211 0.0295Slice 5 0.99 0.205 0.269Slice 6 0.004 0.00164 0.0572Slice 7 N/A 0.492 0.467Slice 8 0.456 0.606 0.680Slice 9 N/A 0.456 0.579Slice 10 0.455 0.214 0.187Slice 11 N/A 0.0420 0.00953Slice 12 N/A 0.0957 0.0802Slice 13 0.0217 0.995 0.00707Slice 14 0.245 0.148 0.217Slice 15 0.375 0.297 0.372Slice 16 0.25 0.254 0.389Slice 17 0.995 0.103 0.0542Slice 18 N/A 0.0516 0.0159Slice 19 N/A 0.00249 0.00305Slice 20 N/A 0.0409 0.0116Slice 21 N/A 0.0993 0.140Slice 22 N/A 0.0184 0.0362Slice 23 N/A 1.84E-05 0.00476Slice 24 0.945 0.301 0.284Slice 25 N/A 0.354 0.313Slice 26 0.0609 0.995 0.99Slice 27 N/A 0.141 0.117Slice 28 0.105 0.364 0.540Slice 29 N/A 0.695 0.681Slice 30 N/A 0.364 0.253Slice 31 N/A 0.161 0.0449Slice 32 N/A 0.015 0.0364Slice 33 0.575 0.335 0.387Slice 34 N/A 0.342 0.444Slice 35 N/A 0.00646 0.0112106Appendix D. Additional Data for Test Time Augmentation ExperimentTable D.2: Mean structure IoUs for the for the second 35 groups of augmentedCT slices.Slice # Heart IoU Left Lung IoU Right Lung IoUSlice 36 N/A 2E-08 0.705Slice 37 0.995 0.383 0.403Slice 38 0.58 0.0650 0.245Slice 39 N/A 0.129 0.0411Slice 40 N/A 0.0327 0.0160Slice 41 N/A 0.0597 0.0894Slice 42 0.398 0.535 0.667Slice 43 0.95 0.135 0.366Slice 44 N/A 0.302 0.188Slice 45 N/A 0.176 0.0893Slice 46 N/A 1.02E-09 0.0469Slice 47 N/A 0.0162 0.0180Slice 48 N/A 0.678 0.663Slice 49 0.348 0.365 0.285Slice 50 N/A 0.542 0.569Slice 51 N/A 0.300 0.154Slice 52 N/A 0.505 0.613Slice 53 0.225 0.376 0.389Slice 54 N/A 0.161 0.247Slice 55 0.66 0.00971 1E-09Slice 56 N/A 0.535 0.555Slice 57 N/A 0.00616 0.00620Slice 58 N/A 2.8E-09 0.00548Slice 59 N/A 0.555 0.617Slice 60 N/A 0.00169 0.000147Slice 61 N/A 0.140 0.0916Slice 62 0.9 1.29E-09 1.23E-09Slice 63 0.101 0.275 0.0558Slice 64 0.765 0.0541 0.118Slice 65 N/A 0.698 0.696Slice 66 N/A 0.0373 0.0147Slice 67 0.138 0.506 0.365Slice 68 0.235 0.0468 0.112Slice 69 0.245 0.339 0.424Slice 70 N/A 0.625 0.755107
Thesis/Dissertation
eng
Medical Physics
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Rajapakshe, Rasika
Feldman, Rebecca
A machine learning approach to computed tomography organ segmentation for quality assurance
Text