Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Gupta, Anubhav
Master of Science - MSc
University of British Columbia
Medical imaging quality control is traditionally done through routine manual testing, which is time-consuming, error-prone, and costly. There has been limited exploration of automating this quality control process using deep learning techniques. Several factors contribute to this gap in research, including 1) the need for a specialized labelled dataset to evaluate this problem, 2) the scarcity of labelled medical imaging data for training state-of-the-art deep learning models capable of robust generalization on medical imaging classification tasks, and 3) the absence of a dedicated deep learning backbone pre-trained on medical imaging data capable of robust generalization for various medical image analysis tasks. This thesis first presents a study that evaluates state-of-the-art deep convolutional neural network models to automate the quality control process for medical images. By harnessing transfer learning, the goal is to enhance the availability and effectiveness of medical image diagnostic scanners for applications in patient diagnosis, treatment, research, and development. We evaluate the performance of 5 different models wherein InceptionResNetV2 achieves the best accuracy of 99%. However, when these models are tested on out-of-domain datasets, the results show a crucial need for more robust models capable of automating quality control within different healthcare facilities. In the second part of the thesis, we employ a self-supervised approach to pre-train a specialized backbone, referred to as MedMAE, using a diverse dataset comprising over two million publicly available medical images. Our results demonstrate that MedMAE effectively captures the unique characteristics of medical images, leading to improved accuracy in downstream medical imaging tasks and even on out-of-domain datasets. This innovative approach holds promise for enhancing the utility and efficiency of medical image analysis, bridging the gap caused by the scarcity of labelled data in the healthcare domain. Furthermore, to evaluate the effectiveness of MedMAE, we employ a Virtual Reality tool for precise manual annotation of discrepancies between the original medical images and the reconstructions produced by MedMAE.
Automating Computed TomographyDiagnostic Image Quality ControlUsing Deep LearningbyAnubhav GuptaB.E., BITS Pilani Dubai Campus, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe College of Graduate Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)December 2023© Anubhav Gupta, 2023The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Automating Computed Tomography Diagnostic Image Quality ControlUsing Deep Learningsubmitted by Anubhav Gupta in partial fulfilment of the requirements ofthe degree of Master of ScienceW. John Braun, The University of British Columbia, Okanagan CampusSupervisorMohamed S. Shehata, The University of British Columbia, Okanagan CampusCo-SupervisorPourang Irani, The University of British Columbia, Okanagan CampusSupervisory Committee MemberJohn Thompson, The University of British Columbia, Okanagan CampusSupervisory Committee MemberKlaske Van Heusden, The University of British Columbia, Okanagan CampusUniversity ExamineriiAbstractMedical imaging quality control is traditionally done through routinemanual testing, which is time-consuming, error-prone, and costly. Therehas been limited exploration of automating this quality control process us-ing deep learning techniques. Several factors contribute to this gap in re-search, including 1) the need for a specialized labelled dataset to evaluatethis problem, 2) the scarcity of labelled medical imaging data for trainingstate-of-the-art deep learning models capable of robust generalization onmedical imaging classification tasks, and 3) the absence of a dedicated deeplearning backbone pre-trained on medical imaging data capable of robustgeneralization for various medical image analysis tasks. This thesis firstpresents a study that evaluates state-of-the-art deep convolutional neuralnetwork models to automate the quality control process for medical im-ages. By harnessing transfer learning, the goal is to enhance the availabilityand effectiveness of medical image diagnostic scanners for applications inpatient diagnosis, treatment, research, and development. We evaluate theperformance of 5 different models wherein InceptionResNetV2 achieves thebest accuracy of 99%. However, when these models are tested on out-of-domain datasets, the results show a crucial need for more robust modelscapable of automating quality control within different healthcare facilities.In the second part of the thesis, we employ a self-supervised approach topre-train a specialized backbone, referred to as MedMAE, using a diversedataset comprising over two million publicly available medical images. Ourresults demonstrate that MedMAE effectively captures the unique charac-teristics of medical images, leading to improved accuracy in downstreammedical imaging tasks and even on out-of-domain datasets. This innovativeapproach holds promise for enhancing the utility and efficiency of medicalimage analysis, bridging the gap caused by the scarcity of labelled data inthe healthcare domain. Furthermore, to evaluate the effectiveness of Med-MAE, we employ a Virtual Reality tool for precise manual annotation ofdiscrepancies between the original medical images and the reconstructionsproduced by MedMAE.iiiLay SummaryComputed Tomography (CT scan) is a critical diagnostic tool routinelyused for diagnosis, treatment planning, and monitoring of disease progres-sion. The accuracy and timeliness of patient diagnosis on radiology scannersis a direct function of the quality of the images produced during a patientscan. The current manual quality control process is time-consuming andprone to errors. In this work, we explore the possibility of automatingthe quality control process using transfer learning-based deep convolutionalneural networks. While deep learning models can achieve impressive imageclassification accuracy when trained on large labelled datasets, the medicalimaging field faces challenges due to the scarcity of such datasets. Therefore,we present a novel self-supervised approach that leverages a diverse datasetof medical images for pre-training. This approach not only enhances ac-curacy in downstream tasks but also demonstrates the ability to generalizeeffectively, even when presented with datasets from different domains.ivPrefaceChapters 3 through 5 of this thesis incorporate contents from researchpublished or submitted to scientific journals and conferences, with Anub-hav Gupta serving as the primary author. Collaborators for these publishedor submitted works include Mohamed S. Shehata, W. John Braun, IslamOsman, Rebecca Emily Feldman, and Thorarin Albert Bjarnason. All pub-lished or accepted papers in this thesis work are used with the permissionof IEEE.Chapter 3 of this thesis incorporates content from the paper titled ‘Au-tomating CT Diagnostic Image Quality Control Using Transfer Learning,’which has been accepted for publication at the IEEE MoSCIom2023 confer-ence. Anubhav Gupta served as the lead investigator, taking on the rolesof implementing, training and testing all evaluated techniques. Addition-ally, Rebecca E. Feldman and Thorarin A. Bjarnason provided access andinsights on dataset curation, in addition to participating in manuscript re-views. Mohamed S. Shehata and W. John Braun supervised all researchand manuscript composition.Chapter 4 of this thesis incorporates content from “MedMAE: A Pre-Trained Masked Autoencoder for Medical Imaging Tasks,” which is currentlyin submission at IEEE Transactions on Biomedical Engineering. Anub-hav Gupta was the lead investigator responsible for formulating, training,and testing the proposed deep-learning backbone. Islam Osman was re-sponsible for helping with technique formulation and manuscript composi-tion. Mohamed S. Shehata and W. John Braun supervised all research andmanuscript composition.Other discoveries have led to publications not included in this thesis:− Seger Nelson, Krystyna Mylostna, Anubhav Gupta, Islam Osman,Thorain A Bjarnason, Mohamed Shehata, Erin L MacMillan, and Re-becca E Feldman, ‘Towards Real-Time RF Coil Failure RecognitionUsing Deep Transfer Learning,’ ISMRM 2024 (submitted).− Gupta, A., Campos, D., Ganeriwala, P., Bhattacharyya, S., OConnor,T., Dcosta, A. (2023). Modeling Internet-of-Things (IoT) Behavior forvEnforcing Security and Privacy Policies. In: Arai, K. (eds) IntelligentComputing. SAI 2023. Lecture Notes in Networks and Systems, vol711. Springer, Cham. https://doi.org/10.1007/978-3-031-37717-4 95.− Parth Ganeriwala, Siddhartha Bhattacharyya, Raja Muthalagu, ShreyaNandanwar, Anubhav Gupta, ‘IPAssess: A Protocol-Based Finger-printing Model for Device Identification in the IoT,’ IntelliSys Confer-ence 2023, Jun/2023.− Adithya T. Praveen, Anubhav Gupta, Siddhartha Bhattacharyya, RajaMuthalagu, ‘Assuring Behavior of Multirobot Autonomous Systemswith Translation from Formal Verification to ROS Simulation,’ IEEESystems Journal, Jan/2022.− Siddhartha Bhattacharyya, Jennifer Davis, Anubhav Gupta, NandithNarayan, Michael Matesa, ‘Assuring Increasingly Autonomous Sys-tems in Human-Machine Teams: An Urban Air Mobility Case Study,’FMAS 2021, Oct/2021.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . . xvAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xviDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xviiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Research Motivation . . . . . . . . . . . . . . . . . . . . . . . 41.3 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Contributions Summary . . . . . . . . . . . . . . . . . . . . . 51.5 Published and Submitted Work . . . . . . . . . . . . . . . . . 61.6 Ethics Approval . . . . . . . . . . . . . . . . . . . . . . . . . . 61.7 Thesis Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.8 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 7Chapter 2: Background and Related Work . . . . . . . . . . . 82.1 Automating CT Diagnostic Image QC . . . . . . . . . . . . . 82.1.1 Problem Definition . . . . . . . . . . . . . . . . . . . . 82.1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . 10vii2.2 2.2.1 2.2.2 2.3 3.1 3.1.1 3.1.2 3.1.3 3.1.4 3.2 3.2.1 3.2.2 3.2.3 3.3 4.1 4.1.1 4.1.2 4.1.3 4.2 4.2.1 4.2.2 4.3 4.3.1 4.3.2 4.3.3 4.4 4.5 List of TablesTable 3.1 An overview of CT QC logs logged by a specialist atthe IHA facility. Scans collected on ‘Day-1’ fall withinthe required specification. However, the Noise (S.D.)in the scans collected on ‘Day-2’ is below the mini-mum threshold value of 4.0. Therefore, the scanneris deemed out-of-calibration on ‘Day-2’ and any scanscollected on that day will be categorized and labelledas miscalibrated. . . . . . . . . . . . . . . . . . . . . . 21Table 3.2 Comprehensive overview of the CT diagnostic QC datasetsdeveloped for comparative evaluation. Images obtainedfrom calibrated scanners are designated as ‘Pass’, whilethose from miscalibrated scanners are labelled as ‘Fail’.The dataset from Hospital-A comprises a total of 14,653images (over 100 patient studies), randomly split into8,120 Pass and 6,533 Fail images. Contrastingly, thedataset from Hospital-B includes a larger set of 38,217images (over 250 patient studies), randomly dividedinto 16,605 Pass and 21,612 Fail images. . . . . . . . 24Table 3.3 Training accuracy results using five different state-of-the-art CNN models. Each model was independentlytrained on the ImageNet training set and fine-tunedon the CT diagnostic image dataset1. As observed,InceptionResNetV2 achieves the highest accuracy onthe training and validation set. . . . . . . . . . . . . . 29ixLIST OF TABLESTable 3.4 Performance evaluation of five state-of-the-art CNNModels on CT diagnostic imaging. Each model un-derwent initial training on the ImageNet dataset, fol-lowed by fine-tuning using the CT diagnostic imagetraining dataset1. The independent test set comprises2,931 images, unseen by the models during the trainingphase. InceptionResNetV2 stands out, demonstratingsuperior accuracy on the test set. . . . . . . . . . . . 29Table 3.5 Training accuracy results using five different state-of-the-art CNN models. Each model was independentlytrained on the ImageNet training set and fine-tunedon the CT diagnostic image dataset2. As observed,InceptionResNetV2 achieves the highest accuracy onthe training and validation set. . . . . . . . . . . . . . 30Table 3.6 Performance evaluation of five state-of-the-art CNNModels on CT diagnostic imaging. Each model un-derwent initial training on the ImageNet dataset, fol-lowed by fine-tuning using the CT diagnostic imagetraining dataset2. The independent test set comprises7,640 images, unseen by the models during the trainingphase. InceptionResNetV2 stands out, demonstratingsuperior accuracy on the test set. . . . . . . . . . . . 31Table 3.7 Cross-Evaluation of top three state-of-the-art CNNMod-els for out-of-domain performance on CT diagnosticimaging. Following initial training on the ImageNetdataset, each model was fine-tuned using either CTdiagnostic image training dataset1 or dataset2. To as-sess robustness, models fine-tuned on dataset1 weretested using the test set from dataset2, and vice versa. 32Table 4.1 A tabular list detailing the sources from which themedical imaging data was sourced, along with theircorresponding web links for reference and access. Allthe sources are openly accessible for research purposes. 38Table 4.2 This table provides a detailed overview of the variousdatasets utilized for the pretraining process. It in-cludes information on the specific body part targetedby each dataset, the number of patient studies or sub-jects included, as well as the type of medical imageeach dataset comprises. . . . . . . . . . . . . . . . . . 41xLIST OF TABLESTable 4.3 This table presents a detailed summary of ‘dataset3’,created to assess the effectiveness of MedMAE-F throughdownstream linear probing and fine-tuning tasks. Thedataset comprises CT diagnostic images, sourced di-rectly from the PACS of the IHA. It includes a va-riety of procedures, each representing a specific bodypart. For each listed procedure, the dataset incorpo-rates both Pass and Fail patient studies. . . . . . . . 53Table 4.4 MedMAE-F out-of-domain linear probing results for‘dataset3’. . . . . . . . . . . . . . . . . . . . . . . . . 54Table 4.5 MedMAE-F end-to-end fine-tuning results for ‘dataset3’.55Table 4.6 Comparative results on ‘dataset3’. . . . . . . . . . . . 56xiList of FiguresFigure 1.1 Progression of ImageNet [24] Classification AccuracyOver Time: Notable milestones in image classificationaccuracy, marked with labelled points, showcase theadvancements in deep learning models . . . . . . . . . 3Figure 2.1 An illustrative example of axial head CT scans: (a)depicts an axial head CT scan obtained from a cal-ibrated scanner, while (b) shows an axial head CTscan from a miscalibrated scanner. Notably, distin-guishing between the two scans is challenging for thehuman eye. . . . . . . . . . . . . . . . . . . . . . . . . 9Figure 2.2 Progression of Self-Supervised Learning Models: No-table milestones in SSL classification accuracies, markedwith labelled points, showcase the advancements inSSL models. . . . . . . . . . . . . . . . . . . . . . . . 14Figure 3.1 An illustrative example of the transfer learning pro-cess flow used in our study. In phase (a), we cu-rate various CT QC diagnostic target datasets, whichserve as the foundation for fine-tuning our pre-trainedmodels. This initial phase is crucial for adapting themodels to the specific requirements of medical QCimaging tasks. Subsequently, in phase (b), we transi-tion to the selection of five state-of-the-art models forautomated QC evaluation. The models under consid-eration are (1) VGG19, (2) ResNet50, (3) Inception-ResNetV2, (4) XceptionNet, and (5) EfficientNetB7. . 20xiiLIST OF FIGURESFigure 3.2 An illustrative example of axial head CT scans. (a)scan from a calibrated scanner at Hospital-A; (b) scanfrom a calibrated scanner at Hospital-B; (c) scan froma miscalibrated scanner at Hospital-A; and (d) scanfrom a miscalibrated scanner at Hospital-B. As evi-dent from the illustration, distinguishing scans origi-nating from a calibrated scanner from those producedby a miscalibrated scanner poses a significant challenge. 23Figure 4.1 Novel MedMAE-B architecture: The process is initi-ated by randomly masking 75% of the original imageand inputting the remaining 25% of visible patchesinto the encoder, which captures the latent represen-tations and encodes the patches. Subsequently, theaim of the decoder is to reconstruct the complete im-age using the encoded and masked patches. The re-construction loss helps to improve the reconstructionwith each iteration. . . . . . . . . . . . . . . . . . . . 37Figure 4.2 MedMAE-B Training Workflow: ViT-B encoder istrained on a huge dataset of medical images for over1000 epochs. The model, after training on this pre-text task, is referred to as MedMAE-B. . . . . . . . . 43Figure 4.3 Training Loss Tensorboard Graph: Progression of lossvalues during the extensive training of our MedMAE-B model over 1000 epochs, each comprising 1000 steps.The Y-axis represents the train loss magnitude, whilethe X-axis denotes the number of training steps. . . . 45Figure 4.4 The figure demonstrates the initial reconstruction ca-pabilities of our MedMAE-B model at checkpoint 0on an in-domain medical image. . . . . . . . . . . . . 46Figure 4.5 The figure demonstrates the reconstruction capabil-ities of our MedMAE-B model at checkpoint 999 onan in-domain medical image. . . . . . . . . . . . . . . 46Figure 4.6 The figure demonstrates the initial reconstruction ca-pabilities of our MedMAE-B model at checkpoint 0on an out-of-domain medical image. . . . . . . . . . . 47Figure 4.7 The figure demonstrates the reconstruction capabil-ities of our MedMAE-B model at checkpoint 999 onan out-of-domain medical image. . . . . . . . . . . . . 47xiiiLIST OF FIGURESFigure 4.8 Novel MedMAE-F Architecture: MedMAE-B’s de-coder is omitted, and a linear classifier is attached andtrained to predict outcomes for downstream medicalimaging tasks. . . . . . . . . . . . . . . . . . . . . . . 49Figure 4.9 Linear Probing Architecture: The decoder is discarded,and the encoder is frozen with the pre-trained weights.A new classification header is added to the encoder.Only the added classifier is trained on the downsam-pling task. . . . . . . . . . . . . . . . . . . . . . . . . 50Figure 4.10 End-to-end Fine-Tuning Architecture: The decoderis discarded, and the encoder is initialized with thepre-trained weights. A new classification header isadded to the encoder. The entire network is trainedon the downsampling task. . . . . . . . . . . . . . . . 51Figure 4.11 Medical Image Visualization in Virtual Reality: Dis-played within the Unity environment, the VR toolallows immersive interaction with medical images, en-abling detailed visualization and annotation. . . . . . 58xivGlossary of NotationAI ............... Artificial IntelligenceAQS ............ Advanced Quality SystemsCNN ........... Convolutional Neural NetworkCT ............. Computer TomographyDL ............... Deep LearningDTL ............ Deep Transfer LearningIHA ............. Interior Health AuthorityMAE ........... Masked AutoencoderMRI ........... Magnetic Resonance ImagingNLP ............ Natural Language ProcessingPACS .......... Picture Archiving and Communication SystemsQC ............. Quality ControlSSL ............. Self-Supervised LearningViT ............. Vision TransformerVR .............. Virtual RealityxvAcknowledgementsFirst and foremost, I would like to express my sincerest gratitude tomy supervisor and co-supervisor, Dr. W. John Braun and Dr. MohamedS. Shehata, for their invaluable guidance, unwavering support, and mentor-ship throughout my research journey. Their expertise and dedication havebeen instrumental in shaping this thesis. I would also like to extend myappreciation to Dr. Islam Osman and Reece Walsh for their collaborationin technique formulation and analysis, as well as their helpful discussionsand ideas. This thesis would not have been possible without your collectivesupport and guidance.xviDedicationTo my Ma, Papa and Bhai (Bhawna, Pankaj and Aditya), whose un-wavering love, support, and sacrifices have been my constant source of in-spiration throughout this academic journey. Your belief in me has fueledmy determination to reach this milestone. To my maternal grandmother(Santosh), your unconditional love has helped me throughout this journey.This thesis is dedicated to you with profound gratitude.xviiChapter 1Introduction1.1 OverviewCanadian healthcare outcomes are often closely tied to the precision ofmedical imaging, reliant on the quality of images produced by diagnosticdevices like Magnetic Resonance Imaging (MRI) and Computed Tomogra-phy (CT) scanners. CT plays a key role in modern medical diagnostics,facilitating disease diagnosis, treatment planning, and monitoring diseaseprogression. Its effectiveness has been highlighted, especially when evaluat-ing the widespread impacts of COVID-19 on vital organs such as the lungsand brain [101], [75], [37]. The efficacy of patient diagnosis is directly linkedto the image quality produced during a scan. Poor-quality images not onlyjeopardize the accuracy of diagnoses but also introduce potential risks andinefficiencies. An image from a miscalibrated scanner necessitates a repeatscan, incurring additional costs and time. Moreover, it can result in misseddiagnoses that can be potentially fatal.Miscalibration of scanners is a leading cause of poor-quality clinical im-ages. While these scanners may still produce images that apparently seemof diagnostic grade, the finer details could be compromised. For instance,air bubble artifacts can mistakenly appear as cerebral infarcts [34] in CTand image shading in MRI [77]. To mitigate such issues, radiology depart-ments undertake periodic quality control (QC) procedures. During these QCprocedures, the scanner is taken offline to capture images of a phantom, awater-filled plastic patient substitute. Subsequently, these phantom imagesundergo a time-consuming manual assessment to ascertain if the scanner isfunctioning within its defined specifications [25], [19]. Notably, executing aQC protocol for a CT scanner can consume between 15 to 20 minutes perevaluation, which is equivalent to 1 or 2 patient scans.At all Interior Health sites in British Columbia, a periodic QC processfor CT scanners is maintained. For CT scanners, routine QC proceduresare conducted weekly, with each QC session lasting approximately 20 min-utes. The QC records are preserved, with a decade’s worth of data for CTscanners. CT QC involves phantom scanning on a weekly basis, ensuring11.1. OVERVIEWongoing accuracy and performance. Moreover, CT modalities receive rou-tine preventative maintenance to ensure optimal functionality. Additionally,CT scanners undergo in-depth physics QC assessments twice a year.Several efforts have been made to automate certain aspects of the QCprocess. These include the monitoring of key performance indicators likeCT numbers [64], CT image noise [76], and automatic artifact detection[98]. Automating the generation, storage, and tracking of these indicators,can save up to 5 minutes in the phantom scanning process alone. Substantialadvances have been made in automating artifact detection using learning ap-proaches [100][11]. However, no research to date has looked into automatingthe QC process, through the analysis of clinical images. International re-search groups have increasingly turned to machine learning tools, includingdeep learning and convolutional neural networks (CNNs) ([51], [86], [43]), toexpedite the diagnosis and treatment of several diseases such as COVID-19and enhance our understanding of the effects of the disease. As the drive todevelop AI-based diagnostic solutions intensifies, the significance of preciseand consistent high-quality medical images becomes even more necessary.AI-driven systems heavily depend on the quality and accuracy of the datathey are trained on. Utilizing machine learning models with inadequate,inaccurate, or miscalibrated data can result in inconsistent and potentiallyerroneous outcomes.Image classification is a fundamental task in computer vision where thegoal is to assign a label or category to an input image. CNNs have revolution-ized image classification due to their ability to automatically learn hierarchi-cal features from raw pixel data. Figure 1.1 depicts different CNN modelsover a period of time that have achieved notable milestones in image classi-fication accuracy. The VGG architectures, especially VGG16 and VGG19,designed by Simonyan and Zisserman, were known for their deep networkdesign with small 3x3 convolutional filters [85]. Residual Networks, intro-duced by He et al., utilized residual connections to train very deep CNNseffectively [33]. The Inception family of architectures, including Inception-v3 and Inception-ResNet-V2, introduced the concept of inception modulesand achieved top accuracy in image classification tasks [91]. The key inno-vation in Xception is the use of depthwise separable convolutions, which aresplit into depthwise convolution and pointwise convolution [17]. This separa-tion results in a more efficient network with fewer parameters. EfficientNet,proposed by Tan and Le, introduced a family of CNN architectures witha novel scaling method, demonstrating high efficiency and accuracy [96].After 2020, the research focus shifted towards transformers. A few modelslooked at combining transformer architecture with traditional CNN archi-21.1. OVERVIEWtecture to achieve high accuracies. For our work, we focus on traditionalstate-of-the-art CNNs, which have been highlighted in red in Figure 1.1.Figure 1.1: Progression of ImageNet [24] Classification Accuracy Over Time:Notable milestones in image classification accuracy, marked with labelledpoints, showcase the advancements in deep learning models. Models suchas VGG19 [85], ResNet50 [33], InceptionResNetV2 [91], XceptionNet [17],EfficientNetB7 [96], EfficientNetB8 [20], VOLO-D5 [103], MAE [31] andRevCol-H [10] have significantly contributed to the field.While CNNs have attained remarkable state-of-the-art performance, theinherent challenge of data dependence remains a prominent issue in deeplearning (DL). DL models often demand extensive volumes of training datato understand latent data patterns and achieve their potential [95]. How-ever, the field of medical imaging faces a significant hurdle, insufficient datafor effectively training DL models. This scarcity arises from the necessity ofmanual labelling by clinical experts [4]. Therefore, training a CNN modelfrom scratch doesn’t lead to the desired results in the medical imaging do-main. In order to overcome this challenge, deep transfer learning (DTL)has garnered substantial interest among researchers as it seeks to mitigate31.2. RESEARCH MOTIVATIONthe need for vast amounts of labelled data [36]. DTL often involves a CNNmodel, which is pre-trained on a large dataset, e.g., ImageNet. After pre-training, the knowledge (pre-trained weights) is transferred to downstreammedical image classification tasks. However, existing research highlights anoteworthy challenge: the performance of these pre-trained models tends todeteriorate when applied to medical images such as chest X-rays and brainMRI scans. This decline is attributed to the fact that these models havenot been originally trained on medical grayscale images [16], leading to amismatch between the features learned from natural images (ImageNet) andthose more specific to medical images such as X-rays. Consequently, there isa pressing need for the development of a dedicated DL backbone trained onmedical images. Such a specialized backbone can ensure that, when appliedto tasks with smaller medical datasets, performance levels are effectivelyretained. In an effort to solve the aforementioned issues, active research en-deavours have been dedicated to crafting adaptable models tailored for usewith medical imaging datasets. Nevertheless, to the best of our knowledge,no such dedicated backbone currently exists within the research communitycapable of serving as a foundational tool for downstream medical imagingapplications.1.2 Research MotivationThis research was motivated by the notable absence of automated so-lutions for CT diagnostic image QC. Recognizing this critical gap, we em-barked on a journey to address the automation of CT diagnostic image QCprocedures. Additionally, we identified a pressing need for a DL-based back-bone that could be pre-trained on medical imaging data to capture the inher-ent characteristics of these images. This thesis introduces a self-supervisedbackbone designed to be adaptable to both in-domain and out-of-domainsettings for healthcare applications. By undertaking this endeavour, ouraim is to make a significant contribution to the field and encourage furtherexploration and research in this promising direction.1.3 Research ObjectiveThe primary goal of this research is to assess state-of-the-art deep neu-ral networks for automating the QC process and develop a novel, robusttechnique capable of effective generalization in out-of-domain settings. Toachieve this overarching goal, we concentrate on the following objectives:41.4. CONTRIBUTIONS SUMMARY1. Our initial focus is on automating the QC process for CT diagnosticimages. We propose a novel approach using DTL, which involves theevaluation of five different DL models. Among these models, Incep-tionResNetV2 stands out as the top performer, achieving an in-domainaccuracy rate of 99% in binary diagnostic CT image classification. Tofacilitate this evaluation, we curate two specialized datasets, rigorouslytest the models, and even subject them to out-of-domain evaluationto assess their robustness and generalization capabilities. The findingshighlight the need for a robust medical backbone adaptable to diversesettings.2. In response to this challenge, we introduce MedMAE, a self-supervisedbackbone designed for medical imaging applications. MedMAE ad-dresses the issue of data dependence by significantly reducing the re-liance on extensive annotated data for medical image tasks. As it hasbeen pre-trained on diverse medical imaging datasets, it is aimed tobe adaptable and generalizable, making it suitable for a wide rangeof medical imaging applications. To assess the efficacy of MedMAE,we fine-tune the model using CT diagnostic image QC datasets, whichresults in improved accuracy, even in diverse scenarios. Our overar-ching objective is to contribute to the research community by makingMedMAE publicly accessible as a foundational tool for future advance-ments in the field of medical imaging.1.4 Contributions SummaryTo summarize, the main contributions of this thesis are as follows:1. This thesis evaluates deep transfer learning as a tool for automatingthe quality control process for CT imaging. We utilize two specializedmedical QC datasets for training and testing, sourced from Hospital-Aand Hospital-B.2. This thesis presents a comparative performance analysis of state-of-the-art deep learning models, including VGG19, ResNet50, Inception-ResNetV2, XceptionNet and EfficientNetB7. After comparison, anout-of-domain assessment is performed to understand the versatilityand robustness of different models.3. This thesis proposes MedMAE, a novel self-supervised Masked Au-toencoder (MAE) backbone model pre-trained on a large publicly51.5. PUBLISHED AND SUBMITTED WORKavailable diverse medical imaging dataset for downstream medical imag-ing applications.4. This thesis presents a case study evaluation of the proposed self-supervised technique showcasing improved performance in both in-domain and out-of-domain settings.1.5 Published and Submitted WorkThis thesis is based on the following published or submitted works:1. Automating CT Diagnostic Image Quality Control Using TransferLearning, Anubhav Gupta, Mohamed S. Shehata, W. John Braun,Rebecca Emily Feldman, and Thorain Albert Bjarnason, IEEE Inter-national Conference on Modelling Simulation & Intelligent Computing(MoSICom), 2023 (accepted)2. Towards Real-Time RF Coil Failure Recognition Using Deep TransferLearning, Seger Nelson, Krystyna Mylostna, Anubhav Gupta, IslamOsman, Thorain A Bjarnason, Mohamed Shehata, Erin L MacMillan,and Rebecca E Feldman, International Society for Magnetic Resonancein Medicine (ISMRM), 2024 (in submission).3. MedMAE: A Masked AutoEncoder for Medical Image Analysis, Anub-hav Gupta, Islam Osman, Mohamed S. Shehata, and W. John Braun,IEEE Transactions on Biomedical Engineering, 2023 (in submission)1.6 Ethics ApprovalThis study was conducted in strict adherence to ethical standards andwas initiated only after securing the necessary approvals from the relevantethics boards. Furthermore, the collection of images from the Interior HealthAuthority’s Picture Archiving and Communication System (PACS) was per-formed with the full approval of the clinical research ethics board. TheCREB approval code for this study is H20-00487.1.7 Thesis ScopeThis thesis is inspired by a project named ‘Automated Medical Diag-nostic Image Quality Control Using AI-Based Techniques,’ which is sup-ported by the Natural Sciences and Engineering Research Council of Canada61.8. THESIS OUTLINE(NSERC) under the Alliance-Mission mechanism. The primary goal of theproject is to develop and implement automated QC protocols for CT andMRI diagnostic machines. This thesis specifically concentrates on a subsetof that objective, focusing on the automation of CT QC through the appli-cation of DTL. Additionally, this thesis aims to design a medical imagingframework optimized for broad applicability across various medical imagingclassification tasks, thereby establishing a versatile backbone for medicalimage analysis.1.8 Thesis OutlineThe subsequent sections of this thesis are structured as follows: Chap-ter 2 provides a comprehensive background, reviewing prior work related tothe conducted experiments and proposed techniques. Chapter 3 delves intothe examination of five contemporary CNN techniques, focusing on theirpotential for automating CT diagnostic image QC. Chapter 4 introducesMedMAE, a novel approach designed to enhance generalization in down-stream medical image applications. Chapter 4 also offers a Virtual Reality(VR) assessment of the proposed self-supervised backbone. Lastly, Chap-ter 5 serves as the conclusion of this thesis, summarizing key findings, andsuggesting promising avenues for future research in this domain.7Chapter 2Background and RelatedWork2.1 Automating CT Diagnostic Image QC2.1.1 Problem DefinitionIn the evolving landscape of healthcare, the role of CT and MRI imaginghas become even more important. The emergence of respiratory diseases,most notably the global COVID-19 pandemic, has highlighted the criticalrole played by CT and MRI scans in diagnostics. These imaging modali-ties are of immense value in understanding the impact of various infectionson the human body, facilitating a deeper understanding of disease progres-sion, and aiding in the formulation of effective treatment strategies. Theprecision and promptness of patient diagnoses conducted on radiology scan-ners are inherently linked to the quality of the images generated duringthe scanning process. When the quality of these images is compromised,it can have dire consequences. An image obtained from a mis-calibratedscanner can necessitate a costly and time-consuming repetition of the pa-tient scan. In more critical scenarios, if such issues go undetected, they cangive rise to missed diagnoses, posing substantial risks to patient health and,in certain instances, even resulting in tragic outcomes, including loss of life[23], [71], [87], [80]. Moreover, it is critical to note that scanner faults canoccasionally be so subtle that they can go undetected by the human eyeas well [62]. Figure 2.1 provides a visual comparison of diagnostic imagesobtained from both a calibrated and a mis-calibrated scanner. Therefore,to ensure the reliability of radiology scanners, quality assurance protocolsare routinely implemented, typically on a daily or weekly basis [57]. Theseprotocols involve the use of a standardized phantom for imaging, allowingassessment of system performance. While specific procedures may vary fromone healthcare facility to another, various metrics are calculated from thephantom image, including crucial parameters such as signal-to-noise ratio.Any significant deviation in the measured value can help detect scanner82.1. AUTOMATING CT DIAGNOSTIC IMAGE QCFigure 2.1: An illustrative example of axial head CT scans: (a) depicts anaxial head CT scan obtained from a calibrated scanner, while (b) shows anaxial head CT scan from a miscalibrated scanner. Notably, distinguishingbetween the two scans is challenging for the human eye.miscalibration, resulting in prompt action. Daily quality assurance proce-dures, while essential, might not be able to evaluate all possible aspects ofa machine’s functionality. Additionally, they may fail to detect intermittentor subtle faults that can occasionally occur [73]. Furthermore, it’s worthnoting that these procedures are time-consuming and necessitate taking thescanner offline during the assessment process.Providing real-time QC of the scanner calibration to radiologists canhave several benefits, as listed:1. Improved patient throughput and reduced wait times, primarily byminimizing scanner downtime2. Enhanced diagnostic accuracy by swiftly identifying and addressingany image quality issuesTo the best of our knowledge, no research exists that proposes automatingQC by utilizing Artificial Intelligence (AI)-based techniques. This thesis92.1. AUTOMATING CT DIAGNOSTIC IMAGE QCstudy is distinct in its approach. Instead of focusing solely on phantom im-ages, we delve into the analysis of patients’ scans. Analyzing patient studiesallows us to understand the inherent characteristic distinctions between im-ages acquired from calibrated and miscalibrated scanners, representing anovel research work in the medical imaging domain.2.1.2 Related WorkConvolutional Neural Networks for Medical Image AnalysisConvolutional neural networks (CNNs) have proven to be highly effec-tive tools for image analysis and understanding, especially when it comesto detecting anomalies within the human body. In numerous image under-standing tasks, CNNs have demonstrated superior performance even whencompared to human experts, as highlighted in a recent study by Sarvaman-gala et al [83]. The CNN model comprises convolutional filters that serve asfundamental components for learning and extracting crucial features essen-tial for effective medical image analysis and classification. In medical imag-ing, CNNs achieved a significant breakthrough when GoogleNet [92] wasemployed to detect cancer with an accuracy of 89%, surpassing the accuracyof human pathologists, who achieved only 70% accuracy [79]. Since then,researchers have achieved remarkable success by applying CNNs to variousmedical image analysis tasks. These include the detection and classificationof tumours as benign or malignant [5], skin lesion detection [61], identifica-tion of anomalies in optical coherence tomography images [45], colon cancerdetection [61], detection of blood cancer, heart abnormalities [48], breastabnormalities [42], chest conditions, and eye-related issues. Notably, CNN-based models like CheXNet [74, 78], designed for classifying 14 differentchest ailments, have achieved superior performance compared to the aver-age performance of human experts. CNN has emerged as a dominant tool inthe field of COVID-19 detection, particularly when applied to the analysisof chest X-rays and CT scans. Its efficacy in accurately identifying COVID-19-related abnormalities has proven to be instrumental in healthcare effortsduring the pandemic [55], [28], [18], [30], [9].Deep Transfer Learning for Medical Image AnalysisDTL is a technique in machine learning where a model developed for aspecific task is reused as the starting point for a model on a second task. Itis a popular approach in deep learning where pre-trained models are usedas the starting point for computer vision tasks, especially medical image102.1. AUTOMATING CT DIAGNOSTIC IMAGE QCanalysis. The process of generating or documenting well-annotated data isboth resource-intensive and time-consuming. The use of DTL in conjunctionwith CNN models aims to alleviate the demand for extensive labelled data,resulting in significant time and hardware resource savings [47]. Severalcontemporary DTL approaches, predominantly network-based, have beenmaking waves in various domains where: (i) A pre-trained model is fine-tuned on target data; and (ii) The initial layers of the Convolutional NeuralNetwork (CNN) model are frozen, and only the lateral full-connected layersare fine-tuned. [49] introduces an innovative DTL approach that fine-tunesthe ResNet50 model, enabling it to classify malware images into their cor-responding malware families accurately. The study in [1] puts forth a novelhuman facial emotion recognition methodology. This supervised techniquefine-tunes a pre-trained deep CNN on the facial emotion dataset to identifyhuman emotions from facial images. Similarly, [81] explores a method wherea deep CNN is fine-tuned to tackle challenges in art classification.In the domain of medical imaging, there is a dearth of adequately labelleddata. Given this challenge, employing transfer learning for medical imageanalysis has yielded significant advances [59]. [2] delves into the classificationof analyzable metaphase images using DTL. [49] explores the classification ofendoscopic colonoscopy images through DTL. Authors in [94] highlight theautomated detection of brain abnormalities with DTL. The study in [104]proposes an automated method for ICD-9 coding utilizing a DTL frame-work. A framework for the automatic classification of pressure injuries isput forth in [7]. Both [22] and [102] illustrate DTL’s capability, focusingon brain tumour classification and the modality classification of medical im-ages, respectively. Radiography and CT imaging have emerged as potentialrapid diagnostic tools for early detection of COVID-19. Numerous studieshave used DTL to automate and expedite the detection of COVID-19, asdescribed in [21], [58], [72], [38], [3].Automating Diagnostic Image Quality ControlAs previously mentioned, CNNs consistently deliver high accuracy inthe realm of image classification tasks, making them a suitable option forconducting such studies. Several studies have implemented CNN models forartifact detection. The study conducted by Fantini et. al in [27] introduces adeep CNN designed for the automatic detection of motion artifacts in brainimages. In [82], the authors introduce QC-Automator, a DL-based auto-mated QC tool designed for dMRI data. This tool is capable of effectivelyaddressing a range of artifacts, including motion, multiband interleaving,112.2. SELF-SUPERVISED TECHNIQUESghosting, susceptibility, herringbone, and chemical shifts. In [90], an au-tomated tool for assessing the quality of multicenter structural brain MRIimages is introduced. This tool employs an ensemble DL model based ondeep CNNs for its evaluation. The primary limitation associated with CNNslies in their substantial data requirements for training. While there are var-ious freely available MRI and CT databases, they may not always alignprecisely with the specific MRI acquisition details under investigation in thestudy. Therefore, researchers have utilized artificial data to detect varioustypes of distortions using CNN models with considerable performance accu-racy [40], [29], [68]. In a very recent study by Tracey et al. [97], syntheticdata is employed to train a CNN model for automating QC in MRI. Theirproof-of-concept focuses on detecting coil element failure within an anteriorbody coil array. While this study shares similarities with our research objec-tives, there are notable distinctions. Firstly, Tracey et al.’s work exclusivelyassesses fault detection in MR scanners, whereas our study concentrates onidentifying faults within CT scanners. Secondly, they utilize synthetic datafor CNN model training, while our approach relies on real patient imagesfor both training and assessment. Furthermore, their training dataset onlyincludes axial localizer sequences, which may not encompass all the inherentimage characteristics necessary for distinguishing between images from MRIcoils with and without coil faults.To the best of our knowledge, there is no existing literature that looks atautomating CT medical diagnostic image quality control using DL tech-niques. Our approach diverges from conventional methods by utilizingdata from clinical patient scans, possibly eliminating the need for regularphantom-based QC. Our goal is to evaluate the possibility of an automatedtool that continuously monitors and analyzes standard patient clinical im-ages in real-time. This approach would enable instantaneous assessment ofthe CT scanner’s quality and calibration. An automated system like thiscould offer numerous benefits, such as enhanced patient throughput and im-proved diagnostic accuracy for both healthcare professionals and patients.2.2 Self-Supervised Techniques2.2.1 Problem DefinitionDL, particularly CNNs, has led to groundbreaking advancements in im-age understanding tasks, including object detection, image classification,and image segmentation. Yet, the effectiveness of these models in imagerecognition depends on supervised learning, which necessitates an extensive122.2. SELF-SUPERVISED TECHNIQUEScollection of datasets that have human-annotated labels [66]. The require-ment for large-scale labelled datasets is a significant challenge, especiallyin fields like medical imaging. The lack of open-access, high-quality anno-tated medical imaging datasets significantly impacts progress in applyingmachine learning techniques to medical image analysis [84]. While trans-fer learning strategies do help, they do not guarantee the development ofrobust and resilient models that are capable of delivering consistent perfor-mance across a range of diverse settings, especially in the medical imagingdomain [4]. In order to mitigate the costly task of accumulating labelleddata Self-Supervised Learning (SSL) stands out as a beneficial and efficientalternative to conventional supervised approaches. Moreover, SSL can beeffectively leveraged in domains where very few standard pre-trained mod-els exist, such as the medical imaging domain. SSL is a recent trainingtechnique where a model is trained to predict parts of the input data fromother parts of the same input, eliminating the need for human annotationsor labelled data [15]. SSL, as the name suggests, leverages its capacity toself-train using unlabelled data, enabling the development of robust repre-sentations. This characteristic is particularly beneficial for medical imagingapplications, where there is a scarcity of labelled data. By mitigating thedependence on extensive supervised datasets, SSL presents a viable solution,ensuring that high-quality models can still be trained and deployed even indata-constrained scenarios. One of the primary goals of this thesis is toaddress the pressing demand for a pre-trained model tailored specifically formedical imaging applications. The adoption of a SSL based backbone forthis purpose brings with it numerous advantages, outlined as follows:1. Reduced Dependency on Labelled Data: A model, adapted tomedical imaging needs, can be pre-trained without requiring vast vol-umes of annotated datasets.2. Enhanced Learning of Medical Image Characteristics: By train-ing on a pre-text task, the model is compelled to capture and under-stand the inherent characteristics of medical images. This process willhelp develop a model that is not only robust and resilient but alsohighly generalizable. As a result, when the model is subsequently fine-tuned for specific medical imaging applications, it might be able todemonstrate superior performance and adaptability.132.2. SELF-SUPERVISED TECHNIQUESFigure 2.2: Progression of Self-Supervised Learning Models: Notable mile-stones in SSL classification accuracies, marked with labelled points, showcasethe advancements in SSL models. Models such as DINO-V1 [12], MocoV3[32], BEiT [8], MAE [31] and I-JEPA [6] have significantly contributed tothe domain of SSL.2.2.2 Related WorkSelf-Supervised TechniquesSelf-Supervised Learning (SSL) techniques have garnered significant at-tention due to their ability to learn generic features without requiring la-belled data. These methods focus on solving one or multiple pretext tasks,enabling models to learn useful representations while mitigating over-fittingto domain-specific biases. The success in NLP through SSL and the emer-gence of Vision Transformers (ViTs) [26] have paved the way for innovativeapproaches in computer vision. Particularly for self-supervised image classi-fication, ViT-based self-distillation models [12], [50] and masked autoencod-ing methods [31], [106] are demonstrating performance levels comparable tosupervised learning methods. Figure 2.2 illustrates the significant stridesmade in image understanding tasks through the development of SSL mod-els. One of the very initial distillation techniques, DINO-V1, demonstrated142.2. SELF-SUPERVISED TECHNIQUESpromising results with an accuracy rate of approximately 82.8% on ImageNetdata. In the subsequent period, various techniques were introduced, eachcontributing to enhancements in performance accuracy. A notable landmarkin this progression was the introduction of Masked Autoencoder (MAE), aViT-based self-supervised approach, which set a new benchmark by attain-ing an impressive accuracy of 87.8%. Figure 2.2 effectively captures theevolution and impactful advancements in the field of SSL models for imageunderstanding tasks. Given that ViTs share architectural similarities withNLP models, reconstructive SSL techniques, which have proven highly effec-tive in NLP, are now being successfully applied to image-based tasks. Thesemethods involve predicting certain parts of the input data using the remain-ing parts, promoting the learning of rich and robust feature representations[31]. As a result, they enable models to capture the underlying structure ofthe visual data, enhancing their capability to perform image classificationand other computer vision tasks with high accuracy, even in the absence oflabelled data [67].Self-Supervised learning for medical image analysisSSL has emerged as a popular learning methodology in medical imageanalysis, particularly beneficial in scenarios where annotated data is scarce,and there is an abundance of unlabeled data [35]. Several researches havedemonstrated the effectiveness of the SSL approach throughout various med-ical image analysis tasks such as detection and classification ([89], [54], [52]),detection and localization ([14], [63], [88]), and segmentation tasks ([99],[13], [93]). Huang et. al. provide an extensive overview of DL approachesemploying SSL for medical image classification in [35]. Our work primar-ily concentrates on self-prediction methods, which aligns with our strategyto create an SSL-based backbone tailored for medical imaging applications.Self-prediction involves augmenting or masking certain segments of an im-age, followed by an attempt to reconstruct the original image using theremaining, unmasked parts. Several studies demonstrate the use of self-prediction methods for medical image restoration tasks. Jung et. al. intro-duced a masked auto-encoder technique for functional connectivity matrixrestoration in rs-fMRI data. The method involves masking different rowsand columns of a matrix and then reconstructing the original matrix, wherethe functional connectivity matrix represents the relationship between dif-ferent regions of interest for each subject [41]. Jana et. al. proposed anencoder-decoder architecture tailored for restoring CT scans that have beencorrupted by swapping several small patches within a single CT slice [39].152.3. MEDICAL IMAGING BACKBONELiu et. al. developed a U-net model for restoring ultrasound images thathave been augmented through local-pixel shuffling. Clinical variables suchas age, gender, and tumour size are then concatenated with the encoder’soutputs for downstream prediction tasks [53]. A similar approach was takenby Zhong et. al., who designed three image restoration tasks for cine-MRIvideos, employing a U-net-like encoder-decoder architecture with skip con-nections [105]. These studies reported slight relative improvements in per-formance when compared to models without self-supervised pre-training,with increases of 1.12% [70] and 0.690% [53] noted in some cases.At present, there seems to be an absence of an SSL-based backbonededicated to medical image understanding tasks. Recognizing the prob-lems posed by the need for extensive annotated datasets, SSL emerges asa promising alternative, capable of mitigating these challenges. Unlike su-pervised learning methods that typically require large quantities of labelleddata, SSL has the potential to generate versatile models. These generalistmodels can be subsequently fine-tuned for a variety of downstream tasks,even in the absence of large labelled datasets. SSL’s ability to work with lim-ited annotations can substantially decrease the time and financial resourcesnecessary for developing medical imaging models for diverse applications,making automated systems more accessible to different specialties and hos-pitals. This, in turn, has the potential to enhance patient care. To addressthis need, our work introduces an SSL-based backbone. This generalistmodel is designed to be applicable to a range of downstream tasks, demon-strating its capabilities in automating medical systems for a diverse range ofapplications. Our approach not only presents this innovative technique butalso provides an empirical assessment of its effectiveness, particularly whenapplied to automating the diagnostic CT QC process.2.3 Medical Imaging BackboneUp to this point, our discussion has centred around CNNs and self-supervised techniques pre-trained extensively on vast datasets of naturalimages. However, it is crucial to note that, to the best of our knowledge,there is an absence of open-source DL models exclusively trained on medicalimaging data, whether through supervised, self-supervised, or unsupervisedlearning paradigms. The medical imaging field is particularly affected by thescarcity of adequately annotated data, as the data labelling process demandsthe expertise of clinical professionals, rendering it both time-consuming andcost-intensive [4]. Additionally, medical imaging data is tied with intricate162.3. MEDICAL IMAGING BACKBONEsecurity and privacy considerations [93]. Consequently, facilitating publicaccess to this category of data is significantly more challenging compared tonatural image datasets. As a result, transfer learning emerged as a strate-gic solution, aiming to diminish the dependency on large-scale datasets byapplying the knowledge acquired from training CNN techniques on vast col-lections of natural images. Despite this advance, it has been observed thatthese models fail to be consistent when subjected to diverse settings. Thisis primarily due to the dissimilarity in features extracted from natural im-ages when compared to those specific to medical images, such as X-rays, CTscans, and MRIs [4].Alzubaidi et al. [4] are the first to discuss around the necessity for adedicated medical imaging backbone, emphasizing its potential to advanceresearch in medical image classification tasks. They propose ‘MedNet’, aCNN-based backbone specifically designed to address this need. ‘MedNet’is trained on a collection of several medical image classification datasets.The authors assert that such a dedicated backbone is poised to exhibit su-perior generalization capabilities across various medical imaging classifica-tion tasks, thereby establishing a solid foundation for future explorations inthis field. Despite the innovative approach, we have identified several pointsof observation that are worthy of attention. Firstly, the description pro-vided for the CNN architecture in the study is somewhat lacking in detail,leaving certain specifics ambiguous. While it is mentioned that unlabelleddata plays a role in the training process, the details of this approach arenot adequately explained, resulting in a lack of comprehension. The fea-sibility of training a CNN-based model on an unlabelled dataset, prior tofine-tuning it on a smaller, labelled dataset, remains unclear. Given thenature of CNN methodologies, large amounts of well-labelled data are aprerequisite for the development of a robust and resilient model. Secondly,the study primarily concentrates on medical image classification, overlook-ing other crucial aspects such as image segmentation and object detection.Classification constitutes just a portion of the diverse applications within themedical imaging sphere. To truly enhance the model’s generalization capa-bilities, it would be beneficial to train it on a more general task, compellingit to understand and learn the specific characteristics of the medical images,rather than merely focusing on the final outcome. Lastly, the original re-search, proposed in 2021, pledged to make ‘MedNet’ openly accessible, withan objective to advance research in this field. Nevertheless, as of the writingof this thesis, the open-source version of the model remains unavailable forscrutiny or utilization. Furthermore, there has been a notable absence ofany subsequent publications or updates from the authors to refine the initial172.3. MEDICAL IMAGING BACKBONEstudy or provide the much-needed clarity and elaboration on their work. Itis important to note that the author of this thesis tried contacting Alzubaidiet al. [4] with the intention of acquiring the model for research purposes orgaining additional insight into their proposed methodology. There has beenno reply to these inquiries up until the completion of this thesis.18Chapter 3Automating CT DiagnosticImage Quality Control UsingDeep Transfer LearningIn this chapter, our objective is to evaluate a range of cutting-edge deepneural networks, with the aim of automating the QC process in medicalimaging. To this end, we have selected five state-of-the-art DL models, eachof which has been pre-trained on the extensive ImageNet dataset. Thesemodels are subjected to a fine-tuning process using CT diagnostic medicalimage datasets to assess their efficacy as automated tools for the QC process.The models included in this study represent the top five techniques identi-fied from a comprehensive review of cutting-edge methods in medical imageclassification, as detailed by Kim et al. in [47]. To facilitate this research, were-implemented, trained, and tested each of the selected techniques to en-sure a rigorous and accurate evaluation. Specifically, the training and testingphases were conducted on two specialized medical QC datasets, which weresourced from Hospital-A and Hospital-B, both located in British Columbia,Canada. A visual representation of the transfer learning process utilized inthis study is provided in Figure 3.1, offering a comprehensive illustration ofour study methodology.In the second phase of this chapter, we shift our focus toward assessingthe versatility and robustness of the different models through out-of-domainassessment. To facilitate this, we have curated an additional specializeddataset specifically for this purpose, enabling us to evaluate how perfor-mance metrics fluctuate when the models are exposed to more varied anddiverse settings.Overall, we find that InceptionResNetV2 stands out as the most ac-curate model, achieving an impressive 99% accuracy across both datasetsand outperforming the other evaluated techniques. However, it is crucialto highlight that when these models are subjected to more diverse settings,a notable decrease in performance accuracy is observed, with a significant19drop of 30-40%. This observation highlights the need for a more robustDL-based backbone specifically designed for medical imaging, ensuring con-sistent performance even in out-of-domain settings.It is worth noting that the results and insights presented in this chapterare based upon the work accepted at the IEEE MoSCIom 2023 conferenceby the author of this thesis.Dense LayersNew Classification LayersTarget LabelsImage fromcalibrated scannerImage frommiscalibratedscannerPre-Trained ModelsVGG19ResNet50InceptionResNetV2XceptionNetEfficientNet(B) Transfer Learning...(A) Diagnostic CT QCTarget Datasetsdataset1dataset2dataset3 . . .>=2Figure 3.1: An illustrative example of the transfer learning process flowused in our study. In phase (a), we curate various CT QC diagnostic tar-get datasets, which serve as the foundation for fine-tuning our pre-trainedmodels. This initial phase is crucial for adapting the models to the specificrequirements of medical QC imaging tasks. Subsequently, in phase (b), wetransition to the selection of five state-of-the-art models for automated QCevaluation. The models under consideration are (1) VGG19, (2) ResNet50,(3) InceptionResNetV2, (4) XceptionNet, and (5) EfficientNetB7. To eachof these models, we append a classification head, the role of which is toclassify the target labels. This classification head undergoes fine-tuning onthe target dataset to optimize its performance. In the context of our study,the target labels are defined as ‘Image from calibrated scanner’ and ‘Imagefrom miscalibrated scanner’, enabling binary classification.203.1. PROPOSED STUDY DESIGN3.1 Proposed Study DesignIn this section, we discuss our proposed DTL study design, crafted toassess the automation potential within the CT QC process. Our discussioncommences with a detailed description of the curated datasets, each specif-ically chosen to provide a comprehensive basis for comparative evaluation.Following this, we will explore the various CNN models selected for thiscomparative study, shedding light on their unique features and relevance toour research objectives. Concluding this section, we will introduce and elab-orate upon our proposed approach aimed at automating the quality controlprocedures in CT imaging.3.1.1 Diagnostic CT QC DatasetsSpecialized datasets were built to meet the unique requirements of thisresearch study. Local ethics board permission was obtained prior to datacollection. The Interior Health Authority (IHA) granted us access to theirPicture Archiving and Communication System (PACS). Collaboratively,Advanced Quality Systems (AQS) and IHA provided us with their QCrecords. These records comprise logs of the periodic QC tests conductedacross healthcare facilities under the IHA’s purview. Table 3.1 shows theCT QC Results - Daily/Weekly/Monthly TestsControlLimitsCT number+/-4 HUNoise (S.D.)Range 4.0 to 5.0Baselinevalues0.63 4.40DateTechInitialsCentre ROICT #Centre ROIStd. Dev.Day-1 AG 1.01 4.25Day-2 AG 0.98 3.89Table 3.1: An overview of CT QC logs logged by a specialist at the IHAfacility. Scans collected on ‘Day-1’ fall within the required specification.However, the Noise (S.D.) in the scans collected on ‘Day-2’ is below theminimum threshold value of 4.0. Therefore, the scanner is deemed out-of-calibration on ‘Day-2’ and any scans collected on that day will be categorizedand labelled as miscalibrated.results from a QC test executed by a specialist at an IHA facility. Whileall parameters from Day-1, meet the expected range, the standard devia-21tion (noise) on Day-2, falls short of the 4.0 minimum threshold, leading thetechnician to mark the test on that day as a failure. We recorded datesmarked with test failures, focusing specifically on entries from Hospital-Aand Hospital-B. For accuracy, we collaborated with a QC specialist from theIHA facility to cross-check and verify these dates, ensuring there were nofalse positives. Upon completion of this process, we recorded specific dateswhen the scanner calibration was off or when QC errors were present. It’simportant to emphasize that these discrepancies are often so small that theyare hard to detect just by looking at the scans produced by these scanners,as can be seen in Figure 3.2. As a consequence, images from miscalibratedscanners can result in incorrect diagnoses, leading to potentially harmfulconsequences.Next, we procured patient studies from IHA. These studies were retrievedin the form of DICOM files. A meticulous de-identification process wasapplied to each scan during the export from PACS, ensuring the absenceof any identifiable patient data, in line with our ethics approval. Afterde-identification, we stored these DICOM files on an encrypted hard drive.Leveraging the python Pydicom library [56], each DICOM file was convertedto an image format. Processing each study took around 10 minutes, with astudy containing about 70-200 CT slices.Hospital-A Head Scans - ‘dataset1’We initiated our dataset creation with a CT scanner at Hospital-A, form-ing what we denote as ‘dataset1’. As outlined in Table 3.2, we compiledhead CT scans from Hospital-A. By aligning these scans with our previouslymentioned validated QC logs, we identified images from periods when thescanner was deemed out of calibration; these were tagged as ‘Fail’ scans forsubsequent training and testing. Specifically, our dataset comprised 3919‘Fail’ training images; and 1307 ‘Fail’ validation and testing images each,summing up to a total of 6533 ‘Fail’ images. Similarly, we also had QC datapertaining to periods when the scanner was operating without any errors.Scans from these phases were categorized as ‘Pass’ scans. For this category,we collected 4872 training images; and 1624 images each for validation andtesting. This culminated in a total of 8120 ‘Pass’ images, slightly outnum-bering the ‘Fail’ images. The total training, validation and testing imageswithin ‘dataset1’ are 8791, 2931 and 2931, respectively. Figure 3.2 depictsaxial head CT scans from Hospital-A.223.1. PROPOSED STUDY DESIGNHospital-B Head Scans - ‘dataset2’For ‘dataset2’, we directed our focus onto a CT scanner at Hospital-B.To break it down, our dataset included 12967 ‘Fail’ training images; 4326validation; and 4319 testing images, leading to a grand total of 21612 ‘Fail’images. Scans were labelled as ‘Pass’ when the scanner was functioningwithout any issues. Within this classification, we compiled 9963 trainingimages; and 3321 images each for validation and testing purposes. Thisamounted to a comprehensive collection of 16605 ‘Pass’ images, slightlylower than the count of ‘Fail’ images. Collectively, ‘dataset2’ comprised22930 training, 7647 validation, and 7640 testing images. Figure 3.2 depictsaxial head CT scans from Hospital-B.3.1.2 Pre-Trained ModelsOwing to its convolutional layers, the CNN architecture has become afavourite for computer vision and image classification tasks. These layersempower the network to extract high-level features from images [26], [46],[65]. In this study, we investigated the VGG19, ResNet50, InceptionRes-NetV2, XceptionNet and EfficientNetB7 CNN architectures. The aim is toevaluate and compare the performances of different models in automatingQC using CT scans.VGG19 [85] architecture is a deep CNN comprising 19 layers, including16 convolutional layers, 3 fully connected layers, and 5 max-pooling layers.One of the significant advantages of VGG19 is its ability to capture intricatepatterns and details in images due to its depth.ResNet50 [33] is a variation of the Residual Network (ResNet) architec-ture designed for DL tasks in computer vision. What sets ResNet architec-tures apart is the introduction of “skip connections” or “shortcuts”, whichallow activations to bypass one or more layers. These connections com-bat the vanishing gradient problem, enabling the training of much deepernetworks without a degradation of performance.InceptionResNetV2 [91] is a hybrid architecture that merges thestrengths of two prominent DL models: Google’s Inception networks andMicrosoft’s ResNet. It was developed with the intent of combining the ben-efits of the two models to create a high-performing and efficient architecturesuitable for intricate computer vision tasks.XceptionNet [17] builds upon the foundational ideas of the Inceptionarchitecture but introduces an alteration: instead of the standard inceptionmodules, Xception employs depthwise separable convolutions. Empirical253.1. PROPOSED STUDY DESIGNevaluations have shown that Xception can achieve performance on par with,or surpass, other state-of-the-art models while requiring fewer parameters.EfficientNetB7 [96] is one of the variants of the EfficientNet family,which was proposed by Google Research. EfficientNetB7, being one of thelarger variants, has more layers and is wider than its predecessors in the Effi-cientNet series. Despite its increased size, the architecture retains efficiencyby balancing the depth, width, and resolution.It is worth noting that the selection of the top five models consideredmost appropriate for conducting the proposed evaluation, was made subse-quent to an exhaustive and thorough review of the available options [47].3.1.3 Automating Quality Control Using Deep TransferLearningIn DTL, a variety of model-based strategies are employed. Typically,these methods integrate aspects of pre-training, freezing certain layers, fine-tuning, and/or appending new layer(s) to the existing model. Figure 3.1provides an in-depth visualization of our transfer learning process. Duringthe first phase, we prepare target datasets for fine-tuning the pre-trainedmodels. After that, we select five state-of-the-art pre-trained CNN models.These models are pre-trained on the expansive ImageNet dataset, servingas our foundational source of knowledge. The knowledge obtained from thesource dataset is transferred in the form of frozen convolutional layers. Byfreezing certain layers, we ensure that the parameters or weights within theselayers remain consistent, effectively retaining the knowledge acquired fromthe original pre-trained model. Following this, we introduce new classifica-tion layers to each of these foundational models, fine-tuning them to ourtarget dataset. The final layers of each model undergo a flattening processto transform their structure. Subsequently, a dense layer, consisting of 1024units and a ’relu’ activation function, is appended. This is followed by adropout layer to minimize overfitting. Lastly, we introduce a dense layerwith a single unit governed by a ’sigmoid’ activation function, apt for ourbinary classification objective. With this setup in place, each model under-goes training on our designated dataset, and their performance is gaugedbased on selected evaluation metrics.The proposed methodology was implemented using Python, leveragingmachine learning libraries like TensorFlow and Keras. To accelerate thetraining process, we utilized GPU acceleration, specifically the NVIDIAA6000. The DL models were configured with the following hyperparameters:a batch size of 10, a learning rate set at 0.0001, and a total of 70 epochs,263.1. PROPOSED STUDY DESIGNwith ’adam’ as the optimizer, and ’binary crossentropy’ loss function.3.1.4 Evaluation MetricsFor evaluation, we determine several performance metrics that are crucialin gauging the performance and effectiveness of each model in automatingthe CT QC process. These metrics include accuracy, recall (R), precision(P), F1 Score and the Loss function. [60].Accuracy =TP + TNTP + TN + FP + FN(3.1)Recall(R) =TPTP + FN(3.2)Precision(P ) =TPTP + FP(3.3)F1 Score = 2 ∗ P ∗RP +R(3.4)Loss = − 1nn∑i=1[yi log(yˆi) + (1− yi) log(1− yˆi)] (3.5)The definitions for these metrics are outlined below:1. True positive denotes the instances where the model accurately pre-dicts that an image originates from a calibrated/good scanner, andthe ground truth confirms that the image indeed comes from a cali-brated/good scanner. The cumulative count is represented as TP.2. True negative is when the model correctly predicts that an image isfrom a miscalibrated/bad scanner, and the actual ground truth con-firms that the image was indeed from a miscalibrated/bad scanner.The cumulative number is termed TN.3. False positive refers to the instances where the model incorrectly pre-dicts that an image originates from a calibrated/good scanner when,in fact, the ground truth indicates that the image comes from a mis-calibrated/bad scanner. The aggregate count is denoted as FP.4. False negative occurs when the model mistakenly predicts that animage is from a miscalibrated/bad scanner when in actuality, the im-age was produced by a calibrated/good scanner. The total count isdenoted as FN.273.2. EXPERIMENTAL RESULTS5. Accuracy (Eq. 3.1) is the ratio of the number of correct predictionsto the total number of predictions made (or the total number of in-stances).6. Recall (Eq. 3.2) is the ratio of the number of true positives to thesum of true positives and false negatives. Recall gives insight into howmany of the actual positive cases the model is able to capture.7. Precision (Eq. 3.3) is the ratio of true positives to the sum of truepositives and false positives.8. F1 Score (Eq. 3.4) is the harmonic mean of precision and recall. Itprovides a single score that balances between the precision and recallof a model.9. Loss (Eq. 3.5) is the the cross-entropy loss function. This loss functionis widely used in binary classification tasks, especially with modelsthat output probabilities, like logistic regression or neural networkswith a sigmoid activation function in the output layer. It effectivelymeasures the difference between the predicted probabilities and theactual labels.3.2 Experimental ResultsIn this section, we detail our comparative analysis and out-of-domainassessment results.3.2.1 Comparative AnalysisWe conducted a performance assessment of several leading models toclassify CT images accurately. Our initial evaluation focused on ‘dataset1,’sourced from Hospital-A. The results from all models on ‘dataset1’ are de-tailed in Table 3.3 and 3.4. Amongst the tested models, InceptionResNetV2showcased superior performance, securing the highest classification accu-racy of 99% on both the training and test datasets. In contrast, EfficientB7recorded the lowest accuracy levels, with 80.50% and 86.45% on the trainingand test datasets, respectively. VGG19 and ResNet50 demonstrated com-parable performance, each achieving around 95% classification accuracy onthe test dataset. Notably, VGG19, with the least number of parameters(20,550,721), exhibited the highest computational efficiency, while Efficient-NetB7, with the highest number of parameters (64,425,617), took longer283.2. EXPERIMENTAL RESULTStime to train in comparison to other models. In conclusion, InceptionRes-NetV2, with the highest testing accuracy, outperformed other models inclassifying diagnostic CT images.Training Set Validation SetModel Accuracy Loss Accuracy LossVGG19 0.9968 0.0214 0.9604 0.1061ResNet50 0.981 0.0364 0.9621 0.1096InceptionResNetV2 0.9999 5.29E-04 0.9911 0.0438XceptionNet 0.9788 0.0657 0.9338 0.1872EfficientB7 0.8051 0.3889 0.8676 0.292Table 3.3: Training accuracy results using five different state-of-the-art CNNmodels. Each model was independently trained on the ImageNet trainingset and fine-tuned on the CT diagnostic image dataset1. As observed, Incep-tionResNetV2 achieves the highest accuracy on the training and validationset.Test SetModel Accuracy Loss TP TN FP FNVGG19 0.9556 0.128 1244 1560 64 63ResNet50 0.9593 0.1357 1250 1562 62 57InceptionResNetV2 0.9918 0.0424 1294 1613 11 13XceptionNet 0.9372 0.1811 1223 1553 100 84EfficientB7 0.8645 0.3045 1152 1382 242 155Table 3.4: Performance evaluation of five state-of-the-art CNN Models onCT diagnostic imaging. Each model underwent initial training on the Ima-geNet dataset, followed by fine-tuning using the CT diagnostic image train-ing dataset1. The independent test set comprises 2,931 images, unseenby the models during the training phase. InceptionResNetV2 stands out,demonstrating superior accuracy on the test set.The second round of evaluations was focused on ‘dataset2’. The distinc-tive performances of each model are itemized in Table 3.5 and 3.6. In thisseries of assessments also, InceptionResNetV2, emerged as the most profi-cient model, recording an impressive classification accuracy of 99% on thetraining dataset and 99.21% on the test dataset. Conversely, EfficientNetB7lagged, marking the lowest accuracies of 80.48% and 86.01% on the trainingand test sets, respectively. VGG19 and ResNet50 displayed analogous per-293.2. EXPERIMENTAL RESULTSformances, each securing approximately 93% classification accuracy on thetest set. Moreover, ResNet50 recorded the highest F1 Score of 0.775 and0.761 on the training and validation set, respectively. XceptionNet demon-strated superior classification accuracy, achieving 94.81% on this particulartest set, thereby surpassing the performances of both VGG19 and ResNet50.A plausible explanation for this enhancement could be that the increase indataset size corresponds to an improvement in the performance of Xception-Net. It’s worth noting that VGG19, having the smallest parameter count,demonstrated optimal computational efficacy. To summarise, Inception-ResNetV2 outperformed the other models in terms of correctly classifyingdiagnostic CT images within ‘dataset2’.Training Set Validation SetModel Accuracy Loss Accuracy LossVGG19 0.9924 0.0356 0.9315 0.1794ResNet50 0.9801 0.0431 0.9375 0.2637InceptionResNetV2 0.999 4.54E-04 0.9935 0.0307XceptionNet 0.9993 0.007 0.9529 0.1646EfficientB7 0.8048 0.399 0.8663 0.2979Table 3.5: Training accuracy results using five different state-of-the-art CNNmodels. Each model was independently trained on the ImageNet trainingset and fine-tuned on the CT diagnostic image dataset2. As observed, Incep-tionResNetV2 achieves the highest accuracy on the training and validationset.As discussed above, numerous models yield high accuracies in the clas-sification of diagnostic CT images. However, elevated accuracy does notinvariably equate to model efficacy and resilience. To rigorously assess themodels’ adaptability and robustness across varying domains, we implementan out-of-domain assessment for these trained constructs. This step is cru-cial to ensuring that the model’s high performance is not solely a result ofoverfitting to a specific dataset and can maintain its reliability and precisionwhen exposed to diverse and unseen data.3.2.2 Out-of-domain AssessmentModels chosen for this study undergo an initial pre-training phase ondatasets comprising natural images (non-medical images) and are subse-quently fine-tuned using much smaller and balanced medical imaging datasets.While certain models demonstrate remarkably high predictive accuracies,303.2. EXPERIMENTAL RESULTSTest SetModel Accuracy LossVGG19 0.9339 0.1697ResNet50 0.9383 0.2634InceptionResNetV2 0.9921 0.0361XceptionNet 0.9481 0.1608EfficientB7 0.8601 0.3083Table 3.6: Performance evaluation of five state-of-the-art CNN Models onCT diagnostic imaging. Each model underwent initial training on the Ima-geNet dataset, followed by fine-tuning using the CT diagnostic image train-ing dataset2. The independent test set comprises 7,640 images, unseenby the models during the training phase. InceptionResNetV2 stands out,demonstrating superior accuracy on the test set.asserting with certainty the robustness and generalizability of these modelsfor accurate classification of CT QC images remains a complex challenge.This uncertainty highlights the necessity for rigorous evaluation to authenti-cate the model’s capability to maintain performance integrity across variedand unseen datasets, ensuring reliability in diverse application scenarios.When working with CT images, the characteristics of the images can varydue to alterations in specifications like the scanner make and manufacturer.Furthermore, scanners can deviate from calibration due to several reasons,including oil spills within the scanner or CT numbers falling below or ex-ceeding predetermined thresholds, among other factors. Each of these errorsor specifications can influence the attributes of the image uniquely, affectingthe quality and reliability of the diagnostic information derived from theimages. This variability underscores the importance of robust and versatilemodels capable of accommodating such diversities in image characteristicswhile maintaining high accuracy in image analysis. We perform additionalexperiments to assess the models’ reliability across diverse scenarios. Thefindings from these supplementary experiments are compiled and presentedin Table 3.7. These experiments aim to provide a more holistic view of eachmodel’s robustness in varying conditions, ensuring their practical utility inreal-world applications. The evaluations also highlight each model’s abilityto generalize and perform accurately in out-of-domain scenarios.Based on the results of our previous experiments, we selected the threemodels that exhibited superior performance, specifically VGG19, ResNet50,and InceptionResNetV2. Each of these models has been fine-tuned utilizing313.2. EXPERIMENTAL RESULTSModel Accuracy Recall PrecisionModeltrained on‘dataset1’VGG19 0.5112 0.495 0.5794ResNet50 0.5163 0.521 0.58072InceptionResNetV2 0.4943 0.577 0.55047Modeltrained on‘dataset2’VGG19 0.5199 0.551 0.4675ResNet50 0.5032 0.619 0.4578InceptionResNetV2 0.4961 0.59 0.551Table 3.7: Cross-Evaluation of top three state-of-the-art CNN Models forout-of-domain performance on CT diagnostic imaging. Following initialtraining on the ImageNet dataset, each model was fine-tuned using eitherCT diagnostic image training dataset1 or dataset2. To assess robustness,models fine-tuned on dataset1 were tested using the test set from dataset2,and vice versa.‘dataset1’ and ‘dataset2’. A crucial consideration is the disparity in CTscanner manufacturers between ‘dataset1’ and ‘dataset2’. Additionally, thetypes of errors leading to scanner miscalibration differ between ‘dataset1’and ‘dataset2’. Informed by insights from the QC team and QC experts,we can assert with considerable confidence that images from ‘dataset1’ and‘dataset2’ exhibit distinct characteristics, rendering them as out-of-domainsets relative to each other. Consequently, a model that has been fine-tunedon ‘dataset1’ can have its resilience and adaptability assessed by evaluatingits performance on ‘dataset2’, and vice-versa. This cross-evaluation servesto verify the model’s capability to maintain accuracy and reliability acrossdivergent datasets, showcasing its potential for generalized application.Initially, we train all three selected models on ‘dataset1’ and assess theirperformance on ‘dataset2’. Notably, InceptionResNetV2, while superior inperformance within its training domain, exhibits the least adaptability, reg-istering the lowest accuracy at 49.43%. Similarly, when models fine-tuned on‘dataset2’ are applied to ‘dataset1’, InceptionResNetV2 again lags behind,recording the lowest accuracy of 49.61%.3.2.3 DiscussionIt is evident from the experiments that the models, despite their im-pressive accuracies in their respective training domains, struggle when con-fronted with out-of-domain datasets. Moreover, it is ironic that the modelwhich showcased the best performance within its domain displayed the leastresilience and adaptability under varied conditions. Possible rationales for323.2. EXPERIMENTAL RESULTSsuch outcomes include:− The CT datasets employed may be insufficiently extensive, potentiallyleading to overfitting during the fine-tuning of the additional classifi-cation layers.− Given the pre-training of the models on natural images, their abilityto generalize effectively to medical images is likely compromised.− There exist substantial inherent differences between the two datasets,lacking common transferrable image features, making effective knowl-edge transfer challenging.In this study, we acknowledge the risk of overfitting during the fine-tuning process of additional classification layers in transfer learning. Over-fitting is a prevalent issue when a model, initially trained on a source task,excels with training data but underperforms with new, unseen data in thetarget task. This challenge is common in transfer learning when there is asubstantial disparity between the source and target domains or when thetarget dataset is limited in size. In the medical imaging domain, this risk isheightened due to the distinct nature of medical images compared to natu-ral images. To address this, our approach involves incorporating a smallerset of additional layers rather than a large classification head, which couldpotentially lead to overfitting. Furthermore, we strategically unfreeze spe-cific layers within the original model, allowing them to train along with thenewly added classification head. This strategy allows the adjustment of theoriginal layers’ weights to better suit the target task, thereby aiding in themitigation of overfitting. Additionally, the use of a ‘ReLU’ activation func-tion coupled with dropout layers serves as an effective measure to furtherminimize the likelihood of overfitting. These steps, when applied collectively,help mitigate the potential effects of overfitting, thereby safeguarding themodel’s ability to generalize effectively. However, it is important to explic-itly acknowledge that the high accuracies reported for the 5 state-of-the-artmodels are likely due to overfitting. The dataset utilized in this study isrelatively small, a factor that often is responsible for models to overfit thespecific characteristics of the target problem. This limitation implies thatwhile the observed accuracies are high, they may not accurately reflect themodels’ ability to generalize to new, unseen data. Recognizing this issue isessential for a realistic evaluation of the models’ performance and indicatesthe need for further testing on larger and more diverse datasets to trulyassess their effectiveness in practical applications.333.3. CONCLUSIONIt is crucial to craft models that not only perform well within theirspecific domains but also sustain robustness and adaptability when exposedto out-of-domain data. The challenges observed through this study invitemore exploration and scrutiny in this domain. In response, we present aresearch solution aimed at addressing these complexities, the details of whichare elaborated in the next chapter.3.3 ConclusionIn this chapter, we have explored the feasibility of automating the CTQC process through the application of transfer learning across various DLmodels. Our findings revealed that, among the tested models, InceptionRes-NetV2 demonstrated superior performance, achieving an accuracy of 99% onthe in-domain dataset. To further assess the robustness and adaptability ofour refined models, an out-of-domain evaluation was conducted, whereinInceptionResNetV2 showed a significant decrease in accuracy, recording49.43%. Through this study, we realized the need for a resilient DL-basedbackbone capable of adapting to diverse scenarios. In the next chapter, weadvance this line of research by developing a self-supervised backbone, us-ing medical images to construct a pre-trained model. This self-supervisedbackbone will act as a foundation, enabling fine-tuning on scarce labelleddatasets.34Chapter 4MedMAE: A NovelPre-Trained MaskedAutoencoder for MedicalImaging TasksIn this chapter, our objective is to develop a pre-trained DL backbonefor medical image applications by leveraging Self-Supervised Learning (SSL)ability to extract meaningful representation from unlabeled data. Our pri-mary goal is to craft a model, pre-trained on a suitable pretext task, thatdemonstrates the capacity to seamlessly adapt to a variety of downstreamtasks within the domain of medical image analysis, without the need forwell-annotated large datasets. The idea behind using a pretext task stemsfrom our intention to create a backbone that delves deep into the uniquecharacteristics of medical images, distinguishing them from typical natu-ral images and extracting their intrinsic features. Currently, the field lacksopen-source models pre-trained through pretext tasks designed for versatiledownstream applications in medical imaging. By reducing the dependencyon manual annotations for training medical imaging models, we anticipatea significant reduction in both developmental costs and time. The final in-tention is to make automated systems more widely available across differentbranches of medicine and healthcare settings.In the initial phase of our research, we developed a novel backbone thatdraws inspiration from the Masked Autoencoder (MAE) approach [31]. Thisbackbone, which we have named MedMAE-B, is characterized by a uniqueself-supervised pretext training designed specifically for medical imaging ap-plications. For training the backbone, we utilized a dataset of over 2 millionmedical images, which was assembled by collecting numerous medical imag-ing datasets available in the public domain.Building upon the foundation laid by MedMAE-B, we have further de-veloped an end-to-end novel model tailored for medical image applications,354.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKSnamed MedMAE-F. This model refines the capabilities introduced by MedMAE-B and extends them through training on specific downstream medical imag-ing tasks. A detailed description of MedMAE-F’s architecture is provided,showcasing its novel features and potential for medical image analysis.During the subsequent phase of this chapter, we leverage the capabilitiesof our novel MedMAE-F to assess its effectiveness in automating QC of CTdiagnostic images. This evaluation aims to highlight the model’s enhancedgeneralization capabilities and accuracy. To provide a comprehensive anal-ysis, we conduct a case study and draw comparisons with the performanceof various CNN models. In order to carry out this research analysis, wecreate additional specialized CT QC datasets. This enables us to evaluateboth the performance and the ability of the proposed MedMAE-F model togeneralize across diverse settings within the medical image analysis domain.Overall, we find that MedMAE-F performs well when subjected to var-ious downstream tasks in the context of automating CT diagnostic imageQC. Our final objective is to make MedMAE-B and MedMAE-F publiclyaccessible to expand research and innovation in medical imaging analysis.We believe that such a model can be effectively used for downstream medicalimaging tasks generating higher accuracy and robustness.The results and insights presented in this chapter are based upon thework submitted to the IEEE Transactions on Biomedical Engineering journalby the author of this thesis.4.1 MedMAE-B: A Self-Supervised Backbone forMedical Imaging TasksIn this section, we delve into the extensive process of developing an SSL-based DL backbone, tailored for pretext training on medical images. Ourexploration begins with an in-depth analysis of the datasets selected for pre-training the SSL model. Following this, we introduce the MAE approach,which serves as the foundational backbone for our study. Additionally, wepresent the results obtained from the pretext training, outlining the out-comes and insights gained from this task.Figure 4.1 illustrates the MedMAE-B architecture, comprising an encoder-decoder model. The SSL learning paradigm proposed in this study is in-spired by the natural learning mechanisms of infants, who, within theirfirst 3 to 4 months, start forming meaningful expectations about their sur-roundings. Infants mostly rely on observation, common sense, and minimalinteraction rather than constant and explicit guidance [69]. The capacity364.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKSEncoderDecoderOriginal Image Reconstructed ImageInput PatchesReconstruction Loss(MedMAE-B: Pretext Task using Medical Images)Figure 4.1: Novel MedMAE-B architecture: The process is initiated by ran-domly masking 75% of the original image and inputting the remaining 25%of visible patches into the encoder, which captures the latent representa-tions and encodes the patches. Subsequently, the aim of the decoder is toreconstruct the complete image using the encoded and masked patches. Thereconstruction loss helps to improve the reconstruction with each iteration.for self-learning stems from their ability to derive supervision from theirenvironment, aiding in the development of a general understanding of theworld. MedMAE-B aims to replicate this self-supervised mechanism, allow-ing the model to discover intrinsic features in the medical imaging trainingdata, independent of annotated labels. The ultimate objective is to learnversatile representations from input medical images that can be effectivelytransferred to downstream tasks, even when labels are scarce.37∼4.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKS4.1.1 Medical Imaging DatasetsWe explore the potential of leveraging an SSL-based DL model to ad-dress existing challenges in the field of medical image analysis. The mainobjective of our approach is to pre-train MedMAE-B using an extensive anddiverse dataset for a pretext task. This strategy is aimed at enabling themodel to capture latent representations, specifically the unique character-istics inherent to medical images. The expectation is that such a trainingprocess will result in a robust backbone model, that potentially has a deeperunderstanding of medical images, and which can be efficiently fine-tuned fora variety of downstream tasks. Furthermore, this approach is expected to en-hance the model’s generalizability, ensuring it can handle diverse scenarios.This is attributed to the pretext task-based training, which would enable adeeper understanding of medical image characteristics. To realize this, wehave aggregated multiple medical image datasets into one single huge anddiverse dataset, encompassing a variety of body parts and image types. Ourgoal is to expose the model to a wide array of characteristics, thereby en-abling it to learn diverse medical imaging characteristics that will improvegeneralization and increase versatility.Table 4.1 provides a comprehensive list of the various repositories andplatforms that have been tapped into for compiling our medical imagedataset. We have made a dedicated effort to document all possible sourcesutilized during the data gathering process. Despite these efforts, the au-thor acknowledges the possibility of unintentionally overlooking one or twosources, and this oversight is unintended. The goal of this extensive compi-lation is to ensure transparency and to provide a thorough account of thedata origins and acquisition for our research.During the pre-processing phase of our study, every dataset we compiledunderwent a series of transformations to prepare it for input into the MAEarchitecture. This involved converting each file into an image format andresizing it to meet the input specifications of the MAE model. Each DI-COM file was processed and converted to an image file. Moreover, NIfTI3D models were also processed and converted to 2D image files using thepython’s pydicom library. Additionally, we removed any corrupted or erro-neous files, ensuring the integrity and accuracy of our data. It is crucial tohighlight that we decided against employing data augmentation techniquesto artificially inflate the size of our dataset. Our strategy was centred aroundgathering a vast variety of datasets to naturally incorporate diversity intoour study, rather than relying on augmentation to simulate this variety. Weacknowledge the uneven distribution of dataset sizes, with some datasets394.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKSbeing significantly smaller than others. This is an aspect we have identi-fied for future improvement, as we aim to work towards creating a morebalanced and equitable dataset for comprehensive evaluation and analysis.Table 4.2 details the various datasets that were used to compile the bigdataset. The big dataset comprises 36,473 patient studies (subjects) whichtranslates to 2,121,619 (more than 2 million) images in total. These patientstudies include 1) CT images (Pancreas, Abdomen, Lung, Pelvis, Colon,etc.), 2) X-rays (Frontal Chest, Lungs, etc.), 3) MRI (Liver, Cardiovascular,Oncology, etc.), 4) PET scans, 5) skin cancer images, among others. Thisvast dataset ensures a diverse coverage across different medical imaging sce-narios, aiming to equip our model with a robust and versatile understandingof medical images. It is worth noting that we are still expanding the datasetand adding more and more datasets as and when they become available.Moreover, we also plan to add studies from IHA’s PACS server, for whichwe have ethics approval. The aim is to utilize the vast and diverse unlabelleddata to increase the model’s understanding of medical images. Introducinggreater diversity is pivotal, as it will improve the model’s ability to general-ize across various scenarios. We intend to make MedMAE openly accessiblefor research purposes within this domain.4.1.2 MedMAE-B: Self-Supervised Masked AutoencoderMedMAE-B draws inspiration from the masked autoencoder method in-troduced by He et al. [31], which serves as the core backbone. As illustratedin Figure 4.1, MedMAE-B attempts to reconstruct the complete image basedon its partially observed patches, leveraging a straightforward autoencoderarchitecture. The autoencoder functions by transforming the input patchesinto a latent representation, which is then utilized by the decoder to recre-ate the original image, incorporating both the latent representation and themasked patches. Featuring an asymmetric architecture, the encoder pro-cesses merely 25% of the original patches, while the decoder undertakes thetask of reconstructing the entire image using 25% of the latent representa-tion combined with 75% of the masked tokens. The process involves severalsteps to accomplish the task of image reconstruction from partial observa-tions. The steps, described below, are designed to ensure that the modellearns robust and meaningful representations of the input data:1. Random Masking The input image is divided into a set of patchesfollowing the ViT [26] approach. A substantial portion of these patches,typically more than 50% (75% in our case), are randomly masked, leav-404.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKSing behind half or less than half for the model to work with. A highlysparse input enables the opportunity to create an efficient encoder.2. Encoder The visible patches are fed into the encoder. The encoderarchitecture is a ViT [26], which transforms patches into a latent space,creating a compact and dense representation of the input data. Theonly difference is that it is applied only to the set of visible patches.Removing the masked patches allows the training of an encoder withonly a fraction of the computational power.3. Decoder The decoder receives both the latent representations of thevisible patches and the masked tokens (placeholders for the maskedpatches) [44]. Utilizing this information, the decoder attempts to re-construct the full original image, including the parts that were masked.The decoder has another series of transformer blocks. It is importantto note that the decoder is only used during pre-training to performimage reconstruction.4. Reconstruction and Loss Calculation In the output generated bythe decoder, each element is a vector of pixel values that representsa patch. A linear projection layer constitutes the final layer of thedecoder, its output channels matching the number of pixels presentin a patch. Following this, the decoder’s output undergoes reshapingto reconstruct the image. This process enables accurate and detailedreconstructions, even when only a small portion of the image is pro-vided. The loss function computes the mean squared error betweenthe reconstructed and the original image in the pixel space.5. Model Training Steps 1-4 are repeated for numerous iterations, withthe model continuously learning and adapting to minimize the recon-struction error.6. Pretext Task Learning Throughout this process, the encoder learnsto generate informative latent representations that capture the essen-tial characteristics of the input data.42 Un-Labelled ith instance Medical Image DB Feed-Forward Self-Attention Feed-Forward Self-Attention 4.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKSMedMAE-B Training WorkflowFor the training of our pretext task, we have employed ViT-B encoder,repurposing and tailoring it to accommodate the training of medical imagedata, as illustrated in Figure 4.2. This adaptation ensures that the modelis suited to handle the unique characteristics and requirements of medicalimages, setting the stage for effective and specialized downstream applica-tions. We fix the masking ratio at 75%, which implies that three-quartersof the image is randomly masked and only one-quarter is used as an inputto the encoder. Moreover, we only use ViT-B to cut down on the memorysize and computation time. We select MAE ViT-B, previously trained onImageNet as our base model to start with. This strategic choice enablesus to leverage the existing knowledge embedded within the model regard-ing general image characteristics, ultimately contributing to a reduction inboth computational time and memory requirements. Subsequently, we fur-ther train this model using our extensive medical imaging dataset, spanningover 1000 epochs, to adapt and specialize its capabilities for medical imageanalysis. This approach ensures that the model is not only proficient inhandling generic image features but is also able to interpret and analyzemedical-specific visual patterns. The results of MedMAE-B are discussed inthe subsequent section.4.1.3 Experimental ResultsIn this section, we delve into the outcomes of our MedMAE-B model,which underwent an extensive pre-training process spanning over 1000 epochs.Each epoch took approximately 1 hour and 20 minutes to train. Therefore,the model has a training time of more than 1000 hours in total. As pre-viously stated, our evaluation was exclusively based on the MAE ViT-Bmodel, given that the larger ViT-L and ViT-H models necessitate signifi-cantly greater computational resources in terms of time and memory. Weconfigured our model with a batch size of 64, and the dimensions of the inputimages were set to 224x224 pixels. The transformation and normalizationof the entire image dataset were facilitated through the transform function.It is crucial to highlight that our study solely focuses on grayscale medicalimages, without incorporating any colored image data. To set the learningparameters, we established a base learning rate of 10−3, and we employedthe mean squared error as our loss function to guide the model’s trainingand optimization process. Figure 4.3 provides a visual representation of themodel’s loss progression throughout the training epochs, as illustrated via444.1. MEDMAE-B: A SELF-SUPERVISED BACKBONE FOR MEDICAL IMAGING TASKSFigure 4.3: Training Loss Tensorboard Graph: Progression of loss valuesduring the extensive training of our MedMAE-B model over 1000 epochs,each comprising 1000 steps. The Y-axis represents the train loss magnitude,while the X-axis denotes the number of training steps.the TensorBoard graph. Each epoch comprises 1000 steps, and at each step,the loss value is plotted on the graph. In the initial phase of training, anoticeable and sharp decrease in loss is evident, indicating rapid learningand adjustment of the model’s parameters. This trend of decreasing losspersists until approximately the 300-epoch mark. Post this phase, the lossincreases for a bit and then starts to plateau, showcasing a more gradualand steady decrease. By the end of the 1000 epochs, the loss has successfullybeen reduced to below 0.025, signifying a substantial improvement from thestarting loss value of 0.045. This graph is instrumental in evaluating theeffectiveness of the training process, showcasing how the loss metric evolvesand diminishes over time, which is indicative of the model’s learning andoptimization progress.To offer a deeper understanding of the impact of the training processon the model’s performance, we present examples of image reconstructionsboth before and after the extensive training period. Figures 4.6, 4.7, 4.4 and4.5 are examples of reconstruction results on two different images. The firstimage is taken from the training set. Whereas, the second image is takenfrom a dataset that was not utilized during the training process.454.2. MEDMAE-F: NOVEL SSL MODEL FOR DOWNSTREAM MEDICAL IMAGING APPLICATIONSFigures 4.4 and 4.6 present the comparison between the original andreconstructed image. This comparison highlights the initial performance ofour MedMAE-B model in understanding and recreating the medical image.Figures 4.5 and 4.7 highlight the reconstruction capability of MedMAE-B after 1000 training epochs. The refinement in the reconstructed image,when compared to earlier checkpoints, is clearly visible. These examplesunderscore the effectiveness of our extensive training process in enhancingthe model’s proficiency in medical image analysis.4.2 MedMAE-F: Novel SSL model fordownstream medical imaging applicationsFollowing the pre-training phase, we present MedMAE-F, an end-to-endSSL model, built for downstream tasks in medical imaging. This modelbuilds upon the foundational MedMAE-B to offer specialized capabilitiesfor practical medical image applications. As part of the first step, we elimi-nate the decoder, originally utilized for image reconstruction. Then, a linearclassifier tailored for various medical image classification tasks is added tothe encoder model as shown in Figure 4.8. During the linear probing phase,only the added classifier is trained using the annotated labels, thereby trans-ferring the prior knowledge acquired by the MedMAE-B encoder. The en-coder weights remain frozen during the entire process. Alternatively, dur-ing the end-to-end fine-tuning approach, both the MedMAE-B encoder andthe newly added classifier are jointly trained to predict labels for a specificdownstream task. These dual strategies provide valuable insights into theeffectiveness of the pre-training process of MedMAE-B, shedding light onhow well the learned representations from the pretext task can be appliedto solve a range of diverse tasks in the medical imaging domain.4.2.1 MedMAE-F: Linear ProbingLinear probing is an evaluation method to assess the quality of repre-sentations learned during pre-training, particularly in SSL models. Thismethod helps to understand how well the pre-trained model has capturedthe general features and patterns in the data, which are crucial for down-stream tasks. In our experiments, we apply linear probing to the MedMAE-Fmodel. The key steps and components involved in our linear probing processare detailed below:1. Encoder: The pre-trained encoder of MedMAE-B is used as a feature484.2. MEDMAE-F: NOVEL SSL MODEL FOR DOWNSTREAM MEDICAL IMAGING APPLICATIONSMedMAE-F ArchitecturePredictionClassifierMedMAE-BEncoderLabelled Input ImageFigure 4.8: Novel MedMAE-F Architecture: MedMAE-B’s decoder is omit-ted, and a linear classifier is attached and trained to predict outcomes fordownstream medical imaging tasks.extractor. Its weights are frozen to ensure that we are evaluatingthe quality of features learned during the pre-training phase, and notlearning new features specific to the downstream task.2. Classifier: Batch Normalization layer is added to normalize the input,speeding up training and providing some regularization. A Sequentiallayer is used to define a linear stack of layers for our classifier. Soft-max Activation is used in the output layer to transform the outputinto probability distribution form, which is necessary for classificationtasks.3. Loss Function: CrossEntropy loss function (Eq. 3.5) suitable forclassification tasks is used. It helps to measure the performance of ourclassifier.Figure 4.9 provides a visual representation of the linear probing architec-ture utilized in our experiments, showcasing how the different componentsare connected and interact with each other. By performing linear probing,we aim to understand how well the MedMAE-F model can adapt its pre-learned features, especially when faced with diverse datasets from differentprocedures. This method provides a stringent test of the model’s gener-alization capabilities, as the encoder’s weights are not updated during thetraining of the classifier.494.2. MEDMAE-F: NOVEL SSL MODEL FOR DOWNSTREAM MEDICAL IMAGING APPLICATIONSNew ClassificationLayersTarget LabelsPassFailFrozen EncoderLabelled Input Image MedMAE-B>=2Figure 4.9: Linear Probing Architecture: The decoder is discarded, and theencoder is frozen with MedMAE-B’s pre-trained weights. A new classifica-tion header is added to the encoder. Only the added classifier is trained onthe downstream task.4.2.2 MedMAE-F: End-to-end Fine-TuningEnd-to-end fine-tuning is a strategy where, unlike linear probing, boththe MedMAE-B’s encoder and the newly added classifier’s weights are up-dated during training. This approach aims to fine-tune the entire model ona downstream task, allowing the encoder to adjust its pre-learned represen-tations to better suit the specific characteristics of the new task. Below isa detailed explanation of the components and steps involved in end-to-endfine-tuning for our specific use case:1. Encoder: The weights of the MedMAE-B encoder are no longerfrozen, meaning they will be updated during training. This allowsthe encoder to tailor its pre-learned representations to the specifics ofthe CT diagnostic image QC task. The encoder not only extracts fea-tures from the input data but also adjusts its internal representationsbased on the feedback received during training on the downstreamtask.2. Classifier: The classifier remains similar to the one used for linearprobing. Careful selection and potential adjustment to the learningrate is required to ensure that the pre-trained representations are fine-tuned rather than overridden. The learning rate was set as ‘1e-5’ forour case.504.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-FNew ClassificationLayersTarget LabelsPassFailPre-trained EncoderLabelled Input Image MedMAE-B>=2Figure 4.10: End-to-end Fine-Tuning Architecture: The decoder is dis-carded, and the MedMAE-B encoder is initialized with the pre-trainedweights. A new classification header is added to the encoder. The entirenetwork is trained on the downstream task.Figure 4.10 provides a visual representation of the MedMAE-F fine-tuning architecture utilized in our experiments, showcasing how the differentcomponents are connected and interact with each other. By employing end-to-end fine-tuning, we aim to leverage the generic representations learned byMedMAE-B during its pre-training on the large medical imaging dataset,while also allowing the model to adjust these representations to better suitthe specific characteristics of the downstream tasks. This approach seeks tostrike a balance between maintaining the benefits of the pre-trained modeland adapting to an entirely new task, potentially leading to improved per-formance and better generalization across diverse scenarios.4.3 Case Study: Automating Quality ControlUsing MedMAE-FIn this section, we delve into assessing the capability of MedMAE-F,our novel SSL model, in automating the QC process of CT diagnostic im-ages. The primary objective is to evaluate the model’s performance on aspecific downstream task, given its extensive pre-training on a diverse rangeof medical imaging data. It is crucial to highlight that our motivation be-hind adopting an SSL approach extends beyond merely achieving higheraccuracy rates. We aim to increase resilience and enhance the model’s abil-514.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-Fity to generalize across various settings. This is a critical factor, especiallywhen considering the challenges faced by InceptionResNet-V2 in adaptingto out-of-domain scenarios during our experimental assessments in Chap-ter 3. As observed, InceptionResNet-V2 exhibited impressive performancewithin an in-domain setting, reaching an accuracy rate of 99%. Nonetheless,its efficiency declined when exposed to out-of-domain conditions, with theaccuracy dropping by over 40%. This contrast in performance signals a po-tential issue; it suggests that the supervised model may either be overfittingon the limited dataset or struggling to capture the essential image charac-teristics unique to CT diagnostic medical images. These characteristics arecrucial for accurately distinguishing between ‘Pass’ and ‘Fail’ image classi-fications. By leveraging MedMAE-F, we are keen on establishing a robustmodel that maintains its integrity and delivers consistent performance, evenwhen exposed to a wide array of conditions. We believe that MedMAE-Fhas the potential to deliver stable and consistent performance across a vari-ety of scenarios, owing to its extensive pre-training on a substantial datasetof medical images. This pretext task has resulted in a model that has a deepunderstanding of characteristics intrinsic to medical imagery. As a result,when introduced to any scenario involving medical images, MedMAE-F canleverage this pre-acquired knowledge, utilizing it as a robust foundation toenhance performance and reliability.4.3.1 Out-of-domain AssessmentTo ascertain the efficacy of MedMAE-F in a downstream task, we cre-ated another dataset of CT diagnostic images aimed at automating the QCprocedure, details of which are outlined in Table 4.3. We compiled a diversecollection of more than 30,000 images, originating from 8 distinct procedures,and all performed using the same CT scanner situated at an IHA facility inBritish Columbia. The dataset was refined and pre-processed, resulting in 8small datasets, each corresponding to a specific type of procedure. All theimage data, initially in DICOM format, was converted into image files usinga Python-based library. To ensure accuracy in our annotations, we took thehelp of QC records provided by IHA, cross-checking the data and QC datesalong with the insights of a QC specialist at IHA. This thorough assessmentprocess was crucial in eliminating any potential false positives.It is crucial to highlight that each dataset within our collection, althoughoriginating from the same CT scanner, exhibits unique characteristics tiedto its specific diagnostic procedure. As a result, we can categorize eachdataset as being out-of-domain relative to the others. This categorization524.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-FProcedure StudiesPassImagesFailImagesCHEST/ABD/PELV/C+/CCSI 14 4552 4389HEAD/C- 11 710 701CHEST/C- 9 1338 1463CHEST/ABD/PELV/C+ 8 2585 2695ABD/PELV/C+ 8 1753 1865HEAD/C+ 7 452 457S/T/NECK/CHEST/ABD/PEL/C+/CCSI 5 2123 2342RENAL/C-/C+ 4 961 844Table 4.3: This table presents a detailed summary of ‘dataset3’, created toassess the effectiveness of MedMAE-F through downstream linear probingand fine-tuning tasks. The dataset comprises CT diagnostic images, sourceddirectly from the PACS of the IHA. It includes a variety of procedures, eachrepresenting a specific body part. For each listed procedure, the datasetincorporates both Pass and Fail patient studies.was also supported by inputs from a QC specialist. The categorization ofeach of these datasets as out-of-domain holds considerable importance forour study. It provides a pathway for testing and validating the resilienceof MedMAE-F across diverse medical imaging scenarios. By acknowledgingthe out-of-domain nature of these datasets in relation to each other, weset the stage for a comprehensive assessment of MedMAE-F’s performance,particularly in its ability to generalize and maintain consistent performanceacross varied medical imaging contexts.4.3.2 Experimental ResultsMedMAE-F: Linear ProbingBy performing linear probing, we aim to understand how well the MedMAE-F model can adapt its pre-learned features for the task of automating CTdiagnostic image QC, especially when faced with diverse datasets from dif-ferent procedures. The results presented in Table 4.4 provide a crucial in-sight into the versatility and adaptability of the pre-trained MedMAE-Fmodel. Since the downstream task is unrelated to the pretext task, and theevaluation dataset is entirely new and unseen during training, we can treatthis scenario as an out-of-domain testing for MedMAE-F. Two pre-trainedMedMAE-B checkpoints are considered for evaluation. As illustrated by the534.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-FProcedureAccuracy(Checkpoint 300)Accuracy(Checkpoint 1000)CHEST/ABD/PELV/C+/CCSI 91 96.65HEAD/C- 72.08 73CHEST/C- 93.23 99.82CHEST/ABD/PELV/C+ 97 99.05ABD/PELV/C+ 97.10 97.79HEAD/C+ 62.84 63.20S/T/NECK/CHEST/ABD/PEL/C+/CCSI 97.54 97.87RENAL/C-/C+ 93.094 94.20Average 87.99 90.20Table 4.4: MedMAE-F out-of-domain linear probing results for ‘dataset3’.training loss graph in Figure 4.3, checkpoint 300 is the local minima andcheckpoint 1000 is the global minima. The comparison between the per-formance of MedMAE-B at checkpoint 300 and checkpoint 1000 provides aclear insight into the impact of extended training on the model’s ability togeneralize and perform accurately across a variety of medical imaging tasks.MedMAE-B Checkpoint 3001. Average Accuracy: 87.99%2. Best Performing Procedure: ‘S/T/NECK/CHEST/ABD/PELV-/C+/CCSI’ with an accuracy of 97.54%.3. Interpretation: The model at this stage is already performing well,demonstrating its ability to correctly classify Pass vs Fail images foreach of the procedures with a high level of confidence.MedMAE-B Checkpoint 10001. Average Accuracy: 90.20%2. Best Performing Procedure: ‘CHEST/C-’ with an accuracy of99.82%.3. Interpretation: The increase in average accuracy from checkpoint300 to checkpoint 1000 highlights the positive impact of extendedtraining on the model’s generalization capability. This model is now544.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-Feven more reliable for classifying CT diagnostic medical images acrossdifferent procedures.The key findings from this experimental study can be listed as follows:1. Improved Generalization: The extended training has resulted ina noticeable improvement in the model’s ability to generalize acrossdifferent medical imaging tasks.2. High Performance on Specific Procedures: The model exhibitshigh performance on certain procedures, which might be due to thecharacteristics of the images in these procedures aligning well with thefeatures learned during the pre-training.By selecting these two checkpoints for evaluation, the study effectivelycaptures the progression of the model’s performance, showcasing the benefitsof extended training.MedMAE-F: End-to-end Fine-TuningBy employing end-to-end fine-tuning, we aim to leverage the genericrepresentations learned by MedMAE-B during its pre-training on the largemedical imaging dataset, while also allowing the model to adjust these rep-resentations to better suit the specific characteristics of the CT diagnosticimage QC task.ProcedureFine-tuningAccuracyCHEST/ABD/PELV/C+/CCSI 100HEAD/C- 81.27CHEST/C- 100CHEST/ABD/PELV/C+ 100ABD/PELV/C+ 99.9HEAD/C+ 71.04S/T/NECK/CHEST/ABD/PEL/C+/CCSI 99.44RENAL/C-/C+ 100Average 93.96Table 4.5: MedMAE-F end-to-end fine-tuning results for ‘dataset3’.The fine-tuning results presented in Table 4.5 help showcase the capabil-ities of MedMAE-F when subjected to additional training tailored to specific554.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-Ftasks. MedMAE-B Checkpoint 1000 is selected as the base model due toits superior performance in linear probing. An additional 1000 epochs on‘dataset3’ are carried out as part of the fine-tuning process. Here’s a detailedanalysis based on the experimental results:1. Average Accuracy Improvement: +3.76% compared to linearprobing results.2. 100% Accuracy Procedures: 4 out of the 8 procedures achieved aperfect classification accuracy.3. Lowest Performing Procedure: ‘HEAD/C+’, with an accuracyimprovement of approximately 8%.4.3.3 Comparison with Transfer Learning ResultsIn order to assess the performance improvement, we compare MedMAE’slinear probing and fine-tuning results with the VGG19 backbone discussedin Chapter 3 of this thesis.ProcedureDTL(VGG-19)LinearProbingFine-tuningAccuracyCHEST/ABD/PELV/C+/CCSI 55.6 96.7 100HEAD/C- 54.5 73 81.27CHEST/C- 55.3 99.8 100CHEST/ABD/PELV/C+ 53.9 99.1 100ABD/PELV/C+ 58.0 97.8 99.9HEAD/C+ 52.0 63.2 71.04S/T/NECK/CHEST/ABD/PEL/C+/CCSI 56.4 97.9 99.44RENAL/C-/C+ 55.0 94.2 100Average 55.1 90.2 93.96Table 4.6: Comparative results on ‘dataset3’.As demonstrated in Table 4.6, we selected one of the top-performingCNNmodels from our discussions in Chapter 3, VGG-19, as the backbone forour study, and evaluated its classification accuracy across eight different pro-cedures in dataset3. The results reveal an average accuracy of 55.1%, whichis lower compared to the performance of the MedMAE model. Specifically,MedMAE achieved a linear probing accuracy of 90.2% and a fine-tuningaccuracy of 93.96%. This contrast in accuracy underscores the capability of564.3. CASE STUDY: AUTOMATING QUALITY CONTROL USING MEDMAE-Fthe MedMAE model to generalize well on specialized downstream medicalimaging tasks, indicating its potential as a more effective tool in medicalimage classification. The key findings from this experimental study can belisted as follows:1. Enhanced Adaptability: Fine-tuning has made the model moreadaptable to specific medical imaging tasks, as indicated by the im-provement in average accuracy.2. Perfect Classification: Achieving 100% accuracy on 4 out of 8datasets showcases the model’s capability to accurately distinguishbetween Pass and Fail images in these specific procedures.3. Performance Improvement: Every procedure shows some level ofimprovement, demonstrating that fine-tuning has universally benefit-ted the model’s performance.4. Notable Improvement in Lower Performing Procedures: Thesignificant improvement in the ‘HEAD/C+’ procedure indicates thatfine-tuning has benefitted procedures where the base model might haveinitially struggled.The fine-tuning results clearly demonstrate the value of additional, task-specific training for pre-trained models like MedMAE-B. By fine-tuning on‘dataset3’, the model has not only improved its average performance acrossall procedures but has also achieved perfect classification accuracy in half ofthe evaluated datasets.57 a) b) c) 4.4. VIRTUAL REALITY EVALUATION OF MEDMAE4.4 Virtual Reality Evaluation of MedMAETo assess MedMAE’s reconstruction capabilities, we created an innova-tive Virtual Reality (VR) evaluation tool using Unity software. This VRtool enables simultaneous upload and display of multiple images within animmersive environment. Users, equipped with a VR headset, can enter a vir-tual space where they can interact with and annotate the images as if theywere images in the real world. This immersive experience allows for a closerand more detailed examination of the reconstructed images, enhancing theobserver’s ability to identify subtle differences that may not be as apparentin a conventional 2D view. The aim is to leverage the immersive abilityof VR to provide a more intuitive and insightful evaluation of MedMAE’sperformance.To effectively evaluate MedMAE’s image reconstruction capability, weadopt a three-image comparison strategy within a VR framework, as show-cased in Figure 4.11. The selected images include the original input image,the reconstruction output from the untrained MedMAE-B, and the out-put following extensive training on a large medical imaging dataset. Uponuploading these images into the VR tool within the Unity environment,an observer equipped with a VR headset is invited to step into this virtualspace. The immersive nature of the VR setting allows the observer to engagewith the images more interactively and intuitively. In this 3D environment,the observer examines the differences between the original and reconstructedimages and employs manual annotations to highlight distinctions, as demon-strated by the blue circles drawn in Figure 4.11. This innovative approachaims to reveal the minute differences and details that might otherwise beoverlooked in a standard 2D image analysis, thus providing a more compre-hensive understanding of MedMAE-B’s performance enhancements throughtraining.Through this VR tool, we envision capturing the innovative intersectionof medical expertise with cutting-edge virtual reality technology, represent-ing the future of medical diagnostics and patient care. Our long-term objec-tive is to develop a VR tool designed to offer 3D visualization and editingcapabilities for medical images. This tool will not only assist in detailed ex-amination but also enable precise modifications within a three-dimensionalspace, significantly advancing the way medical professionals interact withand analyze diagnostic imagery.594.5. CONCLUSION4.5 ConclusionIn this chapter, we have introduced MedMAE, a state-of-the-art back-bone tailored for medical imaging applications built upon the vision trans-former architecture. MedMAE has undergone extensive pre-training on adiverse dataset of medical images, leveraging the advantages of SSL. This in-novative approach has enabled us to bypass the necessity for extensive andwell-labelled data, enabling the model’s application across a multitude ofmedical imaging applications. We have evaluated MedMAE’s performancethrough a case study aimed at automating the QC process in medical imag-ing. The findings from this study are promising, indicating that MedMAEexhibits adaptability to specialized downstream tasks, all while maintainingefficiency in terms of computational time and memory usage. As part offuture work, we aim to collect more data that will serve as the foundationfor extended pre-training. Our final objective is to make MedMAE publiclyaccessible to expand research in the medical imaging domain.60Chapter 5ConclusionAutomating QC for diagnostic images is a new area in medical researchthat needs more attention. There are many benefits to automating QCprocesses. It can improve the healthcare system, make it more efficient,and lead to major advancements in how we diagnose and treat patients.Currently, QC relies a lot on manual checks by medical professionals, whichcan be time-consuming and sometimes not accurate. By using automatedmodels, we can make the process faster and more reliable. To this extent,this thesis aimed to evaluate the potential of automating CT diagnosticimage QC by leveraging AI-based learning approaches.We began by exploring the potential of automating CT QC proceduresby leveraging transfer learning across different DL models. Our results indi-cated that InceptionResNetV2 outperformed other models, attaining a 99%accuracy rate on the in-domain dataset. However, when subjected to out-of-domain evaluation, the model’s accuracy dropped to 49.43%, highlightinga critical gap in robustness and adaptability. This finding underscores thenecessity for a more resilient and versatile DL backbone that can efficientlyhandle various scenarios in medical imaging.In response to the challenges identified, we introduced MedMAE-B, aself-supervised learning backbone tailored for medical image applications.The design philosophy behind MedMAE-B is to create a versatile and re-silient model that can operate efficiently across a variety of medical imagingtasks. By leveraging self-supervised learning, MedMAE-B can serve as arobust foundation for fine-tuning on smaller, labelled datasets, addressingthe common issue of data scarcity in the medical domain. This approachaims to enhance performance and adaptability in real-world medical imagingscenarios.We also presented MedMAE-F, an end-to-end SSL model tailored fordownstream medical imaging applications. To assess the resilience andadaptability of MedMAE-F, we employed it as a potential solution for au-tomating QC in medical imaging. The outcomes of our evaluation demon-strate that MedMAE-F exhibits promising capabilities, making it an appro-priate choice for specialized downstream applications in the medical field.61This reflects MedMAE-F’s potential to enhance automated QC processes,contributing to more efficient and reliable medical imaging practices. Toassess the performance of MedMAE-B further, we conducted an evaluationusing virtual reality to assess the differences between the original medicalimages and their reconstructions produced by MedMAE. The goal was todetermine if these differences could be effectively highlighted and assessedin an immersive virtual reality setting, providing a novel perspective andpotentially finding subtle discrepancies that may not be easily noticeablein traditional 2D image evaluations. This approach demonstrates our com-mitment to exploring innovative methodologies to ensure the accuracy andreliability of our model in various practical applications.Future Work: An effective area of future work in this domain could seea combination of CNN and SSL-based approaches to achieve better adapt-ability and generalization in automating medical imaging tasks. Moreover,investigating alternative SSL strategies as potential backbones for develop-ing pre-trained models tailored to medical image applications could yieldbeneficial results. Different SSL paradigms offer diverse ways of learningfrom unlabeled data, and exploring these could lead to the discovery ofmore robust and efficient models for medical image analysis.We strongly advocate for conducting comparative, application-focusedstudies to further assess and explore the capabilities of our proposed Med-MAE model. Notably, in the linear probing and fine-tuning assessments,MedMAE exhibited suboptimal performance on procedures involving the‘HEAD.’ This limitation may stem from the absence of ‘HEAD’ scans in thepre-training phase. To address this, it is advisable to incorporate a diverserange of procedures, including those previously missing, in the pre-trainingof MedMAE. Subsequent to this pre-training, a thorough re-evaluation shouldbe undertaken to compare against the existing results, thereby quantifyingthe impact of this expanded training set.However, a significant challenge in this research avenue is the scarcityof openly available, relevant medical procedures suitable for training deeplearning models. To overcome this, one potential strategy involves leverag-ing CT scans from IHA’s PACS for the pretext task of training MedMAE.A comprehensive comparative study is recommended, comparing the down-stream task performance before and after training with the PACS data.Such a study would provide valuable insights into the model’s adaptabilityand effectiveness when exposed to a more varied dataset, thereby enhancingits practical applicability and robustness in real-world medical settings.62Bibliography[1] MAH Akhand, Shuvendu Roy, Nazmul Siddique, Md Abdus SamadKamal, and Tetsuya Shimamura. “Facial emotion recognition us-ing transfer learning in the deep CNN”. In: Electronics 10.9 (2021),p. 1036.[2] Abdulkadir Albayrak. “Classification of analyzable metaphase imagesusing transfer learning and fine tuning”. In: Medical & BiologicalEngineering & Computing 60 (2022), pp. 239–248.[3] Fouzia Altaf, Syed MS Islam, and Naeem Khalid Janjua. “A novelaugmented deep transfer learning for classification of COVID-19 andother thoracic diseases from X-rays”. In: Neural Computing and Ap-plications 33.20 (2021), pp. 14037–14048.[4] Laith Alzubaidi, Jorge Santamar´ıa, Mohamed Manoufali, BeadaaMohammed, Mohammed A Fadhel, Jinglan Zhang, Ali H Al-Timemy,Omran Al-Shamma, and Ye Duan. “MedNet: pre-trained convolu-tional neural network model for the medical imaging tasks”. In: arXivpreprint arXiv:2110.06512 (2021).[5] John Arevalo, Fabio A Gonza´lez, Rau´l Ramos-Polla´n, Jose L Oliveira,and Miguel Angel Guevara Lopez. “Representation learning for mam-mography mass lesion classification with convolutional neural net-works”. In: Computer methods and programs in biomedicine 127 (2016),pp. 248–257.[6] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski,Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.“Self-supervised learning from images with a joint-embedding predic-tive architecture”. In: Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition. 2023, pp. 15619–15629.63BIBLIOGRAPHY[7] Betul Ay, Beyda Tasar, Zeynep Utlu, Kevser Ay, and Galip Ay-din. “Deep transfer learning-based visual classification of pressureinjuries stages”. In: Neural Computing and Applications 34.18 (2022),pp. 16157–16168.[8] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. “Beit: Bert pre-training of image transformers”. In: arXiv preprint arXiv:2106.08254(2021).[9] Pedro RAS Bassi and Romis Attux. “A deep convolutional neuralnetwork for COVID-19 detection using chest X-rays”. In: Researchon Biomedical Engineering (2021), pp. 1–10.[10] Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong,Jun Li, and Xiangyu Zhang. “Reversible column networks”. In: arXivpreprint arXiv:2212.11696 (2022).[11] Sergio Cantero-Chinchilla, Paul D Wilcox, and Anthony J Croxford.“A deep learning based methodology for artefact identification andsuppression with application to ultrasonic images”. In: NDT & EInternational 126 (2022), p. 102575.[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Herve´ Je´gou, JulienMairal, Piotr Bojanowski, and Armand Joulin. “Emerging proper-ties in self-supervised vision transformers”. In: Proceedings of theIEEE/CVF international conference on computer vision. 2021, pp. 9650–9660.[13] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu.“Contrastive learning of global and local features for medical imagesegmentation with limited annotations”. In: Advances in neural in-formation processing systems 33 (2020), pp. 12546–12558.[14] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Mi-chitaka Fujiwara, and Daniel Rueckert. “Self-supervised learning formedical image analysis using image context restoration”. In: MedicalImage Analysis 58 (2019), p. 101539. issn: 1361-8415. doi: https://doi.org/10.1016/j.media.2019.101539. url: https://www.sciencedirect.com/science/article/pii/S1361841518304699.64BIBLIOGRAPHY[15] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hin-ton. “Self-Supervised Learning: Generative or Contrastive”. In: IEEETransactions on Neural Networks and Learning Systems 32.4 (2021),pp. 1485–1508. doi: 10.1109/TNNLS.2020.3028904.[16] Mehdi Cherti and Jenia Jitsev. “Effect of pre-training scale on intra-and inter-domain, full and few-shot transfer learning for natural andX-ray chest images”. In: 2022 International Joint Conference on Neu-ral Networks (IJCNN). IEEE. 2022, pp. 1–9.[17] Franc¸ois Chollet. “Xception: Deep learning with depthwise separableconvolutions”. In: Proceedings of the IEEE conference on computervision and pattern recognition (CVPR) (2017).[18] Nihad K Chowdhury, Md Muhtadir Rahman, and Muhammad AshadKabir. “PDCOVIDNet: a parallel-dilated convolutional neural net-work architecture for detecting COVID-19 from chest X-ray images”.In: Health information science and systems 8.1 (2020), p. 27.[19] Safety Code. “Radiation Protection in Radiology—Large Facilities”.In: (2008).[20] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. “Ran-daugment: Practical automated data augmentation with a reducedsearch space”. In: Proceedings of the IEEE/CVF conference on com-puter vision and pattern recognition workshops. 2020, pp. 702–703.[21] N Narayan Das, Naresh Kumar, Manjit Kaur, Vijay Kumar, and Dil-bag Singh. “Automated deep transfer learning-based approach for de-tection of COVID-19 infection in chest X-rays”. In: Irbm 43.2 (2022),pp. 114–119.[22] S Deepak and PM Ameer. “Brain tumor classification using deepCNN features via transfer learning”. In: Computers in biology andmedicine 111 (2019), p. 103345.[23] Azimeh NV Dehkordi and Saeideh Koohestani. “The Influence ofSignal to Noise Ratio on the Pharmacokinetic Analysis in DCE-MRIStudies”. In: Frontiers in Biomedical Technologies (2019).65BIBLIOGRAPHY[24] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. “Imagenet: A large-scale hierarchical image database”. In: 2009IEEE conference on computer vision and pattern recognition. Ieee.2009, pp. 248–255.[25] C Dillon, Jessica Clements, Diana Cody, Dustin Gress, Kalpana Kanal,James Kofler, Michael F McNitt-Gray, James Norweck, Doug Pfeif-fer, Thomas G Ruckdeschel, et al. “Computed tomography qualitycontrol manual”. In: Reston: American College of Radiology (2017).[26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis-senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, MatthiasMinderer, Georg Heigold, Sylvain Gelly, et al. “An image is worth16x16 words: Transformers for image recognition at scale”. In: arXivpreprint arXiv:2010.11929 (2020).[27] Irene Fantini, Leticia Rittner, Clarissa Yasuda, and Roberto Lotufo.“Automatic detection of motion artifacts on MRI using Deep CNN”.In: 2018 International Workshop on Pattern Recognition in Neu-roimaging (PRNI). IEEE. 2018, pp. 1–4.[28] Biraja Ghoshal and Allan Tucker. “Estimating uncertainty and inter-pretability in deep learning for coronavirus (COVID-19) detection”.In: arXiv preprint arXiv:2003.10769 (2020).[29] Mark S Graham, Ivana Drobnjak, and Hui Zhang. “A supervisedlearning approach for diffusion MRI quality control with minimaltraining data”. In: NeuroImage 178 (2018), pp. 668–676.[30] Md Kamrul Hasan, Md Ashraful Alam, Md Toufick E Elahi, Shid-hartho Roy, and Sifat Redwan Wahid. “CVR-Net: A deep convolu-tional neural network for coronavirus recognition from chest radiog-raphy images”. In: arXiv preprint arXiv:2007.11993 (2020).[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dolla´r, andRoss Girshick. “Masked autoencoders are scalable vision learners”.In: Proceedings of the IEEE/CVF conference on computer vision andpattern recognition. 2022, pp. 16000–16009.66BIBLIOGRAPHY[32] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.“Momentum contrast for unsupervised visual representation learn-ing”. In: Proceedings of the IEEE/CVF conference on computer vi-sion and pattern recognition. 2020, pp. 9729–9738.[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deepresidual learning for image recognition”. In: Proceedings of the IEEEconference on computer vision and pattern recognition (CVPR) (2016).[34] Wayne R Hedrick, Michael A Markovic, James H Short, and ChidoD Vera. “Computed tomography artifact created by air in the x-raytube oil”. In: Journal of Computer Assisted Tomography 40.1 (2016),pp. 67–70.[35] Shih-Cheng Huang, Anuj Pareek, Malte Jensen, Matthew P Lungren,Serena Yeung, and Akshay S Chaudhari. “Self-supervised learning formedical image classification: a systematic review and implementationguidelines”. In: NPJ Digital Medicine 6.1 (2023), p. 74.[36] Mohammadreza Iman, Hamid Reza Arabnia, and Khaled Rasheed.“A review of deep transfer learning and recent advancements”. In:Technologies 11.2 (2023), p. 40.[37] Riccardo M Inciardi, Laura Lupi, Gregorio Zaccone, Leonardo Italia,Michela Raffo, Daniela Tomasoni, Dario S Cani, Manuel Cerini, Da-vide Farina, Emanuele Gavazzi, et al. “Cardiac involvement in a pa-tient with coronavirus disease 2019 (COVID-19)”. In: JAMA cardi-ology 5.7 (2020), pp. 819–824.[38] Aayush Jaiswal, Neha Gianchandani, Dilbag Singh, Vijay Kumar,and Manjit Kaur. “Classification of the COVID-19 infected patientsusing DenseNet201 based deep transfer learning”. In: Journal of Biomolec-ular Structure and Dynamics 39.15 (2021), pp. 5682–5689.[39] Ananya Jana, Hui Qu, Carlos DMinacapelli, Carolyn Catalano, VinodRustgi, and Dimitris Metaxas. “Liver fibrosis and nas scoring from ctimages using self-supervised learning and texture encoding”. In: 2021IEEE 18th International Symposium on Biomedical Imaging (ISBI).IEEE. 2021, pp. 1553–1557.67BIBLIOGRAPHY[40] Jinseong Jang, Kihun Bang, Hanbyol Jang, Dosik Hwang, and Alzheimer’sDisease Neuroimaging Initiative. “Quality evaluation of no-referenceMR images using multidirectional filters and image statistics”. In:Magnetic resonance in medicine 80.3 (2018), pp. 914–924.[41] Wonsik Jung, Da-Woon Heo, Eunjin Jeon, Jaein Lee, and Heung-Il Suk. “Inter-regional high-level relation learning from functionalconnectivity via self-supervision”. In: Medical Image Computing andComputer Assisted Intervention–MICCAI 2021: 24th InternationalConference, Strasbourg, France, September 27–October 1, 2021, Pro-ceedings, Part II 24. Springer. 2021, pp. 284–293.[42] Michiel Kallenberg, Kersten Petersen, Mads Nielsen, Andrew Y Ng,Pengfei Diao, Christian Igel, Celine M Vachon, Katharina Holland,Rikke Rass Winkel, Nico Karssemeijer, et al. “Unsupervised deeplearning applied to breast density segmentation and mammographicrisk scoring”. In: IEEE transactions on medical imaging 35.5 (2016),pp. 1322–1331.[43] Hengyuan Kang, Liming Xia, Fuhua Yan, Zhibin Wan, Feng Shi,Huan Yuan, Huiting Jiang, Dijia Wu, He Sui, Changqing Zhang, et al.“Diagnosis of coronavirus disease 2019 (COVID-19) with structuredlatent multi-view representation learning”. In: IEEE transactions onmedical imaging 39.8 (2020), pp. 2606–2614.[44] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.“Bert: Pre-training of deep bidirectional transformers for languageunderstanding”. In: Proceedings of naacL-HLT. Vol. 1. 2019, p. 2.[45] Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CSValentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang,Xiaokang Wu, Fangbing Yan, et al. “Identifying medical diagnosesand treatable diseases by image-based deep learning”. In: cell 172.5(2018), pp. 1122–1131.[46] Nikhil Ketkar and Nikhil Ketkar. “Introduction to keras”. In: Deeplearning with python: a hands-on introduction (2017), pp. 97–111.[47] Hee E Kim, Alejandro Cosa-Linan, Nandhini Santhanam, MahboubehJannesari, Mate E Maros, and Thomas Ganslandt. “Transfer learn-68BIBLIOGRAPHYing for medical image classification: a literature review”. In: BMCmedical imaging 22.1 (2022), p. 69.[48] Serkan Kiranyaz, Turker Ince, and Moncef Gabbouj. “Real-time patient-specific ECG classification by 1-D convolutional neural networks”. In:IEEE Transactions on Biomedical Engineering 63.3 (2015), pp. 664–675.[49] Sushil Kumar et al. “MCFT-CNN: Malware classification with fine-tune convolution neural networks using traditional and transfer learn-ing in Internet of Things”. In: Future Generation Computer Systems125 (2021), pp. 334–351.[50] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao,Xiyang Dai, Lu Yuan, and Jianfeng Gao. “Efficient self-supervisedvision transformers for representation learning”. In: arXiv preprintarXiv:2106.09785 (2021).[51] Dasheng Li, Dawei Wang, Jianping Dong, Nana Wang, He Huang,Haiwang Xu, and Chen Xia. “False-negative results of real-time reverse-transcriptase polymerase chain reaction for severe acute respiratorysyndrome coronavirus 2: role of deep-learning-based CT diagnosisand insights from two cases”. In: Korean journal of radiology 21.4(2020), pp. 505–508.[52] Xiaomeng Li, Xiaowei Hu, Xiaojuan Qi, Lequan Yu, Wei Zhao, Pheng-Ann Heng, and Lei Xing. “Rotation-oriented collaborative self-supervisedlearning for retinal disease diagnosis”. In: IEEE Transactions onMedical Imaging 40.9 (2021), pp. 2284–2294.[53] Chengcheng Liu, Mengyun Qiao, Fei Jiang, Yi Guo, Zhendong Jin,and Yuanyuan Wang. “TN-USMA Net: Triple normalization-basedgastrointestinal stromal tumors classification on multicenter EUS im-ages with ultrasound-specific pretraining and meta attention”. In:Medical Physics 48.11 (2021), pp. 7199–7214.[54] Ming Y Lu, Richard J Chen, and Faisal Mahmood. “Semi-supervisedbreast cancer histology classification using deep multiple instancelearning and contrast predictive coding (conference presentation)”.69BIBLIOGRAPHYIn: Medical imaging 2020: digital pathology. Vol. 11320. SPIE. 2020,113200J.[55] Halgurd S Maghdid, Aras T Asaad, Kayhan Zrar Ghafoor, Ali SafaaSadiq, Seyedali Mirjalili, and Muhammad Khurram Khan. “Diagnos-ing COVID-19 pneumonia from X-ray and CT images using deeplearning and transfer learning algorithms”. In: Multimodal image ex-ploitation and learning 2021. Vol. 11734. SPIE. 2021, pp. 99–110.[56] Darcy Mason. “SU-E-T-33: pydicom: an open source DICOM li-brary”. In: Medical Physics 38.6Part10 (2011), pp. 3493–3493.[57] Donald W McRobbie, Scott Semple, and Anna Pauline Barnes. Qual-ity Control and Artefacts in Magnetic Resonance Imaging (update ofIPEM Report 80). Institute of Physics and Engineering in Medicine,2017.[58] Shervin Minaee, Rahele Kafieh, Milan Sonka, Shakib Yazdani, andGhazaleh Jamalipour Soufi. “Deep-COVID: Predicting COVID-19from chest X-ray images using deep transfer learning”. In: Medicalimage analysis 65 (2020), p. 101794.[59] Mohammad Amin Morid, Alireza Borjali, and Guilherme Del Fiol. “Ascoping review of transfer learning research on medical image analysisusing ImageNet”. In: Computers in biology and medicine 128 (2021),p. 104115.[60] Dominik Mu¨ller, In˜aki Soto-Rey, and Frank Kramer. “Towards aguideline for evaluation metrics in medical image segmentation”. In:BMC Research Notes 15.1 (2022), pp. 1–8.[61] Ebrahim Nasr-Esfahani, Shadrokh Samavi, Nader Karimi, S Mo-hamad R Soroushmehr, Mohammad H Jafari, Kevin Ward, and Kay-van Najarian. “Melanoma detection by analysis of clinical imagesusing convolutional neural network”. In: 2016 38th Annual Interna-tional Conference of the IEEE Engineering in Medicine and BiologySociety (EMBC). IEEE. 2016, pp. 1373–1376.[62] M Nessaiver. “Range of results from over 534 ACR-mandated annualMRI performance evaluations on over 204 magnets from 8 vendors70BIBLIOGRAPHYspanning a 10-year period”. In: URL: http://www. simplyphysics.com/MRI% 20QA% 20RSN A 202011 (2011).[63] Xuan-Bac Nguyen, Guee Sang Lee, Soo Hyung Kim, and HyungJeong Yang. “Self-Supervised Learning Based on Spatial Awarenessfor Medical Image Analysis”. In: IEEE Access 8 (2020), pp. 162973–162981. doi: 10.1109/ACCESS.2020.3021469.[64] Patrik Nowik, Robert Bujila, Gavin Poludniowski, and Annette Frans-son. “Quality control of CT systems by automated monitoring of keyperformance indicators: a two-year study”. In: Journal of applied clin-ical medical physics 16.4 (2015), pp. 254–265.[65] Keiron O’Shea and Ryan Nash. “An introduction to convolutionalneural networks”. In: arXiv preprint arXiv:1511.08458 (2015).[66] Kriti Ohri and Mukesh Kumar. “Review on self-supervised imagerecognition using deep neural networks”. In: Knowledge-Based Sys-tems 224 (2021), p. 107090. issn: 0950-7051.[67] Kriti Ohri and Mukesh Kumar. “Review on self-supervised imagerecognition using deep neural networks”. In: Knowledge-Based Sys-tems 224 (2021), p. 107090.[68] Ilkay Oksuz. “Brain MRI artefact detection and correction using con-volutional neural networks”. In: Computer Methods and Programs inBiomedicine 199 (2021), p. 105909.[69] Emin Orhan, Vaibhav Gupta, and Brenden M Lake. “Self-supervisedlearning through the eyes of a child”. In: Advances in Neural Infor-mation Processing Systems 33 (2020), pp. 9960–9971.[70] Junghoon Park, Il-Youp Kwak, and Changwon Lim. “A deep learn-ing model with self-supervised learning and attention mechanism forCOVID-19 diagnosis using chest X-ray images”. In: Electronics 10.16(2021), p. 1996.[71] Todd B Parrish, Darren R Gitelman, Kevin S LaBar, and M-MarselMesulam. “Impact of signal-to-noise on functional MRI”. In: Mag-netic Resonance in Medicine: An Official Journal of the International71BIBLIOGRAPHYSociety for Magnetic Resonance in Medicine 44.6 (2000), pp. 925–932.[72] Yadunath Pathak, Prashant Kumar Shukla, Akhilesh Tiwari, ShaliniStalin, and Saurabh Singh. “Deep transfer learning based classifica-tion model for COVID-19 disease”. In: Irbm 43.2 (2022), pp. 87–92.[73] Juha I Peltonen, Teemu Ma¨kela¨, and Eero Salli. “MRI quality as-surance based on 3D FLAIR brain images”. In: Magnetic ResonanceMaterials in Physics, Biology and Medicine 31 (2018), pp. 689–699.[74] Nick A Phillips, Pranav Rajpurkar, Mark Sabini, Rayan Krishnan,Sharon Zhou, Anuj Pareek, Nguyet Minh Phu, Chris Wang, MuditJain, Nguyen Duong Du, et al. “CheXphoto: 10,000+ photos andtransformations of chest X-rays for benchmarking deep learning ro-bustness”. In: Machine Learning for Health. PMLR. 2020, pp. 318–327.[75] Neo Poyiadji, Gassan Shahin, Daniel Noujaim, Michael Stone, SureshPatel, and Brent Griffith. “COVID-19–associated acute hemorrhagicnecrotizing encephalopathy: imaging features”. In: Radiology 296.2(2020), E119–E120.[76] B. Tanaka R. Walsh and T. Bjarnason. “CT Image Quality Test-ing: Human Vs. Automated Computer Vision System”. In: MedicalPhysics 46 (2019), p. 5392.[77] American College of Radiology et al. “Magnetic resonance imagingquality control manual”. In: Reston, VA: American College of Radi-ology (2015).[78] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, HershelMehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, KatieShpanskaya, et al. “Chexnet: Radiologist-level pneumonia detectionon chest x-rays with deep learning”. In: arXiv preprint arXiv:1711.05225(2017).[79] Rami Al-Rfou, Guillaume Alain, Amjad Almahairi, Christof Anger-mueller, Dzmitry Bahdanau, Nicolas Ballas, Fre´de´ric Bastien, JustinBayer, Anatoly Belikov, Alexander Belopolsky, et al. “Theano: A72BIBLIOGRAPHYPython framework for fast computation of mathematical expressions”.In: arXiv e-prints (2016), arXiv–1605.[80] Joel D Rubenstein, JG Li, S Majumdar, and RM Henkelman. “Imageresolution and signal-to-noise ratio requirements for MR imaging ofdegenerative cartilage.” In: AJR. American journal of roentgenology169.4 (1997), pp. 1089–1096.[81] Matthia Sabatelli, Mike Kestemont, Walter Daelemans, and PierreGeurts. “Deep transfer learning for art classification problems”. In:Proceedings Of The European conference on computer vision (ECCV)workshops. 2018, pp. 0–0.[82] Zahra Riahi Samani, Jacob Antony Alappatt, Drew Parker, AbdolAziz Ould Ismail, and Ragini Verma. “QC-Automator: Deep learning-based automated quality control for diffusion mr images”. In: Fron-tiers in neuroscience 13 (2020), p. 1456.[83] DR Sarvamangala and Raghavendra V Kulkarni. “Convolutional neu-ral networks in medical image understanding: a survey”. In: Evolu-tionary intelligence 15.1 (2022), pp. 1–22.[84] Saeed Shurrab and Rehab Duwairi. “Self-supervised learning meth-ods and applications in medical imaging analysis: A survey”. In:PeerJ Computer Science 8 (2022), e1045.[85] Karen Simonyan and Andrew Zisserman. “Very deep convolutionalnetworks for large-scale image recognition”. In: Proceedings of theInternational Conference on Learning Representations (ICLR). 2015.[86] Dilbag Singh, Vijay Kumar, Vaishali, and Manjit Kaur. “Classifi-cation of COVID-19 patients from chest CT images using multi-objective differential evolution–based convolutional neural networks”.In: European Journal of Clinical Microbiology & Infectious Diseases39 (2020), pp. 1379–1389.[87] Bin Song, Weiquan Tan, Yue Xu, Taihui Yu, Weiping Li, Zhong Chen,Rui Yang, Jingyi Hou, and Yunfeng Zhou. “3D-MRI combined withsignal-to-noise ratio measurement can improve the diagnostic accu-racy and sensitivity in evaluating meniscal healing status after menis-73BIBLIOGRAPHYcal repair”. In: Knee Surgery, Sports Traumatology, Arthroscopy 27(2019), pp. 177–188.[88] Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar.“Moco pretraining improves representation and transferability of chestx-ray models”. In:Medical Imaging with Deep Learning. PMLR. 2021,pp. 728–744.[89] Anuroop Sriram, Matthew Muckley, Koustuv Sinha, Farah Shamout,Joelle Pineau, Krzysztof J Geras, Lea Azour, Yindalon Aphinyanaphongs,Nafissa Yakubova, and William Moore. “Covid-19 prognosis via self-supervised representation learning and multi-image prediction”. In:arXiv preprint arXiv:2101.04909 (2021).[90] Sheeba J Sujit, Ivan Coronado, Arash Kamali, Ponnada A Narayana,and Refaat E Gabr. “Automated image quality evaluation of struc-tural brain MRI using an ensemble of deep learning networks”. In:Journal of Magnetic Resonance Imaging 50.4 (2019), pp. 1260–1267.[91] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi.“Inception-v4, Inception-ResNet and the impact of residual connec-tions on learning”. In: Proceedings of the Thirty-First AAAI Confer-ence on Artificial Intelligence (AAAI-17) (2017).[92] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, ScottReed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, andAndrew Rabinovich. “Going deeper with convolutions”. In: arXivpreprint arXiv:1409.4842 (2014).[93] Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, ThomasGaertner, Benjamin Bergner, and Christoph Lippert. “3d self-supervisedmethods for medical imaging”. In: Advances in neural informationprocessing systems 33 (2020), pp. 18158–18172.[94] Muhammed Talo, Ulas Baran Baloglu, O¨zal Yıldırım, and U Rajen-dra Acharya. “Application of deep transfer learning for automatedbrain abnormality classification using MR images”. In: Cognitive Sys-tems Research 54 (2019), pp. 176–188.74BIBLIOGRAPHY[95] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang,and Chunfang Liu. “A Survey on Deep Transfer Learning”. In: Ar-tificial Neural Networks and Machine Learning – ICANN 2018. Ed.by Veˇra Ku˚rkova´, Yannis Manolopoulos, Barbara Hammer, LazarosIliadis, and Ilias Maglogiannis. Cham: Springer International Pub-lishing, 2018, pp. 270–279. isbn: 978-3-030-01424-7.[96] Mingxing Tan and Quoc V. Le. “EfficientNet: Rethinking model scal-ing for convolutional neural networks”. In: Proceedings of the 36thInternational Conference on Machine Learning (ICML) (2019).[97] John Tracey, Laura Moss, and Jonathan Ashmore. “Application ofsynthetic data in the training of artificial intelligence for automatedquality assurance in magnetic resonance imaging”. In:Medical Physics(2023).[98] A Verones, S Wong, J Cheung, T Lee, and T Bjarnason. “AutomaticCT Air Bubble Artifact Detection in Routine QC Images”. In: MED-ICAL PHYSICS. Vol. 47. 6. WILEY 111 RIVER ST, HOBOKEN07030-5774, NJ USA. 2020, E515–E516.[99] Yutong Xie, Jianpeng Zhang, Zehui Liao, Yong Xia, and ChunhuaShen. “PGL: prior-guided local self-supervised learning for 3D medi-cal image segmentation”. In: arXiv preprint arXiv:2011.12640 (2020).[100] Howard Yanxon, James Weng, Hannah Parraga, Wenqian Xu, UtaRuett, and Nicholas Schwarz. “Artifact identification in X-ray diffrac-tion data using machine learning methods”. In: Journal of SynchrotronRadiation 30.1 (2023).[101] Qian Yu, Yuancheng Wang, Shan Huang, Songqiao Liu, Zhen Zhou,Shijun Zhang, Zhen Zhao, Yizhou Yu, Yi Yang, and Shenghong Ju.“Multicenter cohort study demonstrates more consolidation in upperlungs on initial CT increases the risk of adverse clinical outcome inCOVID-19 patients”. In: Theranostics 10.12 (2020), p. 5641.[102] Yuhai Yu, Hongfei Lin, Jiana Meng, Xiaocong Wei, Hai Guo, andZhehuan Zhao. “Deep transfer learning for modality classification ofmedical images”. In: Information 8.3 (2017), p. 91.75BIBLIOGRAPHY[103] Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan.“Volo: Vision outlooker for visual recognition”. In: IEEE transactionson pattern analysis and machine intelligence 45.5 (2022), pp. 6575–6586.[104] Min Zeng, Min Li, Zhihui Fei, Ying Yu, Yi Pan, and Jianxin Wang.“Automatic ICD-9 coding via deep transfer learning”. In: Neurocom-puting 324 (2019), pp. 43–50.[105] Hai Zhong, Jiaqi Wu, Wangyuan Zhao, Xiaowei Xu, Runping Hou,Lu Zhao, Ziheng Deng, Min Zhang, and Jun Zhao. “A Self-supervisedLearning Based Framework for Automatic Heart Failure Classifica-tion on Cine Cardiac Magnetic Resonance Image”. In: 2021 43rd An-nual International Conference of the IEEE Engineering in Medicine& Biology Society (EMBC). IEEE. 2021, pp. 2887–2890.[106] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, AlanYuille, and Tao Kong. “ibot: Image bert pre-training with onlinetokenizer”. In: arXiv preprint arXiv:2111.07832 (2021).76
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Braun, John, 1963-
Shehata, Mohamed S.
Automating computed tomography diagnostic image quality control using deep learning
Text