Science, Faculty of
DSpace
UBCV
Yang, Jenny
Master of Science - MSc
University of British Columbia
Pancreatic ductal adenocarcinoma (PDAC) is considered the most lethal common cancer, with the highest incidence-to-mortality ratio of any solid tumour. Molecular pathology studies and genomic analyses have improved our understandings of how PDAC develops and progresses, and there has been significant progress in treatment strategies for specific genomic alterations. One of these alterations is the NRG1 gene fusion, which has been found to be a rare, but potentially targetable oncogenic driver. To determine whether PDAC patients have an NRG1 gene fusion, we used convolutional neural networks (CNNs) to analyze digital whole slide images (WSIs) of cancer biopsies. In particular, we used histopathological H&E slides from the Personalized OncoGenomics program to train a deep CNN (VGG-16) framework that automatically classifies normal tissue, NRG1-fusion positive tumour tissue, and NRG1-fusion negative tumour tissue. We implemented the model in two-stages, where the first stage classifies normal from tumour tissue, and the second stage classifies the tumour tissue as being NRG1-fusion positive or negative. The model achieved accuracies of 86.5% and 76.0% for each stage, respectively, and an overall accuracy of 68.8%. Additionally, we found that PDAC cases with high expression of the NRG1 gene (93rd-98th percentile of TCGA PDAC cases) were being classified as NRG1-fusion positive, suggesting a possible correlation between NRG1 gene fusions and high parent gene expression. Finally, we attempted to understand the inner workings and decisions of our CNN model by analyzing internal feature maps. We found activation patterns that matched distinct histological features and compared them with a more traditional image segmentation approach. Overall, our findings demonstrate that deep CNNs have the potential to assist pathologists in detecting therapeutically actionable genomic markers.
USING CONVOLUTIONAL NEURAL NETWORKS TO PREDICTNRG1-FUSIONS IN PDAC BIOPSY IMAGESbyJenny YangB.A.Sc., The University of British Columbia, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THEREQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES(Bioinformatics)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)September 2020c© Jenny Yang, 2020The following individuals certify that they have read, and recommend to the Faculty ofGraduate and Postdoctoral Studies for acceptance, a thesis entitled:Using Convolutional Neural Networks to Predict NRG1-Fusions in PDAC BiopsyImagessubmitted by Jenny Yang in partial fulfillment of the requirements for the degree of Masterof Science in Bioinformatics.Examining Committee:Dr. Steven Jones, Professor, Department of Medical Genetics, UBCSupervisorDr. Elodie Portales-Casamar, Assistant Professor, Department of Pediatrics, UBCSupervisory Committee MemberDr. Stephen Yip, Associate Professor, Department of Pathology, UBCSupervisory Committee MemberiiAbstractPancreatic ductal adenocarcinoma (PDAC) is considered the most lethal commoncancer, with the highest incidence-to-mortality ratio of any solid tumour. Molecular pathol-ogy studies and genomic analyses have improved our understandings of how PDAC developsand progresses, and there has been significant progress in treatment strategies for specificgenomic alterations. One of these alterations is the NRG1 gene fusion, which has been foundto be a rare, but potentially targetable oncogenic driver. To determine whether PDAC pa-tients have an NRG1 gene fusion, we used convolutional neural networks (CNNs) to analyzedigital whole slide images (WSIs) of cancer biopsies. In particular, we used histopathologicalH&E slides from the Personalized OncoGenomics program to train a deep CNN (VGG-16)framework that automatically classifies normal tissue, NRG1-fusion positive tumour tissue,and NRG1-fusion negative tumour tissue. We implemented the model in two-stages, wherethe first stage classifies normal from tumour tissue, and the second stage classifies the tu-mour tissue as being NRG1-fusion positive or negative. The model achieved accuracies of86.5% and 76.0% for each stage, respectively, and an overall accuracy of 68.8%. Additionally,we found that PDAC cases with high expression of the NRG1 gene (93rd-98th percentile ofTCGA PDAC cases) were being classified as NRG1-fusion positive, suggesting a possiblecorrelation between NRG1 gene fusions and high parent gene expression. Finally, we at-tempted to understand the inner workings and decisions of our CNN model by analyzinginternal feature maps. We found activation patterns that matched distinct histological fea-tures and compared them with a more traditional image segmentation approach. Overall, ourfindings demonstrate that deep CNNs have the potential to assist pathologists in detectingtherapeutically actionable genomic markers.iiiLay SummaryPancreatic ductal adenocarcinoma (PDAC) is considered the most lethal commoncancer, as it is often diagnosed at an advanced stage. By using large-scale analyses tounderstand a tumour’s genetic landscape, treatment strategies can be tailored towards eachpatient’s unique profile. In this study, we focus on a particular alteration – the NRG1 genefusion, as PDAC patients with this alteration have been found to respond significantly andrapidly to certain lines of therapy. We present a method to identify patients with this marker,that uses a machine learning approach to analyze images of biopsied cancers. Our resultsshow promise that this type of method can be used quickly and accurately to detect geneticmarkers within a patient’s tumour. Thus, it can greatly benefit patients, as certain markersare crucial for selecting appropriate and tailored therapies.ivPrefaceI completed this work under the supervision of Dr. Steven Jones at the BCCancer Genome Sciences Centre as part of the Canadian Digital Technology Supercluster(CDTS) project. This work is approved by and conducted under the University of BritishColumbia Clinical Research Ethics Board (H19-02174). I designed the research approachesand implemented all the experiments myself. All the code was written by me, and anysoftware libraries, packages, and source code used/modified are acknowledged accordingly.The following parts of the project were not performed by me:• The slides were obtained from the POG project and PanGen project at BC Cancer.• The slides were digitally scanned by Simon Cheung.• The annotations were completed by Dr. Julia Naso.• The comparison of high NRG1 gene expression in TCGA PDAC cases was done by Dr.Cam Grisdale.• The predicted tumour content values were done by Luka Culibrk.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xviii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Biological Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1 Traditional Histopathology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Digital Histopathology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Targeted Therapy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Pancreatic Ductal Adenocarcinoma . . . . . . . . . . . . . . . . . . . . . . . 7vi2.4.1 NRG1 Gene Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Machine Learning Background . . . . . . . . . . . . . . . . . . . . . . . . . 93.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.1.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2 Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2.1 Structure of an ANN . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Convolutional Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 193.3.1 Structure of a CNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.3.2 VGG-16 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.4 Training a Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.4.1 Cost Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.4.2 Gradient-Based Optimization . . . . . . . . . . . . . . . . . . . . . . 273.4.3 Generalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.4.4 Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Related Works for Digital Histopathology in Oncology . . . . . . . . . . . 304.1 Digital Histopathology Databases and Imaging Tools . . . . . . . . . . . . . 304.2 Machine Learning for Cancer Diagnosis and Biomarker Analysis . . . . . . . 315 Data Generation and Normalization . . . . . . . . . . . . . . . . . . . . . . 335.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.2 Annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.3 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3.1 Patch Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3.2 Background Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 375.3.3 Focus Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.3.4 Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 Model Architecture and Performance . . . . . . . . . . . . . . . . . . . . . 416.1 Training and Test Dataset Split . . . . . . . . . . . . . . . . . . . . . . . . . 416.2 Model Selection and Hyperparameter Configuration . . . . . . . . . . . . . . 42vii6.3 Two-Stage Local Hierarchical Learning Framework . . . . . . . . . . . . . . 456.3.1 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456.3.2 Model Training Overview . . . . . . . . . . . . . . . . . . . . . . . . 466.3.3 Model Performance and Results . . . . . . . . . . . . . . . . . . . . . 486.4 Model Additions and Finetuning . . . . . . . . . . . . . . . . . . . . . . . . 516.4.1 Adding Control Liver Images . . . . . . . . . . . . . . . . . . . . . . 516.4.2 Finetuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546.5 Overall Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 566.5.1 Main Test Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566.5.2 Additional Test Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.5.3 NRG1 Gene Expression . . . . . . . . . . . . . . . . . . . . . . . . . 636.5.4 Tumour Heterogeneity . . . . . . . . . . . . . . . . . . . . . . . . . . 656.5.5 Tumour Purity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677 Visualization of CNN Model . . . . . . . . . . . . . . . . . . . . . . . . . . 697.1 Visualizing CNN Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697.2 Network Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 727.3 Comparison to Traditional Image Segmentation . . . . . . . . . . . . . . . . 758 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 778.1 Summary of Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 778.2 Limitations and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . 78Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81viiiList of Tables6.1 Training and test dataset split. . . . . . . . . . . . . . . . . . . . . . . . . . 416.2 Confusion matrix for normal vs tumour tissue classification. . . . . . . . . . 496.3 Model performance metrics for normal vs tumour tissue classification. . . . . 496.4 Confusion matrix for NRG1-fusion classification. . . . . . . . . . . . . . . . . 506.5 Model performance metrics for NRG1-fusion classification. . . . . . . . . . . 506.6 Confusion matrix for overall tissue classification. . . . . . . . . . . . . . . . . 516.7 Model performance metrics for overall classification. . . . . . . . . . . . . . . 516.8 Updated confusion matrix for normal vs tumour classification, after additionof control liver normal images in training. . . . . . . . . . . . . . . . . . . . 526.9 Updated model performance metrics for normal vs tumour classification, afteraddition of control liver normal images in training. . . . . . . . . . . . . . . 526.10 Updated confusion matrix for overall tissue classification, after addition ofcontrol liver normal images in training. . . . . . . . . . . . . . . . . . . . . . 536.11 Updated model performance metrics for overall classification, after additionof control liver normal images in training. . . . . . . . . . . . . . . . . . . . 536.12 Updated confusion matrix for normal vs tumour classification, after finetuning. 556.13 Updated model performance metrics for normal vs tumour classification, afterfinetuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.14 Updated confusion matrix for overall tissue classification, after finetuning. . . 566.15 Updated model performance metrics for normal vs tumour classification, afterfinetuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576.16 Patient-level classification results, for test data cases. . . . . . . . . . . . . . 576.17 Patient-level classification results, for training data cases. . . . . . . . . . . . 58ix6.18 Patient-level NRG1-fusion classification results for additional test data (Pan-Gen). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.19 Patient-level NRG1-fusion classification results for additional test data (POG). 637.1 Number of filters in the VGG-16 network. . . . . . . . . . . . . . . . . . . . 727.2 Most common highest activated filters for all test patches. . . . . . . . . . . 73xList of Figures3.1 Flat multi-class classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Local Hierarchical multi-class classification . . . . . . . . . . . . . . . . . . . 123.3 Comparison of biological and artificial neurons. . . . . . . . . . . . . . . . . 133.4 Schematic of a basic perceptron. . . . . . . . . . . . . . . . . . . . . . . . . . 143.5 Sigmoid Activation Function. . . . . . . . . . . . . . . . . . . . . . . . . . . 163.6 ReLU Activation Function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.7 Multilayer Perceptron. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.8 Example decision boundary for 2-dimensional classification. . . . . . . . . . . 193.9 Schematic of a basic CNN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.10 Image separated into RGB colour channels. . . . . . . . . . . . . . . . . . . . 213.11 5x5 image matrix and 3x3 filter matrix example. . . . . . . . . . . . . . . . . 223.12 Filter moving across image to obtain convolved feature. . . . . . . . . . . . . 223.13 Example of a flipped filter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.14 Comparison of full vs local connectivity. . . . . . . . . . . . . . . . . . . . . 243.15 Max-pooling with a 2x2 window size. . . . . . . . . . . . . . . . . . . . . . . 253.16 Schematic of the VGG-16 model. . . . . . . . . . . . . . . . . . . . . . . . . 265.1 Example of a WSI and its accompanying annotations. . . . . . . . . . . . . . 345.2 Examples of patch images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3 Example of differing tumour to normal tissue area on WSIs. . . . . . . . . . 365.4 Examples of patches with < 30% blank space, that were not discarded. . . . 375.5 Examples of patches with > 30% blank space, that were discarded. . . . . . 375.6 Examples of blurry patches that were discarded. . . . . . . . . . . . . . . . . 385.7 Examples of patch images from different WSIs. . . . . . . . . . . . . . . . . . 38xi5.8 Example of deconvolved H&E stains in from a patch. . . . . . . . . . . . . . 395.9 Examples of normalized patch images from different WSIs. . . . . . . . . . . 395.10 Histograms comparing RGB pixel distribution before and after normalization. 406.1 Comparing training and validation accuracies for different CNN models. . . . 436.2 Comparing training and validation accuracies for different optimizers. . . . . 446.3 Hierarchical model architecture. . . . . . . . . . . . . . . . . . . . . . . . . . 456.4 Transfer learning for feature extraction. . . . . . . . . . . . . . . . . . . . . . 476.5 Examples of augmented images. . . . . . . . . . . . . . . . . . . . . . . . . . 476.6 Training accuracy and loss for normal vs tumour classifier. . . . . . . . . . . 486.7 Training accuracy and loss for NRG1-fusion classifier. . . . . . . . . . . . . . 496.8 Comparing training losses for models trained with and without control livernormal images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546.9 Transfer learning for feature extraction, with finetuning. . . . . . . . . . . . 556.10 Segmented tumour patch predictions for case 1. . . . . . . . . . . . . . . . . 596.11 Segmented tumour patch predictions for case 2. . . . . . . . . . . . . . . . . 596.12 Segmented tumour patch predictions for case 3. . . . . . . . . . . . . . . . . 606.13 Examples of tumour patches that were incorrectly predicted as normal. . . . 616.14 Examples of tumour patches that were correctly predicted as tumour. . . . . 616.15 Example of incorrectly predicted normal patches, containing different histo-logical features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.16 NRG1 expression vs the percent of patches predicted as NRG1 (per test case). 646.17 Example of tumour heterogeneity between two WSIs from the same patient. 656.18 Example of tumour heterogeneity between three WSIs from the same patient. 666.19 Predicted tumour content vs percent of correctly predicted patches for eachtest case. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687.1 Top three activated filters in each CNN block, for a particular input image. . 707.2 Top filters activated in each CNN block, for different input images. . . . . . 717.3 Top filters activated in each CNN block, for all images from a particular WSI. 73xii7.4 Comparison of tissue biopsy patch to ImageNet categories that share similarfeature activations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747.5 Comparison of CNN filter to watershed segmentation. . . . . . . . . . . . . . 76xiiiList of AbbreviationsAdaGrad Adaptive Gradient AlgorithmAdam Adaptive Moment EstimationANN Artificial Neural NetworkCAD Computer-Aided DiagnosisCE Cross-EntropyCNN Convolutional Neural NetworkFFPE Formalin-fixed, Paraffin-embeddedGPU Graphics Processing UnitH&E Hematoxylin and EosinMLP Multilayer PerceptronPDAC Pancreatic Ductal AdenocarcinomaRGB Red, Green, and BlueRMSProp Root Mean Square PropagationSGD Stochastic Gradient DescentxivWSI Whole Slide ImagexvAcknowledgementsFirstly, I would like that thank my supervisor, Dr. Steven Jones, for his continuoussupport and guidance. Since my second year of undergrad, you’ve given me incredibleopportunities to explore my interests and advance my skills, and have instilled in me astrong passion for research. I have truly enjoyed working and growing within your lab, andam extremely grateful for your mentorship.Thank you to my supervisory committee members, Dr. Elodie Portales-Casamarand Dr. Stephen Yip, for all the support they’ve provided me throughout my graduatestudies. Your enthusiasm, attention to detail, unique perspectives, and insightful ideasmotivated me to think critically and achieve more.Thank you to Julia Naso, for completing the annotations needed for my project;and to Dave Farnell, for taking the time to explain important histopathology concepts tome.Thank you to members of the CDTS project, Jessica Nelson, Robin Coope, EricChuah, and Sherry Wang, for constantly supporting me throughout my project and makingsure I had the resources I needed.Thank you to Diana Mah for sitting beside me and helping me clean thousandsof microscope slides. Your warmth and optimism made the experience a whole lot moreenjoyable.Thank you to all former and current members of the Jones Lab, who have inspiredxvime and taught me lots through their own research pursuits. In particular, thank you toMicha Disyak, for continuously encouraging me and motivating me throughout my graduatestudies. It has been really enjoyable getting to share our graduate school adventures together,and I am very thankful for your friendship. Thank you to Eric Zhao and Jasleen Grewal, forhelping me immensely during my undergraduate experiences in the lab. And, thank you toEmre Erhan, Luka Culibrk, Zoltan Bozoky, Jahanshah Ashkani, Jean-Michel, Kieran O’Neill,Vahid Akbari, Harwood Kwan, Jimmy Li, and Cam Grisdale, for continuously sharing yourinsights and knowledge.Thank you to other members of the GSC, who have made my experiences hereextremely enjoyable. In particular, thank you to Gordon Robertson, for constantly sharingyour exciting findings with me, championing my goals, and encouraging me to keep an openmind. I have really enjoyed your mentorship and am forever grateful for all that you’ve taughtme. And, thank you to Rene´ Warren, Tina Wong, and Diana Lin, for your friendship, andthe many enjoyable conversations and lunches we’ve shared.Thank you to Jenny Ung, Lucy Shen, and Jaidan Lee, for your unwavering friend-ship and always cheering me on.Finally, I would like to thank my parents, Hui Zhang and ZhiYong Yang, who aremy backbone and biggest inspiration. Thank you for always believing in me.xviiTo my loving mom and dad, Hui Zhang and ZhiYong Yang.xviiiChapter 1Introduction1.1 ObjectiveWe used convolutional neural networks to determine whether or not a patienthas a therapeutically actionable genomic alteration, based on cancer biopsy images. Inparticular, we focused on identifying NRG1 gene fusions within metastatic pancreatic ductaladenocarcinoma cases.1.2 MotivationWhole genome and transcriptome sequencing have tremendously improved ourunderstanding of the molecular mechanisms that underly cancer growth and progression.Analyses of large-scale sequencing data have shown that changes in the molecular profileof a cancer can affect its response to particular drugs (Huang et al., 2014; Sawyers, 2004).These discoveries have led to significant clinical successes in the development and use ofmolecularly targeted cancer therapies, such as imatinib mesylate – a BCR-ABL tyrosinekinase inhibitor – in chronic myeloid leukaemia (CML) (Druker et al., 2001), and gefitinib– an EGFR tyrosine kinase inhibitor – in non-small cell lung carcinoma (NSCLC) (Paez et1al., 2004). Understanding the key molecular changes in both major types and subtypes ofcancer can lead to the development of more targeted therapies and more effective treatmentstrategies that are specific to each patient’s cancer profile (Huang et al., 2014; Verma, 2012).Although molecularly-targeted cancer therapy approaches are an effective treatmentoption, there are inconsistent protocols to implement or evaluate the required specializedgenetic tests, both provincially and nationally, in Canada (Butts et al., 2013). Additionally,because there is limited public funding for specific genetic tests linked to therapeutics, thereare challenges in translating these discoveries into the healthcare system (Butts et al., 2013).Currently, diagnosis from a histopathology slide is the gold-standard for cancer di-agnosis (Gurcan et al., 2009). Tissue slides of biopsied cancers are routinely generated aspart of the clinical workflow for cancer management, and pathologists manually assess andinterpret these slides to identify the type and grade of cancer and different histological fea-tures, based on the structure of the tissue and cells. If therapeutically informative molecularmarkers could be identified directly from histopathology, it will significantly increase theamount of quality information that can be extracted from these slides, without increasingoverhead and requiring additional fees for genetic tests.A machine learning model that has been successfully applied to many image-basedanalysis problems is a convolutional neural network (CNN). Considering the diversity ofspecialized interpretations we can already make manually from tissue slides, a CNN approachwill be able to drastically increase the scope of prognostic interpretations in cancer biology.If genomic changes important for cancer treatment can be quickly identified from the imageof the cancer tissue itself, we can save valuable time, help expedite the relay of genomicdiscoveries to clinical research, and apply this method inexpensively and quickly as part ofroutine analysis for all cancer patients.21.3 Thesis ContributionsThe organization of this thesis is as follows:Chapter 2, 3 These chapters explain the biological and computational theory underlyingthe experiments performed.Chapter 4 This chapter highlights related works pertaining to applications of convolu-tional neural networks within digital histopathology.Chapter 5, 6 These chapters explain the data processing and model building steps weimplemented to answer the research problem. It also highlights and discusses the results weobtained.Chapter 7 This chapter explains steps we took to visualize and understand the internalstructure and decision-making process of our CNN model.Chapter 8 This last section summarizes the thesis and discusses limitations and possiblefuture directions.3Chapter 2Biological Background2.1 Traditional HistopathologyHistopathological image analysis is considered the gold standard for clinically di-agnosing cancer (Gurcan et al., 2009; Harris et al., 2010; He et al., 2012). Traditionalhistopathological analysis involves manually examining a biopsied section of tissue to locateand classify disease (He et al., 2012). This comprises of placing a thin slice of tissue under alight microscope and visually checking for any irregularities in cell shapes and tissue distribu-tions (Junqueira et al., 2005). And, as part of the clinical workflow for cancer management,these tissue slides are routinely generated from biopsied cancers.After a cancer biopsy, the section of tissue undergoes a fixation process to preservecell composition and architecture. A widely used fixative within pathology labs is formalin,as it has a high degree of accuracy and adaptability (Thavaragah et al., 2012). In addition,this fixation process also prevents autolysis and necrosis from occurring. This is essentialfor further processing, examination, and archival preservation. Once the tissue specimenis fixed, it can then be embedded in a paraffin wax block. This makes it easier to sectionthe tissue into thin slices, for placement on a microscope slide. This type of preparation isknown as formalin-fixed, paraffin embedded (FFPE).4Before analysis by a histopathologist, the tissue slides are stained using hematoxylinand eosin (H&E). These stains selectively bind to nucleic acids and proteins, making it easierto identify different nuclear, cytoplasmic, and extracellular matrix structures. Hematoxylinstains the nucleic acid components a deep blue-purple, and the eosin stains the cytoplasmand extracellular matrix pink (Fischer et al., 2008).For cancer diagnosis, histopathologists will analyze these slides to determine whethercancerous regions are present, as well as decide on a malignancy grade (He et al., 2012). Theywill look at the sizes, shapes, and arrangements of cells to identify any cell-type- and can-cer cell-type-specific features. However, this can be very time consuming, and because ofthe subjective nature of histopathological interpretations, there can exist uncertainty andinterobserver variability in diagnosis and grading (He et al., 2012; Higgins, 2015; Nass et al.,2019).2.2 Digital HistopathologyAdvancements in technology are continuously changing and shaping healthcare,including how we visualize and analyze cells and tissues on a microscopic scale. Digitalhistopathology refers to the digitization of conventional glass slides containing a thin sliceof tissue, by means of an optical slide scanner (Aeffner et al., 2018; Farahani et al., 2015;Higgins, 2015). While there are different digital imaging modalities utilized in pathologydepartments worldwide, whole slide imaging (WSI), in particular, is gaining the most mo-mentum within research and diagnostic pathology (Hamilton et al., 2014; Farahani et al.,2015).WSI instrumentation and computer software for imaging cells and tissues haveexisted since the late 1990s (Aeffner et al., 2018; Farahani et al., 2015). However, they’veonly become more widely adopted and accepted over the recent years, due to significanttechnological advancements, including advancements in computer processing power, roboticcapabilities, data transfer speeds, and data storage solutions. These have contributed to the5development of commercially available WSI scanners that produce high-resolution imagesand operate at reasonable speeds and costs (Farahani et al., 2015).WSI is commonly referred to as “virtual microscopy,” as it uses computerized meth-ods to mirror conventional light microscopy (Farahani et al., 2015). Thus, the digital slidesgenerated by WSI scanners can reach up to 40X magnification, giving an approximate reso-lution of 0.25µm per pixel. The standard for most histopathologic evaluations is 20X mag-nification (0.5µm per pixel) (Webster & Dunstan, 2014). Because of the size of these digitalslides, specialized slide-viewing software must be used to view and analyze them. Theseapplications allow for the slide to be viewed at any position and magnification, imitating theutility of a conventional light microscope. In addition, some of these applications allow formarkings and annotations to be made on the digital slide images.Having digitized histopathological slides allows for more efficient sharing for teach-ing and consultation, easier storage management, preservation of slide quality, and detailedfeature annotation (Aeffner et al., 2018; Higgins, 2015). Furthermore, as digital slide datasetsgrow in size, they will enable the development of automated methods for analysis, classifi-cation, and identification of histopathological features.2.3 Targeted TherapyDevelopments in whole-genome and transcriptome sequencing have transformedthe landscape of cancer therapy. Large-scale analyses have shown that genomic variations,and their association with particular cancer types, can impact a patient’s response to partic-ular drugs (do Rosario Pinheiro et al., 2014; Huang et al., 2014; Sawyers, 2004). Therefore,by understanding the molecular changes underlying a cancer, treatment strategies can betailored towards each patient’s unique genetic profile (Mukesh, 2012; Sawyers, 2004). Someof these molecular changes can have a critical role in cancer progression and tumour growth;thus, new lines of therapies have been specifically designed to interfere with these markers.This forms the basis of “targeted therapy.” Examples include EGFR mutations and NTRK6and FGFR gene fusions, which been found to be novel therapy targets across multiple cancertypes (Amatu et al., 2016; Baselga, 2001; Ladanyi & Pao, 2008).2.4 Pancreatic Ductal AdenocarcinomaIn 2018, pancreatic ductal adenocarcinoma (PDAC) was found to be the seventhleading cause of death from cancer worldwide (Bray et al., 2018). It is considered the mostlethal common cancer, with the highest incidence-to-mortality ratio of any solid tumour(Christenson et al., 2020; Ryan et al., 2014). It is often diagnosed at an advanced stage, andmore than 80% of cases are unresectable at diagnosis (Christenson et al., 2020; Ryan et al.,2014). Thus, there is urgent need for novel therapeutic strategies in combating this disease.Molecular pathology studies and genomic analyses have improved our understand-ings of how PDAC develops and progresses. In particular, there has been significant progressin treatment strategies for specific molecular subgroups. For example, within KRAS wild-type PDAC cases, some rare alterations were found to be potentially therapeutically ac-tionable (Christenson et al., 2020). These include ALK amplifications (Singhi et al., 2017),NTRK gene fusions (Pishvaian et al., 2018), and NRG1 gene fusions (Heining et al., 2018;Jones et al., 2019). These findings highlight the importance of understanding the molecularprofile of a tumour, so that the most effective treatment strategy can be used.2.4.1 NRG1 Gene FusionAlthough only 8%-10% of PDAC tumours are KRAS wild-type, they are givenspecial attention due to their enrichment for therapeutically actionable alterations (Aguirre,2019; Christenson et al., 2020). One of these alterations is the NRG1 gene fusion, which hasbeen found to be a rare, but potentially targetable oncogenic driver (Aguirre, 2019; Jones etal., 2019; Jonna et al., 2019).7NRG1 has been found to interact with the ERBB signaling pathway by bindingthe ERBB3 receptor, which heterodimerizes with ERBB2 to activate downstream signalingpathways. Thus, targeting ERBB2 and ERBB3 has been shown to be an effective treatmentstrategy when NRG1 gene fusions are present (Aguirre, 2019; Jonna et al., 2019).Heining et al. performed whole genome and transcriptome sequencing on a cohortof 17 patients with PDAC. They found four tumours that were KRAS wild-type, with three ofthese containing an NRG1 gene fusion. Two of these patients were given therapies targeted atthe ERBB signaling pathway (afatinib or erlotinib), and were found to have clinical benefits(Heining et al., 2018).Jones et al. also comprehensively profiled a cohort of PDAC patients using whole-genome and transcriptome sequencing. The majority of these patients were KRAS wild-type, and three were found to have an NRG1 gene fusion. Two of these patients were givenan ERBB receptor inhibitor (afatinib) and demonstrated significant and rapid response totherapy (Jones et al., 2019).8Chapter 3Machine Learning BackgroundMachine learning combines concepts from computer science, statistics, probability,and optimization. In general, it refers to computational methods that use past information ordata to improve performance or make accurate predictions (Alpaydin, 2020; Mohri, 2018). Ithelps us analyze large amounts of data and solve problems that would otherwise be impossibleto do manually. Therefore, the usages of machine learning are widespread, with applicationsin speech processing, computational biology, computer vision, and fraud detection.3.1 Supervised LearningMany successes in machine learning have been based on supervised learning. Thisis an approach where a model is trained to approximate a function, which maps between aset of inputs and the desired output. The samples used to train a model are labeled (i.e. eachset of inputs has an associated output), and as training progresses, the model will update itsweights to best represent the dependencies and relationships between specific features andoutputs. This learned representation can then be used to predict the output for new sets ofinputs.93.1.1 ClassificationOne category of supervised learning models are classification models. These tryto predict the discrete label or category for a given set of inputs. This is different fromregression models, which instead, try to predict a continuous value from a given set of inputs.Traditionally, classification models were developed to distinguish between just two classes(Aly, 2005; Tax & Duin, 2002). This is often referred to as binary classification. Binaryclassifiers have been well-studied, and various models have been shown to be successful insolving binary classification problems (Aly 2005; Tewari & Bartlett, 2007). However, someclassification problems can have more than two possible target classes, making it necessaryto develop methods for handling the multiclass case. For this, various approaches have beenintroduced, including flat and hierarchical classification methods (Silla & Freitas, 2011).Flat classification can refer to either binary or multiclass classification problems,where a single classifier is used to predict all possible classes (Silla & Freitas, 2011). Theseapproaches try to classify examples to their final “leaf-node” labels, without considering anynatural hierarchy or “parent categories” that may exist between categories. This approachis shown in Figure 3.1. It is simple and can be implemented quickly using any existingout-of-the-box classifier. Some methods include decision trees, k-Nearest Neighbour, Na¨ıveBayes Classifiers, and Support Vector Machines (Aly, 2005; Tax & Duin, 2002). However,this approach can have disadvantages when trying to discriminate between a large numberof classes (Silla & Freitas, 2011), as important dependency and relationship information canbe lost. This can result in reduced model performance.Local hierarchical classification methods use modular local node training within apredefined hierarchical structure. These approaches are commonly referred to as “top-down”classifiers, as each example will go through a series of predictions from a top (generic) class,until it reaches its final (specific) class (Silla & Freitas, 2011). Each prediction is used tonarrow the possible classes at subsequent levels; that is, the only possible candidate classesare the children of the class predicted on the previous level. This approach is shown in Figure3.2.10Figure 3.1: Flat multi-class classificationThe flat classification approach predicts all the leaf-nodes together.11Figure 3.2: Local Hierarchical multi-class classificationA local hierarchical classification approach uses individual, local classifiers on each parent node .12When using some multiclass classifiers, a problem that can occur is training incon-sistency, where an example could belong to more than one class (Huang et al., 2015; Silla& Freitas, 2011; Wang et al., 2010). An advantage of using a local parent-node approach isthat you can avoid and correct for this issue. Each local classifier will only be trained onexamples belonging to the children of the specific node, therefore staying within the boundsof class membership and avoiding the problem of inconsistent predictions (Silla & Freitas,2011).3.2 Artificial Neural NetworksAn Artificial Neural Network (ANN) is a machine learning system motivated byearly models of a biological neural network. It aims to mimic how neurons process informa-tion, in order to learn patterns and solve problems.(a) Biological Neuron (b) Artificial NeuronFigure 3.3: Comparison of biological and artificial neurons.The building blocks of ANNs are artificial neurons – the nodes of the network. Ingeneral, an ANN consists of many nodes with different weighted connections between them.A node will receive input from other nodes, and, depending on the weighted input, it willeither be activated or inactivated to generate subsequent outputs. This is comparable tobiological neural processing systems, as they consist of highly interconnected neurons thatreceive and transmit signals at varying strengths (Krogh, 2008; Yegnanaravana, 2009).This type of architecture allows ANNs to handle specific pattern recognition tasks13through a learning or training process. This is an iterative process that presents a varietyof examples to the model, similar to how humans use examples to learn a new concept ortask. As the model is presented with new examples, it gradually updates its weights andthresholds to improve its understanding of the problem (Krogh, 2008).3.2.1 Structure of an ANNThe PerceptronFigure 3.4: Schematic of a basic perceptron.The smallest building block of an ANN is an artificial neuron – the perceptron. Itis a general computational model that takes in an input xi, associated weights wi, and a biasb (Minsky & Papert, 2017). The numerical weights act as a measure of variable importanceand the bias acts as a thresholding parameter. The model computes the aggregated weightedsum, and outputs yi = 1 if the aggregated sum is greater than the bias and yi = 0 if it isless.y = 1 if y =∑i=1wixi ≥ b,y = 0 if y =∑i=1wixi < b(3.1)14However, as a convention, the bias is added as an additional input with the weightw0 = −b. Here, w0 is sometimes referred to as the prior.y = 1 if y =∑i=0wixi ≥ 0,y = 0 if y =∑i=0wixi < 0,where x0 = 1, w0 = −b(3.2)A single perceptron is used to learn and implement a linearly separable function(a hyperplane). During training, it aims to classify the input into two halves (yi = 1 andyi = 0), and adjusts the weights until an optimal decision boundary is formed.Activation FunctionAn artificial neural network consists of many perceptrons (or neurons) connectedto each other. They can be grouped into several layers, where the perceptrons in one layerare connected to perceptrons in adjacent layers through weighted connections. As previouslymentioned, an individual perceptron is linear; thus, as information from each perceptron ispassed on to subsequent ones, linearity is still maintained through a series of affine trans-formations. This type of neural network would essentially behave as a standard regressionmodel. In order to introduce non-linearity into a perceptron, a non-linear activation functioncan be used. This activation function performs a non-linear transformation on the outputbefore it’s passed on to the next layer, making the model capable of solving more complextasks (Sharma, 2017).15With an activation function, the perceptron model can now be written as:y = α(∑i=1wixi) (3.3)where α(x) is the non-linear activation function andα(x) =0, if x ≤ 0x, if x > 0 (3.4)Different activation functions have unique properties, and choosing which one touse depends on the specific task being performed. For ANNs, the sigmoid activation functionis widely used (Narayan, 1997; Sharma, 2017). As shown in Figure 3.5, the sigmoid functionbehaves as a smooth “step-like” function, making clear distinctions upon output prediction.An advantage of this function is that it is bounded between the range (0, 1), ensuring thatactivations won’t blow up (Karlik & Olgac, 2011). However, in some cases, the gradient canbecome so small that the weights may not update during training. This is known as thevanishing gradient problem (Kolbusz et al. 2017, Sharma, 2017).Figure 3.5: Sigmoid Activation Function.16Another activation function that performs well in many cases is the Rectified LinearUnit (ReLU). This activation function is commonly used in practice when working withCNNs. An advantage of using this function is that not all neurons will be activated atthe same time. As seen in Figure 3.6, a neuron will be deactivated when the output of thetransformation is zero. Additionally, it is fast to compute, and it does not have the vanishinggradient problem (Sharma, 2017).Figure 3.6: ReLU Activation Function.17Multilayer PerceptronFigure 3.7: Multilayer Perceptron.Each circle represents an individual perceptron, and connections between these perceptrons formthe multilayer perceptron network.As mentioned before, artificial neurons can be grouped into layers and connected toone another through weighted connections. In theory, these connections may be arbitrary,but for classification purposes, a feed-forward configuration is often used. This is whenan ANN is made up of neurons in ordered layers, where the first layer is the input layer,the last layer is the output layer, and the intermediate layers are the hidden layers (Syozil,1997). This type of ANN is called a Multilayer Perceptron (MLP), as each layer consists ofindividual perceptrons and activation functions. These layers can be viewed as a series ofnon-linear transformations, where feature representations are being learned in a structuredmanner. Additionally, when all the neurons in a layer are connected to all the neurons in thefollowing layer, the network is considered fully connected. As more hidden layers are added,the function being learned is no longer constrained to a hyperplane, and can instead resultin a non-linear solution.18Figure 3.8: Example decision boundary for 2-dimensional classification.3.3 Convolutional Neural NetworksA machine learning model that has made a significant leap in the field of computervision is the Convolutional Neural Network (CNN). This structure takes into account thespatial dependence of input data, making it particularly effective for performing image anal-ysis tasks. Some areas where CNNs have had notable success include image classification(Ciresan et al., 2011; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), image segmen-tation (Badrinarayanan et al., 2017; Long et al., 2015), and facial recognition (Parkhi et al.,2015). CNNs have also been successfully applied to other types of data, but for the purposesof this thesis, I will be focusing on image data.Similar to how ANNs are inspired by how neurons process information, CNNsare inspired by the visual cortex. Here, different neurons will only respond to stimuli withinspecific regions. These regions are known as receptive fields, and a collection of these overlapto make up the entire visual field. This motivated the development of the “Neocognitron” –a neural network that aims to mimic the concept of receptive fields (Fukushima & Miyake,1982). The Neocognitron then inspired the development of an early model of the CNN,which successfully incorporated back-propagation (explained in a later section) (LeCun etal., 1989). Since then, many other improvements and extensions have been added to CNNs,which have propelled the development of many new, state-of-the-art models. These models19include AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014), Inception(Szegedy et al., 2015), ResNet (He et al., 2016), and so on. These models have had remarkableclassification performance on the benchmark ImageNet dataset, and have been widely appliedin other areas of computer vision.3.3.1 Structure of a CNNFigure 3.9: Schematic of a basic CNN.This diagram is modified from (Phung & Rhee, 2019)CNNs take image data as input. Images can be formatted in different colour spaces,such as grayscale or RGB. Figure 3.10 displays an RGB image, separated into its red, green,and blue channels. Multiple channels, combined with large dimensions or high resolutions,can be computationally expensive to analyze, especially when there can be hundreds ofthousands of images to go through. CNNs have the ability to reduce images into a formthat is easier to process, while still identifying important features needed for classificationor prediction. Thus, a CNN can be viewed as a feature extractor with a classifier attachedto the end, as shown in Figure 3.9.20Figure 3.10: Image separated into RGB colour channels.This image is from (Sharma, 2019).Many CNNs that are used today can extend thousands of layers in depth. However,they generally consist of the same basic building blocks – convolution layers, pooling layers,and a fully connected layer (i.e. an MLP). Earlier layers detect generic, low-level featuressuch as colours, intensity, and edges, whereas deeper layers beginning detecting more specificfeatures pertaining to each class.Convolutional LayerConvolutional layers are responsible for extracting features from the input. Theselayers learn and preserve local patterns within an image, which can later be used in classifi-cation and prediction.Convolution Operation For neural networks, convolution describes a mathematical op-eration that applies a kernel to an image. A kernel can be viewed as a “filter” that extractsa set of feature representations from an image. Depending on the kernel used, a convolutioncan perform operations such as edge detection, sharpening, or blurring.21Mathematically, the formal two-dimensional discrete form of a convolution is:G(i, j) = (h ∗ F )(i, j) =i∑u=0j∑u=0h(u, v)F (i− u, j − v) (3.5)where h is an image, F is the kernel being applied, and i and j are the width and height ofthe kernel, respectively.Figure 3.11: 5x5 image matrix and 3x3 filter matrix example.The kernel extracts feature representations from the section of the image it is cur-rently on. It continues to do this as it moves across the image at a set stride. The operationstops once the entire image has been traversed, resulting in a feature map (i.e. the convolvedoutput).Figure 3.12: Filter moving across image to obtain convolved feature.22In practice, to reduce the computation cost, the cross-correlation operation is oftenused instead of the convolution operation. The operation for cross-correlation is:G(i, j) = (h ∗ F )(i, j) =i∑u=0j∑u=0h(i+ u, j + v)F (u, v) (3.6)The main difference is whether the kernel is flipped or not. However, for the purposeof feature extraction, these two operations are equivalent.Figure 3.13: Example of a flipped filter.Local Connectivity and Parameter Sharing Compared to ANNs, two distinct char-acteristics of CNNs are local connectivity and parameter sharing. Both of these propertieshelp reduce the number of parameters used in CNN models, and can make computationmore efficient.As previously mentioned, standard ANNs are fully connected, meaning that everyneuron in one layer is connected to every neuron in the previous layer. This is not the casefor convolutional layers, as every neuron in one layer is only connected to a small subset ofneurons in the previous layer. These localized regions represent receptive fields, and formthe basis of a sparsely connected net.23(a) Fully-connected layers. (b) Locally-connected layers.Figure 3.14: Comparison of full vs local connectivity.For a particular layer, the same convolution operation is performed on all neurons.This means that the set of weights does not change, and thus is shared by all neurons in thelayer. This property is known as parameter sharing.Activation Function As previously mentioned, the choice of activation function useddepends on the task being performed. For the convolutional layers in a CNN model, theReLU activation function is commonly used (Krizhevsky et al., 2012; Simonyan & Zisserman,2014).Pooling LayerA pooling layer performs down-sampling on the output of a filter (i.e. the featuremap). This reduces the dimensionality of the map, while still retaining important informa-tion. Two common types of pooling used in CNNs are max pooling and average pooling.Based on a set window and stride size, max pooling selects the largest element from a window,24while average pooling takes the average value of all the elements.Figure 3.15: Max-pooling with a 2x2 window size.Fully-Connected LayerWhen using CNNs for classification, a typical model consists of consecutive blocksof convolutional and pooling layers, with a fully connected MLP attached at the end. Thepurpose of this MLP is to find a function that maps between resultant features (i.e. thefeatures extracted from the convolutional and pooling layers) and their desired class labels.3.3.2 VGG-16 ModelWith the availability of high-performance graphic processing units (GPUs) andlarge-scale image datasets, CNNs made a major breakthrough in 2012 (Krizhevsky et al.,2012). Since then, many state-of-the-art CNN architectures have been developed, but forthe purposes of this thesis, I will be focusing on the VGG-16 model.The VGG-16 model is a CNN that has helped advance the development of deepvisual recognition architectures. It was built for the purpose of image recognition and classi-fication, specifically for the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC)(Simonyan & Zisserman, 2014). The model obtained first and second places in the imagelocalization and classification categories, respectively.The architecture of the VGG-16 model is shown in Figure 3.16. It consists of 1325convolutional layers, 5 max pooling layers, and 3 fully connected layers for classification.The pooling layers are placed after every 2 or 3 convolutional layers, separating the layersinto 5 convolutional “blocks.”Figure 3.16: Schematic of the VGG-16 model.This image is modified from (Basaveswara, 2019).3.4 Training a Neural Network3.4.1 Cost FunctionTraining a neural network involves determining a set of weights that best representthe associations between sets of features and their desired outputs. These weights are con-tinuously updated through training, with the goal of minimizing a predefined cost function.This function outputs a single value, which is used to compare candidate sets of weights,in order to determine the best solution for a problem (Reed & MarksII, 1999). It is nota measure of accuracy, but instead, a way to evaluate model optimization (i.e. how well amodel is learning).A cost function that is commonly used in practice for classification problems isCross-Entropy (CE). For binary classification, the CE loss function is defined as:CE = − 1mm∑i=0(Yilog(Yˆi) + (1− Yi)log(1− Yˆi)) (3.7)where, m is the number of samples, and Yi and Yˆi represent the actual (or “ground truth”)26and predicted labels, respectively.3.4.2 Gradient-Based OptimizationGradient-based optimization algorithms are used to control how much a modelshould be adjusted, based on the estimated loss when weights are updated. To do this, backpropagation is used (LeCun et al., 1988). This is where an algorithm continuously appliesthe chain rule to calculate the partial derivatives (i.e. gradients) of each parameter, basedon the current weights of the model. It sequentially calculates the gradients for each layer,starting with the final layer. These gradients are used to adjust the weights, such that thecost converges to a minimum. This is referred to as gradient descent (Ruder, 2016). Theformal definition of gradient descent, with respect to one parameter, is:θn+1 = θn − η∇θJ(θ) (3.8)where J(θ) is the cost function, θ is a model parameter, and η is the learning rate. Thelearning rate determines the size of steps taken to reach the local minimum.Because there are many samples in a training set, gradient descent can take a longtime to compute, as it has to run through all the samples for each iteration. Thus, stochasticgradient descent (SGD) (Bottou, 2010) is often used, where only one sample or a subsetof samples is used to update parameters during each iteration. Recently, many adaptivevariations of stochastic gradient descent have been developed, including root mean squarepropagation (RMSprop), adaptive gradient algorithm (AdaGrad), and adaptive momentestimation (Adam).273.4.3 GeneralizationThe goal of a neural network is to be able to perform well on samples that werenot used to train the model. This demonstrates a model’s ability to generalize and is usuallyevaluated by using a validation set. This is a subset of data that is set aside during training,and therefore, not seen by the model. By evaluating the model on a validation set, we canget an estimate of how well it could potentially perform on new, unseen data.The concept of generalization is closely related to overfitting. This is when thevalidation error is much higher than the training error, suggesting that the model has learneddependencies and relationships specific to the training data, instead of a function that can begeneralized to other datasets. To try to prevent overfitting and improve model generalization,regularization techniques can be used. The regularization techniques used in the thesis aredata augmentation and dropout.Data AugmentationModel generalization can be improved by adding more training data. If a modelis trained on only a small sample of datapoints, it can quickly overfit, resulting in poorgeneralization on new datasets. Because it can be difficult to obtain or generate more realdata, methods have been developed to artificially expand a dataset.In terms of image data, image augmentation techniques can be used. This involvestaking existing images in a dataset, and applying image transformation operations to them,such as zooming in, shearing, translating, rotating, flipping, and stretching (Sarkar et al.,2018). Combinations of these transformations can produce new, altered versions of theexisting images, which can then be adding to training.28DropoutAnother method to help improve model generalization is dropout (Hinton et al.,2012). If dropout is added to a layer in the network, a fraction p of output nodes willrandomly be masked (i.e. outputs will be set to 0). For example, if p = 0.3, then 30% ofthe output units in a layer will be excluded as inputs for the following layer.3.4.4 Transfer LearningTransfer learning refers to utilizing knowledge that was learnt in the past, to helpsolve new, related tasks (Sarkar et al., 2018; Pan & Yang 2009). When applied to neuralnetworks, transfer learning consists of leveraging weights from previously trained modelsto train new models. This differs from the traditional approach of training models, whereweights are learned from scratch.A key motivation behind transfer learning is that many models require a lot of datato build, especially when the problem is complex (Sarkar et al., 2018). This is evident in a lotof supervised learning tasks, as obtaining vast amounts of labeled data can be a challenge.Thus, if there is sufficient data for one task, then we can utilize its learning and knowledgefor another task, where there is far less data available. Particularly, in terms of image-basedlearning, certain features such as intensity, edges, and shapes, are common across differenttasks, allowing for previous knowledge (i.e. weights) to be transferred.29Chapter 4Related Works for DigitalHistopathology in OncologyAdvances in whole-slide digital scanners have enabled the efficient digitization ofWSIs. These scanners are being used in pathology labs worldwide and have propelled thefield of digital histopathology within oncology (Bera et al., 2019).4.1 Digital Histopathology Databases and Imaging ToolsThe digitization of WSIs has led to the creation of many large-scale digital slidelibraries (Bera et al., 2019). Many of these can be accessed freely by researchers worldwide,enabling researchers to collaborate, make new discoveries, and offer insight into analysismethods for digital pathology. One of the most popular libraries for cancer research isfrom The Cancer Genome Atlas (TCGA) project (Tomczak et al., 2015), which is a cancergenomics program that molecularly characterized over 20000 tissue and blood samples fromover 30 different human tumour types. The diagnostic and tissue slide images from thisproject are publicly available, and different imaging tools have been created to help usersview them, including The Cancer Imaging Archive (TCIA) (Clark et al., 2013) and The30Cancer Digital Slide Archive (CDSA) (Gutman et al., 2013).In addition to providing access to digital slides, tools have also been created toallow researchers to share their own data with others. One example of this is The DigitalSlide Archive (Gutman et al., 2017), which allows researchers to create and maintain theirown digital slide resources, as well as share them with others for annotation or analysis.More recently, with machine learning approaches being increasingly used in histopathol-ogy, some databases are being formed specifically for use within deep learning applications.The Atlas of Digital Pathology (ADP) comprises of patch images extracted from 100 digitalslides with accompanying annotations (Hosseini et al., 2019). Thus, the images in the ADPare already labeled and appropriately sized for use in machine learning tasks.4.2 Machine Learning for Cancer Diagnosis and BiomarkerAnalysisWith improvements in computational power and an influx of digital WSIs in cancerdiagnostics, increasing attention is being given to machine learning for histopathologicalanalysis (Bera et al., 2019; Gurcan et al., 2009). These digital WSIs are able to capture tissueand cell content from a tumour resection or biopsy at diagnostic resolution, and thus containa high volume of information that can be critical in diagnosis (Serag et al., 2019). Quite a fewstudies have shown the potential of using machine learning to analyze these high-resolutionimages, specifically for cancer diagnostics and biomarker analysis. In (Xu et al., 2017), aCNN transfer learning approach was used to distinguish between histopathology images ofglioblastoma multiform (GBM) and low-grade glioma (LGG) cancer, achieving an accuracy of97.5% on the test data. In (Yonekura et al., 2018), researchers also analyzed histopathologyimages of GBM, but instead, used a CNN to classify the disease stage of GBM. They wereable to achieve an average classification accuracy of 96.5%. Studies have also shown thepotential for machine learning models to predict mutated genes from histopathology images.31In (Coudray et al., 2018), researchers trained a CNN on non-small cell lung cancer WSIsto classify between adenocarcinoma (LUAD), squamous cell carcinoma (LUSC), and normallung tissue, achieving an AUC of 0.97. Additionally, they found that the network could alsobe trained to predict six commonly mutated genes in LUAD with AUCs ranging from 0.733to 0.856. In (Chen et al., 2020), a similar study was performed for liver cancer. Here, a CNNwas trained to classify between benign and malignant hepatocellular carcinoma (HCC), aswell as the level of tumour differentiation. The model achieved accuracies of 96% and 89.6%,respectively. They were also able to predict four common and prognostic mutated genesin HCC with AUCs ranging from 0.71 to 0.89. There have also been studies investigatingmachine learning for WSIs in other cancers, including breast cancer (Alom et al., 2019) andgastric cancer (Sharma et al., 2017). These all contribute to the growing body of researchsuggesting that machine learning models; namely CNNs, can potentially assist pathologistsin determining the grade of a tumour, classifying a cancer subtype, and detecting prognosticgene mutations.It is important to note that many of these studies used a “patch-level” imageapproach, where they do not use entire WSIs in training, but instead extract a small numberof patches from each individual WSI (Dimitriou et al., 2019). This is because digital WSIs arehigh-resolution and can reach over 100000x100000 pixels per image. However, benchmarkimage datasets that CNN models perform well on are low resolution, with only around200x200 pixels per image (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014).32Chapter 5Data Generation and Normalization5.1 DatasetThe primary dataset used in this project is from the Personalized OncoGenomics(POG) program at BC Cancer in Vancouver, British Columbia. This cohort consists ofmetastatic cancer patients who were selected based on cancer stage, biopsy availability, suf-ficient tumour content, and complete clinical data. Biopsies were taken from the metastaticsites using needle-core, ultrasound-guided or CT-guided biopsies. These samples were thenmolecularly profiled using whole-genome and transcriptome sequencing to identify poten-tially clinically actionable genomic events (Pleasance et al., 2020).For this thesis, I am specifically investigating metastatic PDAC cases and usingdigital images of the biopsy tissue slides. The tissue on these slides all come from the liver,and are FFPE, H&E stained, and scanned using WSI scanners at 40X magnification. Basedon the sequencing data, I have also determined which ones contain an NRG1-fusion.In addition to the primary testing performed, I also tested the model on additionaltest data. These new cases were obtained from the POG project and the PanGen project atBC Cancer. Similarly, these slides are all liver biopsies from metastatic PDAC cases, FFPE,33H&E stained, and scanned using WSI scanners at 40X magnification.All images used have been anonymized.5.2 AnnotationsBefore processing and normalizing the WSIs, regions containing tumour cells wereannotated by a pathologist. These annotations were completed in QuPath (Bankhead et al.,2017), an open source platform for viewing and annotating WSIs.(a) Original WSI (b) Annotation (c) Close-upFigure 5.1: Example of a WSI and its accompanying annotations.345.3 Data Processing5.3.1 Patch ExtractionTo prepare the dataset for analysis with a CNN, patches of a smaller, fixed sizewere extracted from each WSI. This was done using a sliding-window approach, where a384x384 pixel window was shifted across the image, extracting patches from varying regionsof the tissue slice. For each WSI, patches were extracted from both the normal and tumourtissue areas, based on the annotations provided by the pathologist.Figure 5.2: Examples of patch images.The stride size used varied between 384-768 pixels, depending on the amount oftissue area on each slide. This was done to try to balance the number of patches producedfrom each WSI and prevent bias towards any particular sample. However, because thetumour to normal tissue ratio differed between WSIs, different samples will have producedfewer or more patches for a given class. Additionally, because there were fewer positiveNRG1 gene fusion samples than negative, more patches were extracted from the positivecases in order to maintain an even class distribution during training.35(a) Small Annotation (b) Large AnnotationFigure 5.3: Example of differing tumour to normal tissue area on WSIs.365.3.2 Background DetectionEach patch was checked for the amount of tissue content present. Patches thatcontained more than 30% of blank space were discarded in the dataset.Figure 5.4: Examples of patches with < 30% blank space, that were not discarded.Figure 5.5: Examples of patches with > 30% blank space, that were discarded.5.3.3 Focus QualityPatches were checked for clarity using the Fast Fourier Transform (FFT). This is anefficient algorithm for computing the Discrete Fourier Transform (DFT), which calculatesthe frequency components within an image. The higher the number of high frequenciespresent, the clearer the image (De & Masilamani, 2013).The FFT was performed on all patch images using the built-in FFT function in37Python’s NumPy (Oliphant, 2006) package. Then, a mean value of the magnitudes of thefrequencies was calculated. Images with a mean value below a fixed threshold were discarded.This threshold was arbitrarily determined through manual visual observation.Figure 5.6: Examples of blurry patches that were discarded.5.3.4 NormalizationThe dataset has slides that were prepared at different facilities and scanned bydifferent scanners. These differences can introduce artifacts that can hinder the performanceof a machine learning model. However, in order to be clinically translatable to large patientpopulations, these models need to be generalizable across different slide preparations andimaging workflows (Dimitriou et al., 2019). Examples of these differences are shown in Figure5.7.Figure 5.7: Examples of patch images from different WSIs.As mentioned in the previous section, artifacts such as focus quality can be filteredout, as important visual information can be lost. However, other artifacts such as illumi-nation, contrast, and staining colour variability do not necessarily reduce the quality of the38visual information in an image, and thus should still be included in the dataset. In orderto generalize over these artifacts, each image was normalized using a stain-vector method(Macenko et al., 2009). This method, introduced by Macenko et al., assumes that there is aspecific stain vector that corresponds to each of the two H&E stains. It finds these two stainvectors for each image and then transforms them to match a fixed, target colour space andintensity. Detailed explanations of these processes can be found in (Macenko et al., 2009).(a) Original patch (b) H stain (c) E stainFigure 5.8: Example of deconvolved H&E stains in from a patch.For our purposes, the target values chosen were based on the average stain andintensity values of the entire dataset, therefore minimizing the collective magnitude of staincorrections performed on the images. The results from the normalization step can be seenin Figure 5.9.Figure 5.9: Examples of normalized patch images from different WSIs.Figure 5.10 shows the RGB colour distributions before and after normalization. Itcan be seen that the patches share a similar RGB colour distribution after normalization.Thus, this new, normalized dataset reflects a representative distribution of artifacts andstaining, and corrects for some of the variation.39(a) RGB pixel distribution before.(b) RGB pixel distribution after.Figure 5.10: Histograms comparing RGB pixel distribution before and after normalization.40Chapter 6Model Architecture and Performance6.1 Training and Test Dataset SplitBefore implementing and training our machine learning model, the dataset wassplit into training and test sets. Each WSI represents one patient.WSIs PatchesTraining SetNRG1-fusion neg: 5NRG1-fusion pos: 3Normal: 477NRG1-fusion neg: 223NRG1-fusion pos: 263Test SetNRG1-fusion neg: 2NRG1-fusion pos:1Normal: 80NRG1-fusion neg: 154NRG1-fusion pos: 106Table 6.1: Training and test dataset split.Normal patches are distributed across both NRG1-fusion negative and positive WSIs.416.2 Model Selection and Hyperparameter Configura-tionAs previously mentioned, there have been many state-of-the-art machine learningarchitectures that have been trained on the benchmark ImageNet dataset. Many of thesemodels, as well as their pre-trained stable weights, have been made openly available for othersto use within their own research. To predict which model and hyperparameter configurationwill perform best on our data, k-fold cross-validation (CV) was used. In particular, we used5-fold CV, where the model was trained 5 times on 80% of the data and tested on the other20%. Each iteration used a different fold of the data for testing, such that all points were usedin the 20% validation set at least once. For this step, only the training set was used. Thisensured that the chosen model would not be biased towards the test set, as that was used toevaluate the final, fully-trained classifier. In total, the training set contained 5 NRG1-fusionnegative cases, and 3 NRG1-fusion positive cases. The folds were split based on each case,so that patches from the same case didn’t show up in both training and validation folds.Thus, each negative case was used exactly once in validation, and each positive case wasused either once or twice. For the purposes of testing different models and configurations,we only investigated the ability to classify between normal and tumour patches in this step.Using 5-fold CV, the architectures we compared were the pre-trained ResNet-50,Inception V3, and VGG-16 networks. As a baseline, a simple CNN with no pre-trainedweights was also compared. Using Python’s Keras library (Chollet, 2015), we implementedthe pre-trained ResNet-50, Inception V3, and VGG-16 models, and built our own simpleCNN from scratch.42(a) Simple CNN (b) ResNet-50(c) Inception-V3 (d) VGG-16Figure 6.1: Comparing training and validation accuracies for different CNN models.The training accuracy is shown in grey and the validation accuracy is shown in black.From Figure 6.1, we can see that the simple CNN, without pre-trained weights,overfits and does not improve on the validation set over the course of training. The pre-trained, deep CNNs all performed significantly better, with only the ResNet-50 showingsmall signs of overfitting. Both the Inception-V3 and the VGG-16 models achieved similarlyhigh accuracies on our data.Optimizers were also compared. In particular, 5-fold CV was used to compare SGDand variations of it, including AdaGrad, RMSProp, and Adam. These optimizers were alsoavailable in Python’s Keras library (Chollet, 2015).43(a) SGD (b) AdaGrad(c) Adam (d) RMSpropFigure 6.2: Comparing training and validation accuracies for different optimizers.The training accuracy is shown in grey and the validation accuracy is shown in black.From Figure 6.2, we can see that using the SGD optimizer resulted in erraticaccuracy on the training set, and even more so on the validation set. Overall, it did notappear to learn the task. However, variations of SGD, namely the adaptive learning methods,appeared to learn the task more efficiently. They had much less erratic training and validationaccuracies, while continuing to make minor weight adjustments near the end of training.The purpose of these steps was not to build a final model, but instead, to checkwhich model and hyperparameters seemed to perform the best on our dataset. Based onthese comparisons, we decided to use the VGG-16 model and the Adam optimizer in therest of the experiments.446.3 Two-Stage Local Hierarchical Learning Framework6.3.1 Model ArchitectureThe model implemented has a two-stage hierarchical structure, which uses theVGG-16 model as the base for each local classifier. This is shown in Figure 6.3, where thefirst node classifies a patch as either tumour or normal tissue, and the second node classifiesthe tumour predictions as either NRG1-fusion negative or NRG1-fusion positive.Figure 6.3: Hierarchical model architecture.456.3.2 Model Training OverviewTo build the final model, the training set was used in training. Here, local classifierswere individually trained for each parent node, using exclusively the samples belonging tothe children of that node. They were trained using a transfer learning approach, as describedin the next section.Transfer Learning for Feature ExtractionAs previously mentioned, the VGG-16 model consists of 5 convolutional blocksand 3 fully-connected layers. The convolutional blocks serve as feature extractors, while thefully-connected layers are for classification.To leverage the advantages of transfer learning, we initialized the convolutionalblocks using their pre-trained ImageNet weights. We used these blocks as feature extractorsand fed them directly into the shallow classifier. Here, the pre-trained weights were “frozen,”and thus were not updated during training.Because we trained multiple local classifiers, the fully-connected layers on the orig-inal VGG-16 were removed, as those layers were specific to the ImageNet dataset. Instead,these layers were replaced with a new, shallow classifier that was specific for our purposes.This shallow classifier consisted of 3 fully-connected layers, utilized dropout, and used ReLUand Sigmoid activation functions. Unlike the convolutional layers, these weights were ran-domly initialized and continuously updated during training.For model training, the loss function used was binary cross-entropy and the opti-mizer used was Adam.46Figure 6.4: Transfer learning for feature extraction.This image is modified from (Basaveswara, 2019).Image AugmentationDuring training, we used image transformation operations to produce new, alteredversions of the data. To do this, we used the ImageDataGenerator class in Keras (Chollet,2015). This accepts a batch of images and applies a series of random transformations onthem. The transformations we chose included random combinations of shearing, translating,rotating, flipping, and stretching. We kept the extent of transformations minimal to avoiddistorting the images too much. This transformed batch replaces the original batch intraining (i.e. the original data itself is not used in training). Thus, the number of trainingimages remains the same; however, each batch is transformed slightly differently, in effectaugmenting the training data. Examples of some altered images are shown below.Figure 6.5: Examples of augmented images.476.3.3 Model Performance and ResultsClassification of Normal and Tumour TissueThe results in this section are for the first local classifier used in our model, andthus discusses our model’s ability to classify patches as normal or tumour tissue.Figure 6.6: Training accuracy and loss for normal vs tumour classifier.Figure 6.6 shows the training accuracy and loss curves during training. From theseplots, it appears that the model is learning, as the accuracy steadily increases and the losssteadily declines. However, to evaluate the model’s ability to generalize on new, unseen data,we tested the trained model on the held out test set. The metrics shown below are calculatedon a per-patch basis.On the test data, this classifier achieved an accuracy of 83.5%. The confusion matrixin Table 6.2 shows the discrepancies between the different classifications. A complete set ofperformance metrics is shown in Table 6.3.48PredictedNormal TumourTrueNormal 59 21Tumour 35 225Table 6.2: Confusion matrix for normal vs tumour tissue classification.Accuracy 0.835Precision 0.847Recall 0.835F1-Score 0.840Table 6.3: Model performance metrics for normal vs tumour tissue classification.Classification of NRG1-FusionThe results in this section are for the second local classifier used in our model,and thus discusses our model’s ability to classify tumour patches as NRG1-fusion positiveor NRG1-fusion negative.Figure 6.7: Training accuracy and loss for NRG1-fusion classifier.Figure 6.7 shows the training accuracy and loss curves during training. Again,49from these plots, it appears that the model is learning, as the accuracy increases and theloss declines.Here, the patches tested are those that were correctly predicted as containing tu-mour by the previous classifier. The metrics shown below are calculated on a per-patchbasis.This classifier achieved an accuracy of 76.0%. The confusion matrix in Table 6.4shows the discrepancies between the different classifications. A complete set performancemetrics is shown in Table 6.5.PredictedNRG1-fusion neg. NRG1-fusion pos.TrueNRG1-fusion neg. 96 26NRG1-fusion pos. 28 75Table 6.4: Confusion matrix for NRG1-fusion classification.Accuracy 0.760Precision 0.760Recall 0.760F1-Score 0.760Table 6.5: Model performance metrics for NRG1-fusion classification.Overall Classification ResultsThe overall patch-level results for all the final classification labels are shown belowin Table 6.6.The overall accuracy achieved was 67.7%. The majority of patches within each classare predicted accurately; however there is still cross-calling that occurs between classes. Acomplete set of performance metrics is shown in Table 6.6.50PredictedNormal NRG1-fusion pos. NRG1-fusion neg.TrueNormal 59 11 10NRG1-fusion pos. 3 75 28NRG1-fusion neg. 32 26 96Table 6.6: Confusion matrix for overall tissue classification.Accuracy 0.677Precision 0.681Recall 0.677F1-Score 0.676Table 6.7: Model performance metrics for overall classification.6.4 Model Additions and Finetuning6.4.1 Adding Control Liver ImagesAs a way to introduce more images to the model during training, we added twonormal liver parenchyma images to the training process. These tissue slides were also FFPE,stained using H&E staining, and digitally scanned at 40X magnification. As before, patcheswere extracted from different areas of the slides, filtered and normalized, and then addedto the training set. These slides produced 360 patches, expanding the training dataset to837 normal patches and 486 tumour patches. To differentiate between these new normalliver patches and the normal liver patches from the cancer biopsies, the new patches will bereferred to as “control liver normal” and the patches from the cancer biopsies will be referredto as “background liver normal.”This newly expanded training set is then used to retrain the model. Because of classmembership and local training, the newly added data only effects the first classification node,51which predicts between normal and tumour tissue. This updated model is then reevaluatedon the test data.This retrained local classifier achieved an accuracy of 84.1%, which improves on theprevious model by 0.6%. From the confusion matrix in Table 6.8, we can see that the truetumour patches are predicted with the same accuracy, but the number of correctly predictedtrue background normal patches increases. A complete set performance metrics is shown inTable 6.9.PredictedNormal TumourTrueNormal 61 19Tumour 35 225Table 6.8: Updated confusion matrix for normal vs tumour classification, after addition ofcontrol liver normal images in training.Accuracy 0.841Precision 0.855Recall 0.841F1-Score 0.846Table 6.9: Updated model performance metrics for normal vs tumour classification, afteraddition of control liver normal images in training.The overall patch-level results are also shown below in Tables 6.10 and 6.11. Theupdated overall accuracy is now 68.2%, which is a 0.5% improvement from before.52PredictedNormal NRG1-fusion pos. NRG1-fusion neg.TrueNormal 61 8 11NRG1-fusion pos. 4 74 28NRG1-fusion neg. 31 26 97Table 6.10: Updated confusion matrix for overall tissue classification, after addition of controlliver normal images in training.Accuracy 0.682Precision 0.686Recall 0.682F1-Score 0.682Table 6.11: Updated model performance metrics for overall classification, after addition ofcontrol liver normal images in training.In addition to accuracy, the training loss can also be used to evaluate whether ornot adding control liver normal patches improved training. Figure 6.8 shows a comparisonof the previous training loss and the new training loss. The new training loss is consistentlylower, suggesting that the model’s ability to learn has improved with the addition of the newcontrol normal patches.53Figure 6.8: Comparing training losses for models trained with and without control livernormal images.6.4.2 FinetuningAs mentioned earlier, shallow layers of a CNN capture generic, low-level featuressuch as colour, intensity, and edges, while deeper layers detect more class-specific features.Previously, we extracted these features by keeping all convolutional blocks frozen duringtraining. However, to attempt to make the model more specific towards our particular task,we used a finetuning approach on the deeper layers. Here, we only froze certain layers duringretraining, and finetuned the rest. In particular, we left the last two convolutional blocksunfrozen, while still keeping the first three frozen. This process is shown below in Figure6.9.54Figure 6.9: Transfer learning for feature extraction, with finetuning.This image is modified from (Basaveswara, 2019).Finetuning was performed on all local classifiers in the model, using the new ex-panded dataset (i.e. included the new control liver patches). The first local classifier, whichpredicts between normal and tumour patches, achieved an accuracy 86.5% after finetuning.This is a 2.5% improvement from the previous model. The results for this classifier are shownbelow.PredictedNormal TumourTrueNormal 70 10Tumour 36 224Table 6.12: Updated confusion matrix for normal vs tumour classification, after finetuning.Accuracy 0.865Precision 0.887Recall 0.865F1-Score 0.871Table 6.13: Updated model performance metrics for normal vs tumour classification, afterfinetuning.The second local classifier, which predicts for the NRG1-gene fusion, achieved alower accuracy compared to the previous model, only classifying 72.9% of patches correctly.55Thus, in the final model, we will use the finetuned version to predict between normal andtumour patches, and the non-finetuned version to determine if an NRG1-fusion is present.These results make sense, as finetuning adjusts the model’s weights to make themmore specific to our particular task. Thus, in cases where more training data is available (asseen with the first local classifier), it can help improve model accuracy; however, when thereis less training data (as seen with the second local classifier), it can result in overfitting.6.5 Overall Results and Discussion6.5.1 Main Test DataAfter adding control liver normal images to training and finetuning additionallayers, the final accuracy achieved is 68.8%. This improves on previous versions of the modelby up to 1.1%. The overall patch-level results are shown below in Tables 6.14 and 6.15.PredictedNormal NRG1-fusion pos. NRG1-fusion neg.TrueNormal 70 8 2NRG1-fusion pos. 4 84 18NRG1-fusion neg. 32 42 80Table 6.14: Updated confusion matrix for overall tissue classification, after finetuning.56Accuracy 0.688Precision 0.713Recall 0.688F1-Score 0.681Table 6.15: Updated model performance metrics for normal vs tumour classification, afterfinetuning.To be clinically translatable, results should be given on a patient-level rather thana patch-level. Thus, if the majority of patches (or a certain threshold) are predicted asNRG1-fusion positive, then that patient would be classified as having the gene fusion. Thepatient-level results from the test set are shown in Table 6.16. All cumulative predictionsare correct, except one, where a true NRG1-fusion negative tumour was predicted as normaltissue.WSI True class Predicted class # Patches matching prediction1Normal Normal 31/31NRG1-fusion pos. NRG1-fusion pos. 84/1062Normal Normal 18/26NRG1-fusion neg. NRG1-fusion neg. 77/1133Normal Normal 21/23NRG1-fusion neg. Normal 32/41Table 6.16: Patient-level classification results, for test data cases.Because three cases is very limited, Table 6.17 displays the training cases at thepatient-level. These results are biased, as they were used to build our model; however, at thepatient-level, they can give us insight into how well the model has learned the training data,as well as show whether certain classes are predicted more accurately than others. Similarly,with the training data cases, most incorrectly predicted patches were from the NRG1-fusionnegative patches.57WSI True class Predicted class # Patches matching prediction1Normal Normal 54/54NRG1-fusion pos. NRG1-fusion pos. 73/732Normal Normal 77/77NRG1-fusion pos. NRG1-fusion pos. 126/1283Normal Normal 71/71NRG1-fusion pos. NRG1-fusion pos. 62/624Normal Normal 31/31NRG1-fusion neg. NRG1-fusion neg. 46/645Normal Normal 15/15NRG1-fusion neg. NRG1-fusion neg. 48/726Normal Normal 72/73NRG1-fusion neg. NRG1-fusion neg. 20/277Normal Normal 13/13NRG1-fusion neg. NRG1-fusion neg. 34/368Normal Normal 143/143NRG1-fusion neg. NRG1-fusion neg. 21/249 Normal (Control Liver) Normal 182/18210 Normal (Control Liver) Normal 178/178Table 6.17: Patient-level classification results, for training data cases.Figures 6.10, 6.11 and 6.12 display the distribution of patch predictions on eachWSI test case. For each figure, the red region represents patches predicted as NRG1-fusionnegative, the blue region represents patches predicted as NRG1-fusion positive, and the greenregion represents patches predicted as normal.58(a) Original annotation (b) Patch predictionsFigure 6.10: Segmented tumour patch predictions for case 1.This case was correctly predicted as NRG1-fusion positive tumour.(Blue - NRG1-fusion positive; Red - NRG1-fusion negative; Green - Normal)(a) Original annotation (b) Patch predictionsFigure 6.11: Segmented tumour patch predictions for case 2.This case was correctly predicted as NRG1-fusion negative tumour.(Blue - NRG1-fusion positive; Red - NRG1-fusion negative; Green - Normal)59(a) Original annotation (b) Patch predictionsFigure 6.12: Segmented tumour patch predictions for case 3.This case was incorrectly predicted as normal.(Blue - NRG1-fusion positive; Red - NRG1-fusion negative; Green - Normal)60From Figures 6.10 and 6.11, we can see that the majority of the predicted tumourregion matches those of the original annotation. However, from Figure 6.12, we can seethat very few patches were predicted as general tumour during the first classification stage.Figure 6.13 shows a few examples of patches from the incorrectly predicted sample. Theseimages show poor tumour differentiation and do not exhibit typical histological featuressuch as gland formation. However, the other cases used in training and testing are eithermoderately or well differentiated tumours. Thus, the machine learning model may not havebeen able to differentiate poorly differentiated tumours from normal tissue, as they do notshare the same histological features found in the other samples, and are not representedwithin the training data. These differences can be seen in Figures 6.13 and 6.14.Figure 6.13: Examples of tumour patches that were incorrectly predicted as normal.These patches are poorly differentiated.Figure 6.14: Examples of tumour patches that were correctly predicted as tumour.These patches are moderately or well differentiated.Because tissue samples are diverse and heterogeneous, there are many features thatcan be present including inflammation and blood vessels. These features may not have beenfully represented during training, and thus can contribute to misclassification. Two examplesare shown in Figure 6.15. These patches are both true normal patches but were predicted astumour. One contains blood vessels and the other contains a benign duct, and these featuresmay have been confused for tumour.61(a) Blood vessels (b) Benign DuctFigure 6.15: Example of incorrectly predicted normal patches, containing different histolog-ical features.Although the majority of normal patches were correctly predicted, 12.5% weremisclassified as tumour, with the majority of them being predicted as NRG1-fusion positive.Previously, it has been shown that the expression of fusion transcripts can be found in normalcells, and thus is not limited to cancer tissue (Akiva et al., 2006; Latysheva & Babu, 2016;Luo et al., 2015). In particular, a large screening performed on TCGA data identified 192gene fusions in 394 normal tissue samples (Yoshihara et al., 2015). Thus, it is possible thatsome normal tissue regions may contain features relating to NRG1-fusions.6.5.2 Additional Test DataWe also obtained additional PDAC WSIs from the POG project and the Pan-Gen project. Because NRG1-fusion positive cases are rare, these additional slides consistedentirely of NRG1-fusion negative cases. From the PanGen project, we evaluated 11 newcases, and from the POG project, we evaluated an additional 17 new cases. As before, allof these tissue slides were FFPE, stained using H&E staining, and digitally scanned at 40Xmagnification. We extracted patches from different areas of the slides, resulting in 571 and1546 patches for the PanGen and POG data, respectively. The patches were then filteredand normalized, and evaluated on the model. It should be noted that annotations were not62available for all of these cases, thus, we extracted patches from different regions of the WSIsfor testing on the model. Based on the majority of tumour patch predictions, we classifiedeach case as NRG1-fusion positive or negative. Although all the cases were NRG1-fusionnegative, we wanted to see if the model could consistently classify them correctly. Thepatient-level results are shown in Tables 6.18 and 6.19, for the PanGen data and the POGdata, respectively.Total casesevaluatedCorrectly predictedNRG1-fusion neg.Incorrectly predictedNRG1-fusion pos.11 7 4Table 6.18: Patient-level NRG1-fusion classification results for additional test data (Pan-Gen).Of the 11 PanGen cases, 7 were correctly predicted as being NRG1-fusion negative,while 4 were incorrectly predicted as having the NRG1-fusion.Total casesevaluatedCorrectly predictedNRG1-fusion neg.Incorrectly predictedNRG1-fusion pos.17 10 7Table 6.19: Patient-level NRG1-fusion classification results for additional test data (POG).Of the 17 addditional POG cases, 10 were correctly predicted as being NRG1-fusionnegative, while 7 were incorrectly predicted as having the NRG1-fusion.These results emphasize how more training data is necessary in order for the modelto accurately generalize across new, external datasets.6.5.3 NRG1 Gene ExpressionFor the additional POG data, we also had matching gene expression data available.From this, we found that 5 of the incorrectly predicted cases had high NRG1 gene expression63relative to other PDAC cases. Compared to the other 12 POG cases used in testing, theselevels were in the 75th percentile and higher; and, compared to TCGA PDAC cases, thesewere in the 93rd percentile and higher. As for the true NRG1-fusion positive test case inthe main test set, the expression was found to be much higher than fusion negative cases,with an RPKM of 42.1. However, the other NRG1-fusion positive cases used in training hadNRG1 expression levels ranging from RPKM values of 2.3 to 23.4.Figure 6.16: NRG1 expression vs the percent of patches predicted as NRG1 (per test case).Previous studies have suggested that oncogenic gene fusions may be correlated withconstituent parent gene expression (Latysheya & Babu, 2016). For example, in prostatecancer, it has been found that fusion transcript expression is related to higher expressionof the parent genes (Lai et al., 2015; Nacu et al., 2011). Similar results were also foundfor fusions in hematological cancers, as the parent genes were more highly expressed thanaverage (Akiva et al., 2006). This could suggest a similar relationship for NRG1-fusions, andthus it is possible that our model could be learning cases of high NRG1 gene expression, inaddition to true NRG1-fusions. However, additional data is needed in order to validate thisfinding.646.5.4 Tumour HeterogeneityDifferent cellular subpopulations, with unique phenotypic features can exist withinvarying regions of the same tumour (Bedard et al., 2013; Jamal-Hanjani et al., 2015). Thisis known as intratumour heterogeneity, and has been found in many tumour types, includingpancreatic cancer (Campbell et al., 2010; Yachida et al., 2010). Additionally, it has beenfound that cell divisions within metastatic tumours can also evolve independently, resultingin intrametastatic heterogeneity (Jamal-Hanjani et al., 2015). Thus, in our case, where weonly used one slide per biopsy, it is unlikely that we captured the complete tumour landscapefor each patient.As previously mentioned, once a biopsy is performed, it undergoes a fixation andembedment process before being sliced and mounted onto slides. Each biopsy produces morethan one slide; however, in our analyses, we only used one slide per biopsy in training. Fromtesting, we found that predictions can vary depending on which slide is chosen. An exampleof this can be seen in Figure 6.17. This figure shows the results for a NRG1-fusion negativecase, where the first slide was incorrectly predicted as NRG1-fusion positive, but the secondslide was correctly predicted. The second slide also contains a larger tumour area; thus, ifwe were to use the cumulative predictions of the two slides, then the case would be correctlypredicted as NRG1-fusion negative.(a) Incorrectly Predicted WSI. (b) Correctly Predicted WSI.Figure 6.17: Example of tumour heterogeneity between two WSIs from the same patient.65Another example is shown in Figure 6.18, where different slides from the samebiopsy have different classifications. Again, when evaluated together, the cumulative predic-tion is correct.(a) Correct prediction. (b) Incorrect prediction. (c) Correct prediction.Figure 6.18: Example of tumour heterogeneity between three WSIs from the same patient.666.5.5 Tumour PurityTumour purity (or tumour content) is the proportion of cancer cells in a solid tu-mour sample (Aran et al., 2015; Qin et al., 2018). Because there is normal cell contaminationin tumour tissue, sequencing data obtained from tumour samples contain components fromboth normal and cancer cells (Qin et al., 2018). To see if tumour purity correlates with ourmodel’s ability to classify cases, we compared predicted tumour content to the proportionof patches correctly predicted for each test case. Here, predicted tumour content is obtainedusing Ploidetect (Culibrk, 2020), which is a software package that estimates tumour contentby fitting Gaussian mixture models (GMMs) to read depth and single nucleotide polymor-phism (SNP) data. This is shown in Figure 6.19, which displays results for each stage of theclassifier. The results shown are based on the additional POG test set, as predicted tumourcontent information was available.Because of the number of samples available, a conclusive interpretation cannot bemade; however, upon the availability of more data, there may be interesting opportunities todiscover a correlation. Especially in relation to discriminating between normal and tumourpatches, as tissue with lower tumour content can have a similar appearance to that of normaltissue. This could possibly make it more difficult for a CNN model to accurately predictbetween classes.67(a) Stage 1: Normal vs Tumour Classification(b) Stage 2: NRG1-fusion ClassificationFigure 6.19: Predicted tumour content vs percent of correctly predicted patches for each testcase.68Chapter 7Visualization of CNN Model7.1 Visualizing CNN FiltersA common question that arises with deep machine learning models is how tointerpret and understand the decisions they’re making. CNNs have typically been associatedwith image data, and, unlike other types of data, images are visually interpretable. Thus, wecan visualize what types of features are being extracted and what sort of patterns activateparticular filters.One way to visualize a CNN is by using a Deconvolutional Network (deconvnet)(Zeiler et al., 2011). A deconvnet has the same components as a CNN, but in reverse.So, where CNNs map input pixels to features, deconvnets map features (the convolutionoutputs) to the input. To do this, a deconvnet is attached to each layer of the CNN. Animage is then fed into the CNN and feature maps are computed in each layer. These featuremaps are then passed to the deconvnet and used to reconstruct the activity that gave riseto certain activations. These activations are then mapped back to the input pixel space. Adetailed explanation of this process can be found in (Zeiler et al, 2011).69For each convolutional block in the VGG-16 model, we determined the top threefilters that were activated by the test data. We did this by comparing the mean activationsproduced by each filter, in all convolutional layers of a block. We then visualized thesefilters by applying the deconvnet to each block. In our case, we applied a deconvnet usingthe implementation in (Rawlani, 2018), and used the pre-trained ImageNet weights, as thisis what was used during training. Figure 7.1 shows the top three filters activated in eachblock, for a particular image in our test set.(a) Input Image (b) 1 (c) 2 (d) 3 (e) 4 (f) 5Figure 7.1: Top three activated filters in each CNN block, for a particular input image.The first and second convolutional blocks retain most of the image features andis activated on every part of the image. The first block appears to be detecting low-levelfeatures such as colour and intensity, and the second block appears to be detecting directionaledges.The third and fourth blocks are detecting more complex shapes and features, andsome regions of the image are not activated at all. In some of the top activated filters, itappears that the nuclei and the glands are highly activated.In the fifth block, there are many more inactivated outputs. This is because thepatterns being encoded are high-level and less interpretable, as the network is trying toabstract visual features and convert them into class-specific domains. Thus, they have lowerchances of appearing in a given input image.70More examples can be seen in Figure 7.2, where the top filters in each block areshown.Figure 7.2: Top filters activated in each CNN block, for different input images.717.2 Network AnalysisWe then analyzed the top activated filters as a network, to visualize the path thatan input image takes through the model. In the case of the VGG-16 model, there are 4224different filters. A breakdown of the number of filters in each convolutional layer is shownin Table 7.1.Convolutional Block Convolutional Layer Number of Filters11 642 6421 1282 12831 2562 2563 25641 5122 5123 51251 5122 5123 512Table 7.1: Number of filters in the VGG-16 network.We found that many patches appear to share the same highly activated filters inthe first three convolutional blocks, before branching in blocks four and five. This makessense, as the initial blocks detect low-level, generic features, while the deeper blocks detectmore complex, specialized features.Figure 7.3 shows the highest activated filters for each patch in a particular sample.From this figure, we can see that most patches started out sharing the same highest activatedfilters, before diverging once they traveled deeper in the network.72Figure 7.3: Top filters activated in each CNN block, for all images from a particular WSI.Table 7.2 shows the most common highest activated filters for all the test cases.Again, we see that there is greater variation in deeper blocks.Patient True class Predicted class Block 1 Block 2 Block 3 Block 4 Block 51Normal Normal 18 100 19 476 2NRG1-fusion pos. NRG1-fusion pos. 52 100 19 463 4712Normal Normal 52 100 19 476 149NRG1-fusion neg. NRG1-fusion neg. 52 100 19 463 4713Normal Normal 52 100 19 211 392NRG1-fusion neg. Normal 52 100 19 211 392Table 7.2: Most common highest activated filters for all test patches.These findings suggest that the filters in a deep CNN could potentially be com-pressed to a smaller set of filters, consisting of only those that have the highest activations.This would increase the efficiency of the model, as it would decrease the number of param-eters used, as well as make the model more specific towards our particular task. Previousstudies have found that the number of parameters in a deep neural network could be sig-73nificantly reduced without affecting model accuracy (Nakkiran et al., 2015; Wang et al.,2019).We then mapped these highly activated filters to their matching ImageNet activa-tions. These activations were found to be most closely related to ImageNet categories thathave repetitive edges and shapes, and spread-out patterns. This makes sense since eachpatch image we used consists of recurring histological features (glands, nuclei, etc.) thatcover the entire image area. A comparison of these different categories are shown in Figure7.4.(a) Tumour (b) Velvet (c) Cabbage(d) Stole (e) Nematode (f) HankerchiefFigure 7.4: Comparison of tissue biopsy patch to ImageNet categories that share similarfeature activations.747.3 Comparison to Traditional Image SegmentationTraditional computer-aided diagnostic (CAD) systems for digital histopathologytypically consist of image segmentation methods and clustering-based machine learning al-gorithms (He et al., 2012). Examples of some conventional segmentation techniques includecolour and intensity thresholding, edge detection, and denoising. These steps can help ex-tract different types of objects, such as nuclei and stroma, which can then be used to trainmachine learning classifiers. Some clustering-based classifiers that have traditionally beenused in digital histopathology include support-vector machines (Doyle et al., 2007, 2008;Glotsos et al., 2004) and k-nearest neighbours (Ganster et al., 2001).The magnification level of histology slides is sufficient for analysis at the tissuelevel, and some analysis at the cell level (e.g. finding normal and tumour tissue boundaries,nuclei counting, and cell deformity identification) (He et al., 2012). However, factors suchas overlapping tissue types, cell boundaries, and varying cell shapes and sizes can makeit difficult to accurately extract features of interest, especially through use of conventionalsegmentation techniques. This can ultimately affect the performance of machine learningmodels that are trained on these features.As previously mentioned, the CNN model has made a significant impact in the fieldof computer vision, as it is able to take into account the spatial features of input data. Theyautomatically reduce images into a form that is easier for a computer to process, while alsoidentifying important features needed for classification or prediction. Each layer of a CNNmodel can consist of hundreds of different filters, and some of these filters perform the sametypes of operations as conventional image segmentation. To compare CNNs to traditionalCAD systems, we analyzed highly activated filters in our CNN model against the watershedalgorithm – a common image segmentation method. The figure below shows examples ofpatches that were segmented using the watershed approach, as well as comparable featuremaps produced by the CNN. The watershed algorithm used is modified from the OpenCVPython library (Bradski & Kaehler, 2000).75(a) Original image (b) Watershed (c) CNN FilterFigure 7.5: Comparison of CNN filter to watershed segmentation.This comparison of watershed segmentations to CNN filters shows similarities be-tween the two techniques, as both methods appear to be segmenting (or activating) onregions with dark clusters of nuclei. However, it should be noted that each layer of a CNNcan consist of hundreds of different filters, and that each of these filters can activate on differ-ent features or combinations of features. Thus, unlike conventional segmentation techniques,where a user has to manually set unique thresholds for different features, CNNs are able toautomatically apply a large variety of different filters. Furthermore, traditional CAD sys-tems typically use image segmentation as a feature extraction step before machine learninganalysis, whereas a CNN inherently has this step built-in to its architecture.76Chapter 8Conclusion8.1 Summary of FindingsIn this thesis, we proposed a method for identifying NRG1 gene fusions fromimages of metastatic PDAC biopsies, using a CNN. Our results demonstrated that using atransfer learning approach with a deep CNN, such as the pre-trained VGG-16 model, canassist in distinguishing normal and tumour tissue types (86.5% accuracy), as well as inferthe presence of an NRG1 gene fusion (76.0% accuracy). The overall accuracy achieved at thefinal classification level was 68.8%. We found that finetuning and adding additional controlliver normal images to training increased the accuracy of our model by up to 1.1%. Althoughthis value seems small, it could be improved through the availability of more data. Table 6.16shows the overall patient-level results for the testing cohort, and Figure 6.13 shows a fewexamples of misclassified patches from the incorrectly predicted case. These misclassifiedpatches depict poorly differentiated tumours, whereas the rest of the cases used in bothtraining and testing were moderately to well differentiated. This highlights the importanceof using a consistent tumour grade for training, or introducing more training samples thatdisplay a variety of different tumour grades.77In addition to our main POG test set, we also tested our model on additional dataobtained later on, from both the POG and PanGen projects. When evaluated on our model,both the POG and PanGen datasets had several misclassified cases. This emphasizes theneed for more training data in order for the model to be generalizable across external datasets.However, interestingly, of the 7 POG cases that were incorrectly predicted as harbouring anNRG1 gene fusion, 5 were found to have high NRG1 gene expression relative to other PDACcases. And, in accord, previously studies have found that fusions are correlated with highexpression of the parent genes.We also looked at the network activation pathways for the data, and found commonfilters that were highly activated across the majority of patches. This suggests that a com-pressed version of the model, with fewer filters, could potentially be developed to performthe same task.Finally, we compared CNN approaches to more traditional CAD approaches indigital histopathology. By comparing CNN filters to traditional watershed segmentation,we were able to show the similarities between extracted features, as well as highlight theadvantages of using a CNN.Overall, this type of tool provides a quick and inexpensive way to determine genomicalterations within a particular cancer. Despite the presence of various artifacts and limiteddata availability, these findings show promise that a CNN can predict NRG1 gene fusionsdirectly from biopsy images.8.2 Limitations and Future DirectionsAlthough our model was able to correctly classify most test cases, there are stilllimitations and improvements that should be considered.The biggest limitation in this project is the amount of data available. As NRG1-fusions are already quite rare within PDAC, this especially affected the number of NRG1-78fusion positive cases we could obtain for analysis. This caused our model to overfit to thefew cases we had, reducing its ability to generalize over new, unseen data. This is reflected inthe lower test accuracy achieved, in comparison to the training accuracy. Thus, in general,this work would largely benefit from greater data availability.Similarly, because we only used one slide from each biopsy, it is unlikely that theone slide used is able to represent the full heterogeneity of features within a tumour. Todemonstrate this, we evaluated multiple slides from the same biopsy on our final model.We were able to show that different slides from the same biopsy could result in differentpredictions. This suggests that more slides from the same biopsy should be utilized in train-ing and testing, in order to better represent the features within a tumour. This would alsoprovide opportunities for 3D analysis and visualization. For example, the model could bemodified to allow information sharing between spatial slices, and additional image registra-tion techniques could be applied. Thus, it could be interesting to see if adding an additionaldimension can improve model performance and prediction.There are also changes we can consider in terms of how we structure our classifier.In terms of the overall CNN building blocks, we can investigate the accuracy and efficiencyof a compressed model structure, where we reduce the number of parameters. And in termsof the classification elements, we can add additional features, as there are many differenthistological features that we did not consider in our training data (e.g. necrosis, bloodvessels). Adding these features as additional labels could be useful for training, and mayfurther improve model performance. Additionally, another class that we could try to predictfor is high NRG1 expression, as this may be linked to NRG1-fusions.Because our transfer learning approach used pre-trained ImageNet dataset weights,it would be beneficial to investigate how model training and accuracy could improve uponpre-training on a tissue/cellular image bank. This would allow for weights to be learnedspecifically for cell and tissue morphological structures, making the model more specific forour purposes. This would require a lot a data, and because of the sizes of currently availabletissue databases, a combination of these would likely need to be used.79For fusions in particular, more consideration and analysis of normal tissue shouldbe considered, as previous studies have shown that the expression of fusion transcripts canbe found in normal samples as well.We also compared the prediction accuracy of the POG cases to their predictedtumour content. Because of the dataset size, no conclusive correlation could be made.However, this could be interesting to investigate upon more data availability.It has also been found that frozen slides (instead of FFPE) can be trained to identifyparticular genomic alterations (Coudray et al., 2018). Thus, future work could include frozenslides in training.Overall, as machine learning techniques and digital histopathology evolve, there arepromising opportunities to innovate and develop new approaches for diagnostic pathologyand targeted therapy. And, given the importance of certain genomic markers, particularlyin terms of treatment response, it will be beneficial to develop methods that can bypassadditional genetic testing. Thus, computational approaches that can be integrated withinroutine pathology can accelerate workflows, improve diagnostic consistency, and help selectthe best possible therapy for a patient.80Bibliography[1] Aeffner, F., Adissu, H. A., Boyle, M. C., Cardiff, R. D., Hagendorn, E., Hoenerhoff,M. J., ... & Wilson, K. (2018). Digital microscopy, image analysis, and virtual sliderepository. ILAR journal, 59(1), 66-79.[2] Aguirre, A. J. (2019). Oncogenic NRG1 fusions: A new hope for targeted therapy inpancreatic cancer. Clinical Cancer Research, 25(15), 4589-4591.[3] Akiva, P., Toporik, A., Edelheit, S., Peretz, Y., Diber, A., Shemesh, R., ... & Sorek, R.(2006). Transcription-mediated gene fusion in the human genome. Genome research,16(1), 30-36.[4] Alom, M. Z., Yakopcic, C., Nasrin, M. S., Taha, T. M., & Asari, V. K. (2019). Breastcancer classification from histopathological images with inception recurrent residualconvolutional neural network. Journal of digital imaging, 32(4), 605-617.[5] Alpaydin, E. (2020). Introduction to machine learning. MIT press.[6] Aly, M. (2005). Survey on multiclass classification methods. Neural Netw, 19, 1-9.[7] Amatu, A., Sartore-Bianchi, A., & Siena, S. (2016). NTRK gene fusions as novel targetsof cancer therapy across multiple tumour types. ESMO open, 1(2).[8] Aran, D., Sirota, M., & Butte, A. J. (2015). Systematic pan-cancer analysis of tumourpurity. Nature communications, 6(1), 1-12.81[9] Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutionalencoder-decoder architecture for image segmentation. IEEE transactions on patternanalysis and machine intelligence, 39(12), 2481-2495.[10] Bankhead, P., Loughrey, M. B., Ferna´ndez, J. A., Dombrowski, Y., McArt, D. G.,Dunne, P. D., ... & James, J. A. (2017). QuPath: Open source software for digitalpathology image analysis. Scientific reports, 7(1), 1-7.[11] Basaveswara, S. K. (2019). CNN Architectures, a Deep-dive [Online image]. TowardsData Science. Retrieved from:https://towardsdatascience.com/cnn-architectures-a-deep-dive-a99441d18049[12] Baselga, J. (2001). The EGFR as a target for anticancer therapy—focus on cetuximab.European journal of cancer, 37, 16-22.[13] Bedard, P. L., Hansen, A. R., Ratain, M. J., & Siu, L. L. (2013). Tumour heterogeneityin the clinic. Nature, 501(7467), 355-364.[14] Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V., & Madabhushi, A. (2019). Arti-ficial intelligence in digital pathology—new tools for diagnosis and precision oncology.Nature reviews Clinical oncology, 16(11), 703-715.[15] Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. InProceedings of COMPSTAT’2010 (pp. 177-186). Physica-Verlag HD.[16] Bradski, G., & Kaehler, A. (2000). OpenCV. Dr. Dobb’s journal of software tools, 3.[17] Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R. L., Torre, L. A., & Jemal, A. (2018).Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality world-wide for 36 cancers in 185 countries. CA: a cancer journal for clinicians, 68(6), 394-424.[18] Butts, C., Kamel–Reid, S., Batist, G., Chia, S., Blanke, C., Moore, M., ... & Bonter, K.(2013). Benefits, issues, and recommendations for personalized medicine in oncologyin Canada. Current Oncology, 20(5), e475.82[19] Campbell, P. J., Yachida, S., Mudie, L. J., Stephens, P. J., Pleasance, E. D., Stebbings,L. A., ... & McBride, D. J. (2010). The patterns and dynamics of genomic instabilityin metastatic pancreatic cancer. Nature, 467(7319), 1109-1113.[20] Chen, M., Zhang, B., Topatana, W., Cao, J., Zhu, H., Juengpanich, S., ... & Cai, X.(2020). Classification and mutation prediction based on histopathology H&E imagesin liver cancer using deep learning. npj Precision Oncology, 4(1), 1-7.[21] Christenson, E. S., Jaffee, E., & Azad, N. S. (2020). Current and emerging therapiesfor patients with advanced pancreatic ductal adenocarcinoma: a bright future. TheLancet Oncology, 21(3), e135-e145.[22] Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., & Schmidhuber, J. (2011,June). Flexible, high performance convolutional neural networks for image classifica-tion. In Twenty-Second International Joint Conference on Artificial Intelligence.[23] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., ... & Tarbox,L. (2013). The Cancer Imaging Archive (TCIA): maintaining and operating a publicinformation repository. Journal of digital imaging, 26(6), 1045-1057.[24] Chollet, F., & others. (2015). Keras. GitHub. Retrieved fromhttps://github.com/fchollet/keras.[25] Coudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyo¨, D., ...& Tsirigos, A. (2018). Classification and mutation prediction from non–small cell lungcancer histopathology images using deep learning. Nature medicine, 24(10), 1559-1567.[26] Culibrk, L. (2020). Ploidetect. R Package.[27] De, K., & Masilamani, V. (2013). Image sharpness measure for blurred images infrequency domain. Procedia Engineering, 64, 149-158.[28] Dimitriou, N., Arandjelovic´, O., & Caie, P. D. (2019). Deep learning for whole slideimage analysis: An overview. Frontiers in Medicine, 6.83[29] do Rosa´rio Pinheiro, D., Ferreira, W. A. S., Barros, M. B. L., Arau´jo, M. D., Rodrigues-Antunes, S., & do Nascimento Borges, B. (2014). Perspectives on new biomarkers ingastric cancer: Diagnostic and prognostic applications. World Journal of Gastroen-terology : WJG, 20(33), 11574.[30] Doyle, S., Hwang, M., Shah, K., Madabhushi, A., Feldman, M., & Tomaszeweski, J.(2007, April). Automated grading of prostate cancer using architectural and texturalimage features. In 2007 4th IEEE International Symposium on Biomedical Imaging:From Nano to Macro (pp. 1284-1287). IEEE.[31] Doyle, S., Agner, S., Madabhushi, A., Feldman, M., & Tomaszewski, J. (2008, May).Automated grading of breast cancer histopathology using spectral clustering with tex-tural and architectural image features. In 2008 5th IEEE International Symposium onBiomedical Imaging: From Nano to Macro (pp. 496-499). IEEE.[32] Druker, B. J., Talpaz, M., Resta, D. J., Peng, B., Buchdunger, E., Ford, J. M., ... &Sawyers, C. L. (2001). Efficacy and safety of a specific inhibitor of the BCR-ABL ty-rosine kinase in chronic myeloid leukemia. New England Journal of Medicine, 344(14),1031-1037.[33] Farahani, N., Parwani, A. V., & Pantanowitz, L. (2015). Whole slide imaging in pathol-ogy: advantages, limitations, and emerging perspectives. Pathol Lab Med Int, 7(23-33),4321.[34] Fischer, A. H., Jacobson, K. A., Rose, J., & Zeller, R. (2008). Hematoxylin and eosinstaining of tissue and cell sections. Cold spring harbor protocols, 2008(5), pdb-prot4986.[35] Fukushima, K., & Miyake, S. (1982). Neocognitron: A self-organizing neural networkmodel for a mechanism of visual pattern recognition. In Competition and cooperationin neural nets (pp. 267-285). Springer, Berlin, Heidelberg.[36] Phung, V. H., & Rhee, E. J. (2019). A High-Accuracy Model Average Ensemble ofConvolutional Neural Networks for Classification of Cloud Image Patches on SmallDatasets. Applied Sciences, 9(21), 4500.84[37] Gurcan, M. N., Boucheron, L. E., Can, A., Madabhushi, A., Rajpoot, N. M., & Yener,B. (2009). Histopathological image analysis: A review. IEEE reviews in biomedicalengineering, 2, 147-171.[38] Gutman, D. A., Cobb, J., Somanna, D., Park, Y., Wang, F., Kurc, T., ... & Kong, J.(2013). Cancer Digital Slide Archive: an informatics resource to support integrated insilico analysis of TCGA pathology data. Journal of the American Medical InformaticsAssociation, 20(6), 1091-1098.[39] Ganster, H., Pinz, P., Rohrer, R., Wildling, E., Binder, M., & Kittler, H. (2001).Automated melanoma recognition. IEEE transactions on medical imaging, 20(3), 233-239.[40] Glotsos, D., Spyridonos, P., Cavouras, D., Ravazoula, P., Dadioti, P. A., & Niki-foridis, G. (2004). Automated segmentation of routinely hematoxylin-eosin-stained mi-croscopic images by combining support vector machine clustering and active contourmodels. Analytical and Quantitative Cytology and Histology, 26(6), 331-340.[41] Gutman, D. A., Khalilia, M., Lee, S., Nalisnik, M., Mullen, Z., Beezley, J., ... &Cooper, L. A. (2017). The digital slide archive: A software platform for management,integration, and analysis of histology for cancer research. Cancer research, 77(21),e75-e78.[42] Hamilton, P. W., Bankhead, P., Wang, Y., Hutchinson, R., Kieran, D., McArt, D. G.,... & Salto-Tellez, M. (2014). Digital pathology and image analysis in tissue biomarkerresearch. Methods, 70(1), 59-73.[43] Harris, T. J., & McCormick, F. (2010). The molecular pathology of cancer. Naturereviews Clinical oncology, 7(5), 251-265.[44] He, L., Long, L. R., Antani, S., & Thoma, G. R. (2012). Histology image analysisfor carcinoma detection and grading. Computer methods and programs in biomedicine,107(3), 538-556.85[45] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recogni-tion. In Proceedings of the IEEE conference on computer vision and pattern recognition(pp. 770-778).[46] Heining, C., Horak, P., Uhrig, S., Codo, P. L., Klink, B., Hutter, B., ... & Penzel, R.(2018). NRG1 fusions in KRAS wild-type pancreatic cancer. Cancer discovery, 8(9),1087-1095.[47] Higgins, C. (2015). Applications and challenges of digital pathology and whole slideimaging. Biotechnic & Histochemistry, 90(5), 341-347.[48] Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R.(2012). Improving neural networks by preventing co-adaptation of feature detectors.arXiv preprint arXiv :1207.0580.[49] Hosseini, M. S., Chan, L., Tse, G., Tang, M., Deng, J., Norouzi, S., ... & Damaskinos,S. (2019). Atlas of digital pathology: A generalized hierarchical histological tissuetype-annotated database for deep learning. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (pp. 11747-11756).[50] Huang, M., Shen, A., Ding, J., & Geng, M. (2014). Molecularly targeted cancer ther-apy: some lessons from the past decade. Trends in pharmacological sciences, 35(1),41-50.[51] Huang, S., Cai, N., Pacheco, P. P., Narrandes, S., Wang, Y., & Xu, W. (2018). Applica-tions of support vector machine (SVM) learning in cancer genomics. Cancer Genomics-Proteomics, 15(1), 41-51.[52] Jamal-Hanjani, M., Quezada, S. A., Larkin, J., & Swanton, C. (2015). Translationalimplications of tumor heterogeneity. Clinical cancer research, 21(6), 1258-1266.[53] Jones, M. R., Williamson, L. M., Topham, J. T., Lee, M. K., Goytain, A., Ho, J.,... & Karasinska, J. M. (2019). NRG1 gene fusions are recurrent, clinically actionablegene rearrangements in KRAS wild-type pancreatic ductal adenocarcinoma. ClinicalCancer Research, 25(15), 4674-4681.86[54] Jonna, S., Feldman, R. A., Swensen, J., Gatalica, Z., Korn, W. M., Borghaei, H., ...& Wozniak, A. J. (2019). Detection of NRG1 gene fusions in solid tumors. ClinicalCancer Research, 25(16), 4966-4972.[55] Junqueira, L. C. U., & Mescher, A. L. (2005). Junqueira’s basic histology. McGraw-HillMedical.[56] Karlik, B., & Olgac, A. V. (2011). Performance analysis of various activation functionsin generalized MLP architectures of neural networks. International Journal of ArtificialIntelligence and Expert Systems, 1(4), 111-122.[57] Kolbusz, J., Rozycki, P., & Wilamowski, B. M. (2017, June). The study of architectureMLP with linear neurons in order to eliminate the “vanishing gradient” problem. InInternational Conference on Artificial Intelligence and Soft Computing (pp. 97-106).Springer, Cham.[58] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deepconvolutional neural networks. In Advances in neural information processing systems(pp. 1097-1105).[59] Krogh, A. (2008). What are artificial neural networks?. Nature biotechnology, 26(2),195-197.[60] Ladanyi, M., & Pao, W. (2008). Lung adenocarcinoma: guiding EGFR-targeted ther-apy and beyond. Modern pathology, 21(2), S16-S22.[61] Lai, J., An, J., Seim, I., Walpole, C., Hoffman, A., Moya, L., ... & Nelson, C. C.(2015). Fusion transcript loci share many genomic features with non-fusion loci. BMCgenomics, 16(1), 1-12.[62] Latysheva, N. S., & Babu, M. M. (2016). Discovering and understanding oncogenicgene fusions through data intensive computational approaches. Nucleic acids research,44(10), 4487-4503.87[63] LeCun, Y., Touresky, D., Hinton, G., & Sejnowski, T. (1988, June). A theoreticalframework for back-propagation. In Proceedings of the 1988 connectionist models sum-mer school (Vol. 1, pp. 21-28). CMU, Pittsburgh, Pa: Morgan Kaufmann.[64] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W.,& Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition.Neural computation, 1(4), 541-551.[65] Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semanticsegmentation. In Proceedings of the IEEE conference on computer vision and patternrecognition (pp. 3431-3440).[66] Luo, J. H., Liu, S., Zuo, Z. H., Chen, R., Tseng, G. C., & Yan, P. Y. (2015). Discoveryand classification of fusion transcripts in prostate cancer and normal prostate tissue.The American journal of pathology, 185(7), 1834-1845.[67] Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J. T., Guan,X., ... & Thomas, N. E. (2009, June). A method for normalizing histology slides forquantitative analysis. In 2009 IEEE International Symposium on Biomedical Imaging:From Nano to Macro (pp. 1107-1110). IEEE.[68] Minsky, M., & Papert, S. A. (2017). Perceptrons: An introduction to computationalgeometry. MIT press.[69] Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018). Foundations of machine learn-ing. MIT press.[70] Mukesh, V. (2012). Personalized medicine and cancer. J. Pers. Med, 2(1), 1-14.[71] Nacu, S., Yuan, W., Kan, Z., Bhatt, D., Rivers, C. S., Stinson, J., ... & Wu, T. D.(2011). Deep RNA sequencing analysis of readthrough gene fusions in human prostateadenocarcinoma and reference samples. BMC medical genomics, 4(1), 11.[72] Nakkiran, P., Alvarez, R., Prabhavalkar, R., & Parada, C. (2015). Compressing deepneural networks using a rank-constrained topology.88[73] Nass, S. J., Cohen, M. B., Nayar, R., Zutter, M. M., Balogh, E. P., Schilsky, R. L.,... & Elenitoba-Johnson, K. S. (2019). Improving Cancer Diagnosis and Care: PatientAccess to High-Quality Oncologic Pathology. The oncologist, 24(10), 1287.[74] Narayan, S. (1997). The generalized sigmoid activation function: Competitive super-vised learning. Information sciences, 99(1-2), 69-82.[75] Oliphant, T. E. (2006). A guide to NumPy (Vol. 1, p. 85). USA: Trelgol Publishing.[76] Paez, J. G., Ja¨nne, P. A., Lee, J. C., Tracy, S., Greulich, H., Gabriel, S., ... & Naoki, K.(2004). EGFR mutations in lung cancer: correlation with clinical response to gefitinibtherapy. Science, 304(5676), 1497-1500.[77] Pan, S. J., & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions onknowledge and data engineering, 22(10), 1345-1359.[78] Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015). Deep face recognition.[79] Pishvaian, M. J., Rolfo, C. D., Liu, S. V., Multani, P. S., Chow Maneval, E., &Garrido-Laguna, I. (2018). Clinical benefit of entrectinib for patients with metastaticpancreatic cancer who harbor NTRK and ROS1 fusions.[80] Qin, Y., Feng, H., Chen, M., Wu, H., & Zheng, X. (2018). InfiniumPurify: an R packagefor estimating and accounting for tumor purity in cancer methylation research. Genes& diseases, 5(1), 43-45.[81] Rawlani, H. (2018, December). Visualizing CNNs using deconvolutions. Github repos-itory, https://github.com/himanshurawlani/convnet-interpretability-keras.[82] Reed, R., & MarksII, R. J. (1999). Neural smithing: supervised learning in feedforwardartificial neural networks. Mit Press.[83] Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXivpreprint arXiv :1609.04747.[84] Ryan, D. P., Hong, T. S., & Bardeesy, N. (2014). Pancreatic adenocarcinoma. NewEngland Journal of Medicine, 371(11), 1039-1049.89[85] Sarkar, D., Bali, R., & Ghosh, T. (2018). Hands-On Transfer Learning with Python:Implement advanced deep learning and neural network models using TensorFlow andKeras. Packt Publishing Ltd.[86] Sawyers, C. (2004). Targeted cancer therapy. Nature, 432(7015), 294-297.[87] Serag, A., Ion-Margineanu, A., Qureshi, H., McMillan, R., Saint Martin, M. J., Dia-mond, J., ... & Hamilton, P. (2019). Translational AI and deep learning in diagnosticpathology. Frontiers in Medicine, 6.[88] Silla, C. N., & Freitas, A. A. (2011). A survey of hierarchical classification acrossdifferent application domains. Data Mining and Knowledge Discovery, 22(1-2), 31-72.[89] Sharma, M. (2019). Understanding Images with skimage-Python [Online image]. To-wards Data Science. Retrieved from:https://towardsdatascience.com/understanding-images-with-skimage-python-b94d210afd23[90] Sharma, S. (2017). Activation functions in neural networks. Towards Data Science, 6.[91] Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scaleimage recognition. arXiv preprint arXiv :1409.1556.[92] Singhi, A. D., Ali, S. M., Lacy, J., Hendifar, A., Nguyen, K., Koo, J., ... & Zeh,H. J. (2017). Identification of targetable ALK rearrangements in pancreatic ductaladenocarcinoma. Journal of the National Comprehensive Cancer Network, 15(5), 555-562.[93] Svozil, D., Kvasnicka, V., & Pospichal, J. (1997). Introduction to multi-layer feed-forward neural networks. Chemometrics and intelligent laboratory systems, 39(1), 43-62.[94] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich,A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference oncomputer vision and pattern recognition (pp. 1-9).90[95] Tax, D. M., & Duin, R. P. (2002, August). Using two-class classifiers for multiclassclassification. In Object recognition supported by user interaction for service robots(Vol. 2, pp. 124-127). IEEE.[96] Tewari, A., & Bartlett, P. L. (2007). On the consistency of multiclass classificationmethods. Journal of Machine Learning Research, 8(May), 1007-1025.[97] Thavarajah, R., Mudimbaimannar, V. K., Elizabeth, J., Rao, U. K., & Ranganathan,K. (2012). Chemical and physical basics of routine formaldehyde fixation. Journal oforal and maxillofacial pathology: JOMFP, 16(3), 400.[98] Tomczak, K., Czerwin´ska, P., & Wiznerowicz, M. (2015). The Cancer Genome Atlas(TCGA): an immeasurable source of knowledge. Contemporary oncology, 19(1A), A68.[99] Verma, M. (2012). Personalized medicine and cancer. Journal of personalized medicine,2(1), 1-14.[100] Wang, H., Ding, C., & Huang, H. (2010, July). Multi-label classification: Inconsis-tency and class balanced k-nearest neighbor. In Twenty-Fourth AAAI Conference onArtificial Intelligence.[101] Wang, Z., Li, C., Wang, D., Wang, X., & Qi, H. (2019). Speeding up convolutionalnetworks pruning with coarse ranking. arXiv preprint arXiv:1902.06385.[102] Webster, J. D., & Dunstan, R. W. (2014). Whole-slide imaging and automated imageanalysis: considerations and opportunities in the practice of pathology. Veterinarypathology, 51(1), 211-223.[103] Xu, Y., Jia, Z., Wang, L. B., Ai, Y., Zhang, F., Lai, M., ... & Chang, C. (2017).Large scale tissue histopathology image classification, segmentation, and visualizationvia deep convolutional activation features. BMC bioinformatics, 18(1), 1-17.[104] Yachida, S., Jones, S., Bozic, I., Antal, T., Leary, R., Fu, B., ... & Velculescu, V.E. (2010). Distant metastasis occurs late during the genetic evolution of pancreaticcancer. Nature, 467(7319), 1114-1117.91[105] Yegnanarayana, B. (2009). Artificial neural networks. PHI Learning Pvt. Ltd..[106] Yonekura, A., Kawanaka, H., Prasath, V. S., Aronow, B. J., & Takase, H. (2018). Au-tomatic disease stage classification of glioblastoma multiforme histopathological imagesusing deep convolutional neural network. Biomedical engineering letters, 8(3), 321-327.[107] Yoshihara, K., Wang, Q., Torres-Garcia, W., Zheng, S., Vegesna, R., Kim, H., &Verhaak, R. G. (2015). The landscape and therapeutic relevance of cancer-associatedtranscript fusions. Oncogene, 34(37), 4845-4854.[108] Zeiler, M. D., Taylor, G. W., & Fergus, R. (2011, November). Adaptive deconvolutionalnetworks for mid and high level feature learning. In 2011 International Conference onComputer Vision (pp. 2018-2025). IEEE.92
Thesis/Dissertation
eng
Bioinformatics
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Using convolutional neural networks to predict NRG1-fusions in PDAC biopsy images
Text