Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Azizi, Shekoofeh
Doctor of Philosophy - PhD
University of British Columbia
The ultimate diagnosis of prostate cancer involves histopathology analysis of tissue samples obtained through prostate biopsy, guided by either transrectal ultrasound (TRUS), or fusion of TRUS with multi-parametric magnetic resonance imaging. Appropriate clinical management of prostate cancer requires accurate detection and assessment of the grade of the disease and its extent. Despite recent advancements in prostate cancer diagnosis, accurate characterization of aggressive lesions from indolent ones is an open problem and requires refinement. Temporal Enhanced Ultrasound (TeUS) has been proposed as a new paradigm for tissue characterization. TeUS involves analysis of a sequence of ultrasound radio frequency (RF) or Brightness (B)-mode data using a machine learning approach. The overarching objective of this dissertation is to improve the accuracy of detecting prostate cancer, specifically the aggressive forms of the disease and to develop a TeUS-augmented prostate biopsy system. Towards full-filling this objective, this dissertation makes the following contributions: 1) Several machine learning techniques are developed and evaluated to automatically analyze the spectral and temporal aspect of backscattered ultrasound signals from the prostate tissue, and to detect the presence of cancer; 2) a patient-specific biopsy targeting approach is proposed that displays near real-time cancer likelihood maps on B-mode ultrasound images augmenting their information; and 3) the latent representations of TeUS, as learned by the proposed machine learning models, are investigated to derive insights about tissue dependent features residing in TeUS and their physical interpretation. A data set consisting of biopsy targets in mp-MRI-TRUS fusion-biopsies with 255 biopsy cores from 157 subjects was used to generate and evaluate the proposed techniques. Clinical histopathology of the biopsy cores was used as the gold-standard. Results demonstrated that TeUS is effective in differentiating aggressive prostate from clinically less-significant disease and non-cancerous tissue. Evidence derived from simulation and latent-feature visualization showed that micro-vibrations of tissue microstructure, captured by low-frequency spectral features of TeUS, is a main source of tissue-specific information that can be used for detection of prostate cancer.
A Machine Learning Framework forTemporal Enhanced UltrasoundGuided Prostate Cancer DiagnosticsbyShekoofeh AziziB.Sc., Isfahan University of Technology, 2011M.Sc., Isfahan University of Technology, 2013A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Electrical and Computer Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)June 2018c© Shekoofeh Azizi 2018The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:A Machine Learning Framework for Temporal Enhanced Ultra-sound Guided Prostate Cancer Diagnosticssubmitted by Shekoofeh Azizi in partial fulfillment of the requirementsfor the degree of Doctor of Philosophy in Electrical and ComputerEngineering.Examining Committee:Purang Abolmaesumi, Electrical and Computer EngineeringSupervisorShahriar Mirabbasi, Electrical and Computer EngineeringUniversity ExaminerRoger Tam, Electrical and Computer EngineeringUniversity ExaminerZahra Moussavi, Electrical and Computer EngineeringExternal ExaminerAdditional Supervisory Committee Members:Robert Rohling, Electrical and Computer EngineeringSupervisory Committee MemberSeptimiu E. Salcudean, Electrical and Computer EngineeringSupervisory Committee MemberiiAbstractThe ultimate diagnosis of prostate cancer involves histopathology analysis oftissue samples obtained through prostate biopsy, guided by either transrectalultrasound (TRUS), or fusion of TRUS with multi-parametric magneticresonance imaging. Appropriate clinical management of prostate cancerrequires accurate detection and assessment of the grade of the disease and itsextent. Despite recent advancements in prostate cancer diagnosis, accuratecharacterization of aggressive lesions from indolent ones is an open problemand requires refinement.Temporal Enhanced Ultrasound (TeUS) has been proposed as a newparadigm for tissue characterization. TeUS involves analysis of a sequenceof ultrasound radio frequency (RF) or Brightness (B)-mode data using amachine learning approach. The overarching objective of this dissertationis to improve the accuracy of detecting prostate cancer, specifically theaggressive forms of the disease and to develop a TeUS-augmented prostatebiopsy system. Towards full-filling this objective, this dissertation makes thefollowing contributions: 1) Several machine learning techniques are developedand evaluated to automatically analyze the spectral and temporal aspect ofbackscattered ultrasound signals from the prostate tissue, and to detect thepresence of cancer; 2) a patient-specific biopsy targeting approach is proposedthat displays near real-time cancer likelihood maps on B-mode ultrasoundimages augmenting their information; and 3) the latent representations ofTeUS, as learned by the proposed machine learning models, are investigatedto derive insights about tissue dependent features residing in TeUS and theirphysical interpretation.A data set consisting of biopsy targets in mp-MRI-TRUS fusion-biopsieswith 255 biopsy cores from 157 subjects was used to generate and evaluatethe proposed techniques. Clinical histopathology of the biopsy cores wasused as the gold-standard. Results demonstrated that TeUS is effective indifferentiating aggressive prostate from clinically less-significant disease andnon-cancerous tissue. Evidence derived from simulation and latent-featurevisualization showed that micro-vibrations of tissue microstructure, capturedby low-frequency spectral features of TeUS, is a main source of tissue-specificinformation that can be used for detection of prostate cancer.iiLay SummaryProstate cancer is the most frequently diagnosed cancer and the secondleading cancer related cause of death in North American men. If detectedaccurately and managed appropriately, the long-term survival rate is high.The current clinical approach for diagnosis of prostate cancer is throughbiopsy sampling of the prostate gland and pathological examination ofthe samples. The biopsy process is guided by ultrasound images to helpthe physician with selecting the location of tissue samples. The purposeof this thesis is to improve the detection of prostate cancer, especially itsaggressive forms, using a new ultrasound technique called Temporal EnhancedUltrasound (TeUS). This technique overlays additional information aboutthe presence and distribution of prostate cancer on ultrasound images duringbiopsy, and can help improve the detection of aggressive disease.iiiPrefaceThis thesis is primarily based on six journal publications and four conferencepapers, resulting from a collaboration between multiple researchers and in-stitutes. These publications have been modified to make the thesis coherent.The author was responsible for development, implementation and evaluationof the methods and the production of the manuscripts. All co-authors havecontributed to the editing of the manuscripts and providing feedback andcomments. Ethical approvals for clinical human studies conducted for thisresearch have been provided by the ethics review board of the NationalCancer Institute, National Institutes of Health (NIH) in Bethesda, Maryland.The data description in Chapter 2 and the study from Chapter 3 ispresented at:â Shekoofeh Azizi, Farhad Imani, Bo Zhuang, Amir Tahmasebi, Jin TaeKwak, Sheng Xu, Nishant Uniyal, Baris Turkbey, Peter Choyke, PeterPinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi.Ultrasound-based detection of prostate cancer using automatic featureselection with deep belief networks. In Medical Image Computingand Computer Assisted Intervention (MICCAI), pages 70â77. Springer,2015.â Shekoofeh Azizi, Farhad Imani, Sahar Ghavidel, Amir Tahmasebi,Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Detectionof prostate cancer using temporal sequences of ultrasound data: a largeclinical feasibility study. International Journal of Computer AssistedRadiology and Surgery, 11(6):947â956, 2016.â Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi,Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Deeprecurrent neural networks for prostate cancer detection: Analysis ofivPrefacetemporal enhanced ultrasound. IEEE Transactions on Medical Imaging,2018.The contribution of the author was in developing the method, imple-menting and evaluating the method and writing the manuscript. Dr. MehdiMoradi provided valuable scientific inputs to improve the proposed methodfor spectral analysis of TeUS. Dr. Farhad Imani extensively contributed inthe process of data preparation, generating the target location maps, andpreprocessing of TeUS data. Sahar Ghavidel contributed to the proposedspectral feature visualization. Simon Dahonick developed the preliminary ver-sion of the codes for overlaying the prostate cancer likelihood maps on TRUS.The study from Chapter 4 is presented at:â Shekoofeh Azizi, Farhad Imani, Jin Tae Kwak, Amir Tahmasebi, ShengXu, Pingkun Yan, Jochen Kruecker, Baris Turkbey, Peter Choyke, PeterPinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi.Classifying cancer grades using temporal ultrasound for transrectalprostate biopsy. In Medical Image Computing and Computer AssistedIntervention (MICCAI), pages 653â661. Springer, 2016.â Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, GuyNir, Jin Tae Kwak, Sheng Xu, Storey Wilson, Kenneth A Iczkowski,M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean, Peter Pinto,Bradford Wood, Purang Abolmaesumi, and Parvin Mousavi. Detectionand grading of prostate cancer using temporal enhanced ultrasound:combining deep neural networks and tissue mimicking simulations.MICCAIâ16 Special Issue: International Journal of Computer AssistedRadiology and Surgery, 12(8):1293â1305, 2017.â Shekoofeh Azizi, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, ShengXu, Baris Turkbey, Peter Choyke, Peter Pinto, Bradford Wood, ParvinMousavi, and Purang Abolmaesumi. Learning from noisy label statis-tics: Detecting high grade prostate cancer in ultrasound guided biopsy.In Medical Image Computing and Computer Assisted Intervention(MICCAI). Springer, 2018.The contribution of the author was in developing the methods, prepa-ration of data, implementing and evaluating the methods and writing thevPrefacemanuscripts.The study from Chapter 5 is presented at:â Shekoofeh Azizi, Parvin Mousavi, Pingkun Yan, Amir Tahmasebi,Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, and Purang Abolmaesumi. Transfer learning from RFto B-mode temporal enhanced ultrasound features for prostate cancerdetection. International Journal of Computer Assisted Radiology andSurgery, 12(7):1111â1121, 2017.â Shekoofeh Azizi, Nathan Van Woudenberg, Samira Sojoudi, MingLi, Sheng Xu, Emran M Abu Anas, Pingkun Yan, Amir Tahmasebi,Jin Tae Kwak, Baris Turkbey, Peter Choyke, Peter Pinto, BradfordWood, Parvin Mousavi, and Purang Abolmaesumi. Toward a real-time system for temporal enhanced ultrasound-guided prostate biopsy.International Journal of Computer Assisted Radiology and Surgery,pages 1â9, 2018.The contribution of the author was in developing the methods, imple-menting and evaluating the methods and writing the manuscripts. Dr. MingLi acquired the data for the second independent MRI-TRUS fusion biopsystudy presented in Chapter 5. Samira Sojoudi and Nathan Van Woudenbergin collaboration with the author developed the software solution that ispartly presented in Chapter 5 and Appendix B.In all of the studies presented in Chapter 3, 4, and 5, Dr. Bradford Woodand Dr. Peter Pinto performed the biopsy procedures, with technical supportfrom Dr. Amir Tahmasebi, Dr. Sheng Xu, Dr. Pingkun Yan and Dr. Jochenkruecker. Dr. Baris Turkbey and Dr. Peter Choyke provided the radiologicalreadings from mp-MRI for target identification. Dr. Jin Tae Kwak acquiredthe data at NIH, matched the data to pathology reports and provide us withanonymized, deidentified data for analysis. Dr. Farhad Imani contributed tothe data preparation.The study from Chapter 6 is partly presented at:â Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, GuyNir, Jin Tae Kwak, Sheng Xu, Storey Wilson, Kenneth A Iczkowski,M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean, Peter Pinto,Bradford Wood, Purang Abolmaesumi, and Parvin Mousavi. Detectionand grading of prostate cancer using temporal enhanced ultrasound:viPrefacecombining deep neural networks and tissue mimicking simulations.MICCAIâ16 Special Issue: International Journal of Computer AssistedRadiology and Surgery, 12(8):1293â1305, 2017â Sharareh Bayat, Shekoofeh Azizi, Mohammad I Daoud, Guy Nir, FarhadImani, Carlos D Gerardo, Pingkun Yan, Amir Tahmasebi, FrancoisVignon, Samira Sojoudi, et al. Investigation of physical phenomenaunderlying temporal enhanced ultrasound as a new diagnostic imagingtechnique: Theory and simulations. IEEE Transactions on Ultrasonics,Ferroelectrics, and Frequency Control, 2017â Sharareh Bayat, Farhad Imani, Carlos D Gerardo, Guy Nir, ShekoofehAzizi, Pingkun Yan, Amir Tahmasebi, Storey Wilson, Kenneth AIczkowski, M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean,Parvin Mousavi, and Purang Abolmaesumi. Tissue mimicking simula-tions for temporal enhanced ultrasound-based tissue typing. In SPIEMedical Imaging, pages 101390Dâ101390D. International Society forOptics and Photonics, 2017Dr. Sharareh Bayat performed the numerical and TeUS simulation pre-sented in these papers and extensively contributed to the investigation ofunderlying physical phenomena of TeUS which is presented in Chapter 6 andAppendix A. The author contributed by performing the data preparation,data analysis, statistical investigations and evaluating the results in collabo-ration with Dr. Sharareh Bayat. Dr. Francois Vignon and Dr. MohammadDaoud contributed to the physical investigation of TeUS. Dr. Storey Wilson,Dr. Kenneth A. Iczkowski, and Dr. M. Scott Lucia provided the digitalpathology whole-mount slides along the gold-standard labels that we usedin the simulations. Dr. Guy Nir and Dr. Septimiu Salcudean providedthe segmentation of nuclei in those slides. Dr. Larry Goldenberg providedclinical support from the Vancouver Prostate Centre.Finally, scientific inputs and insight of Prof. Purang Abolmaesumi andProf. Parvin Mousavi helped with development, implementation and eval-uation of all of the proposed methods in the above publications and thisthesis. They significantly contributed to editing and improvement of themanuscriptsâ structure through their valuable comments and feedback.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Prostate Cancer Diagnosis . . . . . . . . . . . . . . . . . . . 11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2.1 Ultrasound Techniques for Prostate Cancer Diagnosis 31.2.1.1 Texture-based and Spectral Analysis . . . . 31.2.1.2 Elastography . . . . . . . . . . . . . . . . . 51.2.1.3 Doppler Imaging . . . . . . . . . . . . . . . 61.2.1.4 Ultrasound Time Series Analysis . . . . . . 71.2.2 Machine Learning Approaches . . . . . . . . . . . . . 91.2.2.1 Feature Generation and Classification . . . . 91.2.2.2 Hidden Markov Models . . . . . . . . . . . . 101.2.2.3 Deep Learning Approaches . . . . . . . . . . 111.3 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . 111.3.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . 131.3.2 Contributions . . . . . . . . . . . . . . . . . . . . . . 14viiiTable of Contents1.3.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . 172 Temporal Enhanced Ultrasound Data . . . . . . . . . . . . . 212.1 Data Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . 222.2 Histopathology Labeling . . . . . . . . . . . . . . . . . . . . 232.3 Preprocessing and Region of Interest . . . . . . . . . . . . . 252.3.1 Time-domain Representation of TeUS . . . . . . . . . 272.3.2 Spectral-domain Representation of TeUS . . . . . . . 272.4 Complementary Second Retrospective TeUS Study . . . . . . 273 Detection of Prostate Cancer Using TeUS . . . . . . . . . . 293.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2 Spectral Analysis of TeUS for prostate cancer diagnosis . . . 313.2.1 Background: Deep Belief Networks . . . . . . . . . . 313.2.1.1 Restricted Boltzmann Machine . . . . . . . 313.2.1.2 Deep Belief Network . . . . . . . . . . . . . 333.2.2 Classification Framework Based on DBN . . . . . . . 343.2.2.1 Automatic Feature Learning . . . . . . . . . 343.2.2.2 Cancer Classification . . . . . . . . . . . . . 353.2.2.3 Spectral Feature Visualization . . . . . . . . 363.2.3 Results and Discussion . . . . . . . . . . . . . . . . . 373.2.3.1 Data Division . . . . . . . . . . . . . . . . . 373.2.3.2 Hyper-parameter Selection . . . . . . . . . . 383.2.3.3 Classification Performance . . . . . . . . . . 393.2.3.4 Choice of Training Data . . . . . . . . . . . 403.2.3.5 Colormaps . . . . . . . . . . . . . . . . . . . 413.2.3.6 Analysis of Tumor Size . . . . . . . . . . . . 413.2.3.7 Feature Visualization . . . . . . . . . . . . . 433.3 Temporal Analysis of Temporal Enhanced Ultrasound . . . . 443.3.1 Background: Recurrent Neural Networks . . . . . . . 443.3.2 Classification Framework Based on RNN . . . . . . . 473.3.2.1 Proposed Discriminative Method . . . . . . 473.3.2.2 Cancer Classification . . . . . . . . . . . . . 483.3.2.3 Network Analysis . . . . . . . . . . . . . . . 483.3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . 493.3.3.1 Data Division . . . . . . . . . . . . . . . . . 493.3.3.2 Hyper-parameter Selection . . . . . . . . . . 493.3.3.3 Model Training and Evaluation . . . . . . . 513.3.3.4 Implementation . . . . . . . . . . . . . . . . 523.3.4 Results and Discussion . . . . . . . . . . . . . . . . . 52ixTable of Contents3.3.4.1 Model Selection . . . . . . . . . . . . . . . . 523.3.4.2 Model Performance . . . . . . . . . . . . . . 533.3.4.3 Comparison with Other Methods . . . . . . 543.3.4.4 Network Analysis . . . . . . . . . . . . . . . 553.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574 Detection of High-Grade Prostate Cancer Using TeUS . . 594.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.2 Prostate Cancer Grading Using Spectral Analysis of TeUS . 614.2.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . 614.2.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . 624.2.2.1 Feature Learning . . . . . . . . . . . . . . . 624.2.2.2 Distribution Learning . . . . . . . . . . . . . 644.2.2.3 Prediction of Gleason Score . . . . . . . . . 644.2.3 Results and Discussion . . . . . . . . . . . . . . . . . 654.2.3.1 Prostate Cancer Detection and Grading . . 664.2.3.2 Integration of TeUS and mp-MRI . . . . . . 674.2.3.3 Sensitivity Analysis . . . . . . . . . . . . . . 684.3 Temporal Analysis of TeUS for prostate cancer grading . . . 714.3.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . 734.3.1.1 Discriminative Model . . . . . . . . . . . . . 734.3.1.2 Cancer Grading and Tumor in Core LengthEstimation . . . . . . . . . . . . . . . . . . . 754.3.1.3 Model Uncertainty Estimation . . . . . . . . 764.3.2 Experiments and Results . . . . . . . . . . . . . . . . 764.3.2.1 Data Division and Model Selection . . . . . 764.3.2.2 Comparative Method and Baselines . . . . . 774.3.2.3 Tumor in Core Length Estimation . . . . . 784.3.2.4 Cancer Likelihood Colormaps . . . . . . . . 784.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 795 Decision Support System for Prostate Biopsy Guidance . 815.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.2 Transfer Learning From TeUS RF to B-mode Spectral Features 835.2.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . 845.2.1.1 Unlabeled Data . . . . . . . . . . . . . . . . 845.2.1.2 Labeled Data . . . . . . . . . . . . . . . . . 845.2.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . 845.2.2.1 Unsupervised Domain Adaptation . . . . . . 865.2.2.2 Supervised Classification . . . . . . . . . . . 88xTable of Contents5.2.2.3 Baseline Classification . . . . . . . . . . . . 885.2.2.4 Generalization . . . . . . . . . . . . . . . . . 885.2.3 Results and Discussion . . . . . . . . . . . . . . . . . 885.2.3.1 Unsupervised Domain Adaptation . . . . . . 895.2.3.2 Supervised Classification . . . . . . . . . . . 905.2.3.3 Baseline Classification . . . . . . . . . . . . 925.2.3.4 Generalization . . . . . . . . . . . . . . . . . 935.2.3.5 Colormaps: . . . . . . . . . . . . . . . . . . 955.3 Transfer Learning From TeUS RF to B-mode Using RNN . . 955.3.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . 965.3.1.1 Data Division . . . . . . . . . . . . . . . . . 975.3.1.2 Complementary Second Retrospective Study 975.3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . 985.3.3 Results and Discussion . . . . . . . . . . . . . . . . . 995.3.3.1 Classification model validation . . . . . . . . 995.3.3.2 System assessment . . . . . . . . . . . . . . 1005.3.4 Discussion and comparison with other methods . . . 1015.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1026 Investigation of Physical Phenomena Underlying TeUS . . 1046.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.2 Spectral Feature Visualization . . . . . . . . . . . . . . . . . 1066.2.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . 1066.2.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . 1066.3 Histopathology Mimicking Simulation . . . . . . . . . . . . . 1086.3.1 Digital Pathology Data . . . . . . . . . . . . . . . . . 1086.3.2 Numerical Simulation Design . . . . . . . . . . . . . . 1086.3.3 TeUS Simulation . . . . . . . . . . . . . . . . . . . . . 1096.4 Experiments and Results . . . . . . . . . . . . . . . . . . . . 1106.4.1 Feature Visualization Results . . . . . . . . . . . . . . 1106.4.2 Simulation Results . . . . . . . . . . . . . . . . . . . . 1116.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1127 Conclusion and Future Work . . . . . . . . . . . . . . . . . . 1137.1 Conclusion and Summary . . . . . . . . . . . . . . . . . . . . 1137.2 Future Work and Suggestions for Further Development . . . 117Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119xiTable of ContentsAppendicesA Theoretical Background of Temporal Enhanced Ultrasound 140B TeUS Biopsy Guidance System Implementation . . . . . . 143B.1 TeUS biopsy guidance system . . . . . . . . . . . . . . . . . 143B.1.1 TeUS-client . . . . . . . . . . . . . . . . . . . . . . . . 144B.1.2 TeUS Server . . . . . . . . . . . . . . . . . . . . . . . 146xiiList of Tables2.1 Details of equipment and imaging parameters used for TeUSdata collection. . . . . . . . . . . . . . . . . . . . . . . . . . . 232.2 Gleason score distribution in the first retrospective TeUS dataset. 252.3 Gleason score distribution in the second retrospective TeUSstudy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.1 Gleason score distribution in TeUS test and train dataset.Table shows the number of cores for each category. . . . . . . 383.2 Model performance for classification of testing cores for differ-ent MR suspicious levels. N indicates the number of cores ineach group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.3 Model performance in the fold validation analysis for testingcores in datasets DAtest and DBtest. . . . . . . . . . . . . . . . . 413.4 Model performance for classification of cores in the test data(N = 171). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.5 Model performance for classification of cores in the test datafor Moderate MR suspicious level. N indicates the number ofcores in each group. . . . . . . . . . . . . . . . . . . . . . . . 543.6 Model performance for classification of cores in the test datafor High MR suspicious level. N indicates the number of coresin each group. . . . . . . . . . . . . . . . . . . . . . . . . . . . 544.1 Gleason score distribution in TeUS test and train dataset.Table represents the number of cores for each category. . . . . 624.2 Model performance for prostate cancer grading in the testdataset using TeUS only and by integration of TeUS andmp-MRI. L is the largest length of the tumor visible in mp-MRI. 664.3 Model performance for classification of cancerous vs. non-cancerous cores in the test dataset using TeUS only and Inte-gration of TeUS and mp-MRI. L is the greatest length of thetumor visible in mp-MRI. . . . . . . . . . . . . . . . . . . . . 68xiiiList of Tables4.4 Model performance for classification of cores in the test data(N = 170). AUC1, AUC2 and AUC3 refer to detection ofBenign vs. GSâ¤3+4, Benign vs. GSâ¥4+3, and GSâ¤3+4 vs.GSâ¥4+3, respectively. . . . . . . . . . . . . . . . . . . . . . . 775.1 Model performance measured by AUC for classification indifferent data divisions. . . . . . . . . . . . . . . . . . . . . . 925.2 Model performance measured by specificity and sensitivity forclassification in different data divisions. . . . . . . . . . . . . 925.3 Performance for the combination of mp-MRI and TeUS mea-sured by AUC for classification in different data divisions. . . 935.4 Performance for the combination of mp-MRI and TeUS mea-sured by specificity and sensitivity for classification in differentdata divisions. . . . . . . . . . . . . . . . . . . . . . . . . . . 945.5 Comparison of model performance measured by AUC usingbaselines and the proposed approach in different data divisions. 945.6 Performance for the TeUS only and combination of mp-MRIand TeUS measured by AUC, specificity, and sensitivity forclassification in the test dataset. . . . . . . . . . . . . . . . . 945.7 Gleason score distribution in the second retrospective clinicalstudy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 985.8 Model performance for classification of cores in the test datafrom the first retrospective study for different MR suspiciouslevels. N indicates the number of cores in each group. . . . . 1005.9 Run-time of the steps of the prostate guidance system averagedover N = 21 trials with data from the second retrospectivestudy (given as mean±std). . . . . . . . . . . . . . . . . . . . 101xivList of Figures1.1 A schematic diagram of Temporal Enhanced Ultrasound (TeUS)data generation. A time series of a fixed point in an imageframe, shown as a red dot, is analyzed over a sequence ofultrasound frames to determine tissue characteristics. . . . . . 81.2 A schematic diagram of TeUS-based workflow for prostatebiopsy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.1 UroNav MR/US fusion system: The identified mp-MRI lesionswere delineated on the T2-weighted MR image as the biopsytargets. The target location is shown by the green point alongthe projected needle path in the ultrasound image. . . . . . 232.2 Statistics of histopathology and MR readings in our TeUSdataset: Histopathology reports include the Gleason Score(GS) and the percentage distribution of prostate cancer. TheMR scores were grouped into three descriptors of âlowâ, âmod-erateâ and âhighâ, and referred to as the MR suspicious levelassigned to the area. . . . . . . . . . . . . . . . . . . . . . . . 242.3 Example of distance maps and their corresponding B-modeand RF frames: (a) RF distance map, (b) RF frame, (c)B-mode distance map, (d) B-mode frame. The dark blue isshowing the target location and the color spectrum from blueto yellow is showing farther distance from the target. . . . . . 262.4 Preprocessing and ROI selection: the target region is dividedto 80 ROIs of size 0.5 mmÃ0.5 mm and then a sliding windowis used for the data augmentation. . . . . . . . . . . . . . . . 263.1 An illustration of a Restricted Boltzmann Machine (RBM):RBM consists of a layer of binary stochastic visible units v,connected to a layer of stochastic hidden units h by symmet-rically weighted connections W. . . . . . . . . . . . . . . . . . 31xvList of Figures3.2 An illustration of the proposed method for prostate cancerdetection. Our DBN has a layer of real-valued visible units ofdimension F = 50 and four hidden layers with 100, 50 and 6hidden units. The red box contains the pre-trained DBN, andthe blue box containing one neuron is added for the fine-tuningstep. The latent features are the output of the last layer ofDBN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.3 An illustration of the proposed feature visualization method. 373.4 Cancer probability maps overlaid on B-mode ultrasound im-age, along with the projected needle path in the temporalultrasound data and centered on the target. The ROIs forwhich the cancer likelihood is more than 70% are colored inred, otherwise they are colored in blue. The green boundaryshows the segmented prostate in MRI projected in TRUScoordinates, dashed line shows needle path and the arrowpointer shows the target: (a) Correctly identified benign core;(b) Correctly identified the cancerous core. . . . . . . . . . . . 423.5 Investigation of the effect of tumor size on accuracy. Weobtained the average AUC of 0.77 for cores with MR-tumor-size smaller than 1.5 cm, and the average AUC of 0.93 forcores with MR-tumor-size larger than 2 cm. . . . . . . . . . . 423.6 Differences of distributions between cancerous and benigntissue back projected in the input neurons: (a) corresponds tothe first neuron in the third hidden layer; (b) corresponds tothe sixth neuron in the third hidden layer. Results are shownin the frequency range of temporal ultrasound data analyzedin this section. It is clear that frequencies between 0â 2 Hzprovide the most discriminative features for distinguishingcancerous and benign tissue. . . . . . . . . . . . . . . . . . . . 433.7 Overview of the proposed method. We use two layers ofRNNs with LSTM cells to model the temporal informationin a sequence of TeUS data. x(i) = (x1, , ..., xT ), T = 100 isshowing the ith sequence data and xt is indicating the tth timestep. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.8 Comparison between optimizer performance for different RNNcells: Each curve corresponds to an RNN network structurewith two hidden layers, batch size of 128 with the dropoutrate of 0.2 and regularization term of 0.0001. . . . . . . . . . 51xviList of Figures3.9 Learning curves of different RNN cells using the optimumhyper-parameters in our search space. All of the models usethe RMSprop optimizer and converge after 65±7 epochs. . . . 533.10 Cancer likelihood maps overlaid on B-mode ultrasound images,along with the projected needle path in the TeUS data, andcentered on the target. Red indicates predicted labels ascancer, and blue indicates predicted benign regions. Theboundary of the segmented prostate in MRI is overlaid onTRUS data. The arrow points to the target location. Thetop row shows the result of LSTM and the bottom row showsthe result of spectral analysis [11] for benign targets (a), andcancer targets (b) and (c). . . . . . . . . . . . . . . . . . . . . 563.11 Sequence Length effect: The most discriminative features fordetection of prostate cancer can be learned from a fraction ofthe full TeUS time series. . . . . . . . . . . . . . . . . . . . . 574.1 An illustration of the proposed cancer grading approach usingspectral analysis of TeUS. . . . . . . . . . . . . . . . . . . . . 634.2 An illustration of the proposed GMM initialization method. . 654.3 Cancer likelihood maps overlaid on B-mode US image, alongwith the projected needle path in the TeUS data and centeredon the target. The ROIs for which we detect as Gleason gradeof 4 and 3 are colored in red and yellow, respectively. Thenon-cancerous ROIs are colored as blue. The red boundaryshows the segmented prostate in MRI projected in TRUScoordinates and the arrow pointer shows the target. . . . . . 674.4 Target location and distribution of biopsies in the test data.Light and dark gray indicate central and peripheral zones,respectively. The pie charts show the number of cores andtheir histopathology. The size of the chart is proportionalto the number of biopsies (in the range from 1 to 25), andthe colors dark red, light red and blue refer to cores withGSâ¥ 4+3, GSâ¤ 3+4 and benign pathology, respectively. Thetop, middle, and bottom rows depict histopathology results,TeUS prediction, and integration of TeUS and MRI, respectively. 694.5 Model performance for prostate cancer grading using spectralanalysis of TeUS and distribution learning in the test datasetand permutation set. . . . . . . . . . . . . . . . . . . . . . . . 704.6 Model performance for different sizes of training dataset usingspectral analysis of TeUS and distribution learning. . . . . . . 70xviiList of Figures4.7 Model performance versus the number of features that we usedto generate the final model: (a) For all of the cores; (b) forcores with MR-tumor-sizeâ¥ 2.0 cm. Decreasing the numberof features improves the model performance. . . . . . . . . . 714.8 Illustration of noisy and not finely annotated ground-truthlabel. The exact location of the cancerous ROI in the core,the ratio of the different Gleason grade, and the exact locationof the Gleason grades are unknown and noisy. The bottomvectors show one of the possible multi-label binarization ap-proaches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.9 Overview of the second stage in the proposed method: thegoal of this stage is to assign a pathological score to a sample.To mitigate the problem of imperfect and noisy labels, weembed the length of cancer in the ground-truth probabilityvector as a soft label. . . . . . . . . . . . . . . . . . . . . . . . 744.10 Scatter plot of the reported tumor in core length in histopathol-ogy vs. the predicted tumor in core length. . . . . . . . . . . 794.11 (a) Cancer likelihood maps overlaid on B-mode US image,along with the projected needle path in the TeUS data (GSâ¥ 4 + 3) and centered on the target. The ROIs of size 0.5Ã0.5 mmÃmm for which we detect the Gleason grade of 4 and 3are colored in red and yellow, respectively. The non-cancerousROIs are colored as blue. (b) The red boundary shows thesegmented prostate in MRI projected in TRUS coordinatesand the arrow pointer shows the target.[blue=low uncertainty,red=high uncertainty] . . . . . . . . . . . . . . . . . . . . . . 805.1 An illustration of the proposed approach for domain adap-tation between RF and B-mode time series data in TeUSframework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855.2 Learning curve for DBN training based on the cross-entropy:(a) for first hidden layer size. (b) for different learning rates(LR). (c) for different mini-batch size (BS). In a coarse searchfor the meta-parameters we achieved the lowest cross entropyloss with n = 44, LR = 0.2, and BS = 10. . . . . . . . . . . . 905.3 Distribution shift from B-mode to RF for the top three fea-tures before (top row) and after (bottom row) the shareddeep network. The proposed domain adaptation method caneffectively align features and reduce the distribution shift incommon learned feature space. . . . . . . . . . . . . . . . . . 91xviiiList of Figures5.4 Influence of labeled dataset size in classification accuracy:performance of the method measured by AUC, accuracy, sen-sitivity and specificity in the k-fold cross-validation setting for(a) TeUS RF data and (b) TeUS B-mode data. . . . . . . . . 935.5 The comparative performance of the proposed method mea-sured by AUC over the baselines for DStest . . . . . . . . . . . 955.6 Cancer probability maps overlaid on B-mode US image, alongwith the projected needle path in the temporal US data andcentered on the target. The ROIs for which the cancer like-lihood is more than 50% are colored in red, otherwise theyare colored as blue. The red boundary shows the segmentedprostate in MRI projected in TRUS coordinates, dashed lineshows needle path and the arrow pointer shows the target:(a)-(c) Correctly identified cancerous core using RF time seriesdata; (b)-(d) Correctly identified cancerous core using B-modetime series data. . . . . . . . . . . . . . . . . . . . . . . . . . 965.7 Guidance interface implemented as part of a 3D Slicer module:cancer likelihood map is overlaid on B-mode ultrasound images.Red indicates predicted labels as cancer, and blue indicatespredicted benign regions. The boundary of the segmentedprostate is shown with white and the green circle is centeredaround the target location which is shown in the green dot. . 1006.1 An illustration of the proposed feature visualization method. 1076.2 Pathology mimicking simulations framework. . . . . . . . . . 1086.3 ROI selection and nuclei-based scatterer generation process:(a) Sample of histopathology slide [70], where the red boundarydepicts the cancer area; (b) digitized slide overlaid on thehistopathology slide, where green and red areas represent thebenign and cancer regions, respectively. The selected ROIs areshown by black squares; (c) extracted nuclei positions in theselected ROIs; left: a cancer region, right: a benign region; (d)the extracted positions of nuclei from each ROI is embeddedin an FEM model. . . . . . . . . . . . . . . . . . . . . . . . . 1096.4 (a) A sample whole-mount histopathology slide of the prostate.Different regions of cancer and benign tissue are shown in thepathology slide. (b) The corresponding simulated B-modeultrasound image. . . . . . . . . . . . . . . . . . . . . . . . . 110xixList of Figures6.5 (a) Differences of distributions between cancerous tissues withGleason patterns 3 and 4 as well as benign tissues back pro-jected in the input neurons corresponds to the first neuron inthe third hidden layer; (b) Spectral difference of the simulatedTeUS in benign and different cancer tissues. . . . . . . . . . . 1116.6 (a) Distribution of the power spectrum in the frequency spec-trum of simulated TeUS data at the excitation frequency, (b)Distribution of the power spectrum in the frequency spectrumof simulated TeUS data at the first harmonic of the excitationfrequency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112B.1 Overview of the biopsy guidance system. The three steps in theguidance workflow are volume acquisition, classification andguidance. A client-server approach allows for simultaneous andreal-time execution of computationally expensive algorithmsincluding TeUS data classification, and prostate boundarysegmentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 144B.2 The software system has a three-tiered architecture. Ovalsrepresent processing elements while arrows show the directionof data flow. In the US machine layer, PLUS is responsiblefor US data acquisition and communicates with the TeUS-client via the OpenIGTLink protocol. The TeUS client layerincludes TeUS guidance, an extension module within the 3DSlicer framework. The TeUS-server layer is responsible forthe simultaneous and real-time execution of computationallyexpensive algorithms and communicates with TeUS-client viathe OpenIGTLink protocol. . . . . . . . . . . . . . . . . . . . 145xxGlossaryAUC Area Under the ROC CurveB-mode Brightness ModeCD Contrastive DivergenceCNNs Convolutional Neural NetworksDBN Deep Belief NetworksDNN Deep Neural NetworksDFT Discrete Fourier TransformDRE Digital Rectal ExamDCE Dynamic Contrast EnhancedDWI Diffusion Weighted ImagingEM ElectroMagneticallyGMM Gaussian Mixture ModelGPU Graphics Processing UnitGRU Gated Recurrent UnitGS Gleason ScoreHMMs Hidden Markov ModelsFEM Finite Element ModelFWHM Full Width at Half MaximumICA Independent Component AnalysisKL KullbacâLeiblerxxiGlossaryLSTM Long Short-Term MemoryMRE Magnetic Resonance ElastographyMRI Magnetic Resonance ImagingMSE Mean Squared Errormp-MRI multi-parametric MRINCI National Cancer InstituteNIH National Institutes of HealthPCa Prostate CancerPCA Principal Component AnalysisPSA Prostate Specific AntigenPSF Point Spread FunctionRBF Radial Basis FunctionRBM Restricted Boltzmann MachineRF Radio FrequencyRFE Recursive Feature EliminationRNNs Recurrent Neural NetworksROC Receiver Operating CharacteristicROI Region of InterestSEER Surveillance, Epidemiology, and End ResultsSDM Subspace Disagreement MeasureSGD Stochastic Gradient DescentSVM Support Vector MachineTeUS Temporal Enhanced UltrasoundTRUS Transrectal UltrasoundUS UltrasoundxxiiAcknowledgementsFirst and foremost, I would like to express my sincerest gratitude to Prof.Purang Abolmaesumi, my advisor, for the opportunities you gave to me.Thank you for your trust, patience and invaluable support that opened thischapter of my life. Thanks for the generosity, advice and friendship youoffered me throughout my experience at UBC.I am very grateful for being associated with Prof. Parvin Mousavi, whoseinsightful advices helped me throughout the course of this endeavor. Herpatience, dedication, mentorship and sincere advice in every single step ofthe thesis have been truly crucial. Thanks for being always supportive andenthusiastic.I would like to thank supervisory committee Professors Salcudean andRohling for their valuable comments and advices.I would like to thank the Natural Sciences and Engineering ResearchCouncil of Canada (NSERC), the Canadian Institutes of Health Research(CIHR), Philips Research North America, and UBC for funding this work.xxiiiDedicationI dedicate this to my parents who left fingerprints of grace on my life; myfamily without whom none of my success would be possible, you are mywings to fly.xxivChapter 1IntroductionIf I have seen farther it is by standing on the shoulders of Giants.â Sir Isaac Newton1.1 Prostate Cancer DiagnosisProstate Cancer (PCa) is a significant public health issue, and approximately14% of men will be diagnosed with this disease at some point during theirlifetime 1. According to the American and Canadian Cancer Societies,prostate cancer accounts for 24% of all new cancer cases and results in 33,600deaths per year in North America 2. If diagnosed in the early stages, it canbe managed with a long-term disease-free survival rate above 90%; even if itis identified later, interventions can be used to increase the life expectancyof patients. The prostate cancer-related death rate has declined significantly(almost 4% per annum) between 2001 and 2009 due to improved testing andbetter treatment options. The majority of the cases diagnosed today are theearly-stage disease, where several treatment options are available, includingsurgery, brachytherapy, thermal ablation, external beam therapy, and activesurveillance. Early detection and accurate staging of prostate cancer areessential to the selection of optimal treatment options. Hence, reducing thedisease-associated morbidity and mortality [134].Currently, prostate cancer detection is carried out by a combination ofDigital Rectal Exam (DRE), measurement of the Prostate Specific Anti-gen (PSA) level, and histological assessment of biopsy samples. DRE isthe most common and least expensive way to screen for prostate cancer.However, DRE is only effective for detecting late-stage prostate cancer inthe peripheral zone of the gland, and any abnormalities located in otherprostate zones cannot be felt. The PSA test measures the blood level of1National Cancer Institute (NCI): Surveillance, Epidemiology, and End Results (SEER)Cancer Statistics Review2Canadian cancer society: http://www.cancer.ca/, and American cancer society: http://www.cancer.org/11.1. Prostate Cancer DiagnosisPSA, a protein that is produced by the prostate gland and can be usedas a biological marker for tumors. The elevated levels often indicate thepresence of prostate cancer; however, it also increases by inflammation of theprostate gland (prostatitis), and when prostate enlarges with age (benignprostatic hyperplasia). Therefore, a reliable diagnosis cannot be performedby these two procedures [98]. The definite diagnosis of prostate cancer iscore needle biopsy, under Transrectal Ultrasound (TRUS) guidance. Thebiopsy procedure entails systematic sampling of the prostate followed byhistopathology examination of the sampled tissue. In systematic biopsy, upto 12 cores are taken from predefined anatomical locations. ConventionalUltrasound (US) imaging is not capable of distinguishing cancerous andnormal tissue with high specificity and sensitivity. Therefore, the biopsyprocedure is blind to intraprostatic pathology and can miss clinically signifi-cant disease. The sensitivity of conventional systematic biopsy under TRUSguidance, for detection of prostate cancer, has been reported to be as low as40% [39, 56, 126, 134]. Significant improvement of TRUS-guided prostatecancer biopsy is required to decrease the rate of over-treatment for low-riskdisease while preventing the under-treatment of high-risk cancer [89].Several methods have been proposed to alleviate this issue by enablingpatient-specific targeting to improve the detection rate of prostate cancer.Fusion of Magnetic Resonance Imaging (MRI) and TRUS-guided biopsy [41,67, 122] as an emerging technology for patient-specific targeting is beinggradually adopted and has shown significant potential for improved canceryield [31, 119]. A meta-analysis of seven multi-parametric MRI (mp-MRI)studies with 526 patients shows specificity of 0.88 and sensitivity of 0.74, withnegative predictive values ranging from 0.65 to 0.94 [37]. Although usingmp-MRI in fusion biopsy has resulted in the best clinical results to date,recent studies suggest the high sensitivity of mp-MRI in the detection ofprostate lesions but low specificity [3], hence, limiting its utility in detectingdisease progression over time [145]. This approach has other limitations: (1)mp-MRI is often unfamiliar to the biopsying clinician; (2) the co-alignmentof mp-MRI and TRUS is challenging [80, 81, 149]; and (3) mp-MRI is notspecific for detecting prostate cancer with intermediate risk. Moreover,limited accessibility and high expense of MRI make an ultrasound-basedprostate cancer detection system more preferable. The advantages of anultrasound-based system are several folds: TRUS is already accepted as thestandard prostate biopsy guidance tool; different ultrasound data and imageacquisition techniques are simultaneously available on ultrasound machines,and ultrasound imaging is among the most accessible and least harmfulmedical imaging approaches.21.2. Background1.2 Background1.2.1 Ultrasound Techniques for Prostate Cancer DiagnosisSince the early 1990s, there have been numerous efforts to improve ultrasound-based tissue typing in the TRUS-guided biopsy. When tissue undergoesultrasound imaging, returning echoes contain useful information for tissuetyping. This information can be applied to discriminate among differenttissues or to determine different structures of the same tissue due to dis-eases such as cancer. For prostate tissue typing, methods that not onlydistinguish prostate cancer but also provide information on its grade, havethe potential to improve the management of cancer and its treatment whilepreventing over-diagnosis. In this section, we review ultrasound-based tissuetyping approaches and their application in prostate cancer diagnosis. Majorultrasound-based tissue typing methods for prostate cancer characterizationinclude texture-based and spectral analysis of ultrasound data, elastography,Doppler, and ultrasound time series analysis.1.2.1.1 Texture-based and Spectral AnalysisThe intensity information in Brightness Mode (B-mode) ultrasound imagescan be used to differentiate among various tissue types [112, 126]. Fortexture-based tissue characterization, the image is divided into windowscalled a Region of Interest (ROI). ROI sizes between 0.1 cmà cm and 1.45cmà cm have been reported in the literature [112]. Texture-based methodsanalyze the first and second order statistics of the gray levels of the B-modeimages which form a set of features for texture characterization. The first-order statistics include the mean, standard deviation, skewness and kurtosisof the gray level in each ROI. Moreover, the speckle signal-to-noise ratio,maximum and minimum, and the Full Width at Half Maximum (FWHM)of ROIs have been found useful for prostate tissue typing [140]. Thesefeatures are sensitive to imaging parameters of ultrasound scanners anddissimilar acoustical properties for tissues with the same pathology havebeen observed [112].Probability distribution models (e.g., Rayleigh, Rice, and Nakagami)fitted to the estimated histogram of B-mode image intensities have also beenshown to provide useful clinical information for tissue characterization [30,152]. Tsuni et al. [151] showed that the B-mode image would be affected bythe system settings and user operations. They suggested that the Nakagamiparametric image provides a comparatively consistent image result whendifferent diagnosticians use different dynamic ranges and system gains. Their31.2. Backgroundresult indicated a better performance of this distribution model compared toother models for prostate cancer diagnostics. Although statistical texture-based features extracted from ultrasound B-mode image are importantfor tissue typing, they have not been used alone [126]. The combinationof texture-based features with features extracted from other ultrasoundtechniques [106] and other modalities [7] have been shown to be effective.Second-order statistical features have been introduced to overcome theselimitations. These features are related to the spatial properties of the imageand are extracted from the co-occurrence and auto-correlation matricesof B-mode image [17, 129, 140, 141]. Although statistical texture-basedfeatures extracted from B-mode and envelope detected signals are necessaryfor tissue typing, they are usually combined with features extracted fromother methods for tissue characterization [112]. Combining texture-basedand clinical features such as location and shape of the hypo-echoic regioncan be a promising way to detect prostate cancer. It has been shown thatcancerous tissues can be detected with high specificity (about 90-95%) andhigh sensitivity (about 92-96%) applying the combination of the features [59].The main shortcomings of texture-based methods are their high dependencyon imaging settings of the ultrasound scanner, signal attenuation, dropout,and shadowing.Some of the tissue-dependent features can only be extracted from RadioFrequency (RF) echo signals before they go through the nonlinear processof envelope detection for B-mode image generation [44, 112, 127]. Duringthe past decades, spectral analysis of RF echo signals constituting a singleultrasound image has been used to improve prostate cancer diagnosis. Oneof the earliest attempts at using spectral analysis of ultrasound has beenby developing an analog filter to break the bandwidth of backscatteredultrasound into three bands [64]. A few years later, Lizzi and Feleppa as thepioneers of spectral analysis of ultrasound signals developed a theoreticalframework for the spectral analysis of ultrasound [97]. In this framework,they proposed a method for calibration of the spectrum to account for system-dependent effects. Lizzi et al. categorized tissue structures theoreticallybased on the spectral properties of ultrasound backscattered signals [96].Spectral features extracted from the power spectrum of the signals have beenused to distinguish between normal and cancerous tissues in the prostate byFeleppa et al. [42, 43]. There are models which represent the relation betweenspectral features and tissue microstructure theoretically [96]. However, inpractice, local spectral noise, and system-dependent effects are challengesfor using these techniques [96, 123]. Recently, histoscanning [100, 101], acommercial software that uses features of a single RF frame, has been applied41.2. Backgroundto characterize prostate cancer. Studies have reported an average sensitivityof 60% and specificity of 66% for 146 patients using this approach [101]. InSection 1.2.2, we will further explore this category of techniques in moredetails from a machine learning perspective.1.2.1.2 ElastographySoft tissues tend to exhibit higher deformation than stiffer areas whencompression is applied. Elastography is an ultrasound imaging approachthat aims to capture tissue stiffness. Since cancerous regions of the tissue areusually stiffer than benign regions, elastography can be helpful for prostatecancer characterization [55, 124]. The quasi-static methods and dynamicmethods are two main ultrasound elastography categories.Quasi-static ultrasound elastography or strain elastography of the prostateis based on the analysis of the deformation generated by a static compressionof the tissue using TRUS. Krouskop et al. [88] analyzed the prostate tissuesamples to evaluate the elastic properties of the tissue specimens. Theseproperties were also used by Konig et al. [85] for image-guided biopsy of theprostate in a group of 404 patients. In this study, 84% of positive cancerpatients were correctly identified [85].Improvement in biopsy guidance [25, 79, 85, 130] and prostate canceridentification [25, 34, 38, 52, 139] are reported in the literature. However,some well-designed studies did not confirm such results [130, 152]. The mainlimitations of strain elastography include lack of uniform compression overthe entire gland, dependency on the operator, penetration issues in largeglands, and artifacts due to slippage of the compression plane that can occurin up to 32% of images [25, 34]. A water-filled balloon may be placed betweenthe probe and the rectal wall to improve the homogeneity of the deformationand reduce the artifacts [4].In dynamic methods, a time-varying force is applied to the tissue; it canbe either a short transient mechanical force or an oscillatory force with a fixedfrequency [49, 65]. Several dynamic methods have been proposed includingvibro-elastography [2, 66, 99, 138], Acoustic Radiation Force Impulsion(ARFI) [121], and shear wave elastography [23, 24]. Vibro-elastography isoperator independent and can be used to estimate the stiffness of the tissueusing TRUS [2, 66, 99, 138]. Shear wave elastography is the most recentelastography technique, which is based on the measurement of low-frequencyshear wave velocity propagating through the tissue [32, 49]. While thespecificity of shear wave elastography has been reported to be as high as91%, the reported sensitivity can be relatively low (63%) [78, 120]. Other51.2. Backgroundlimitations of shear wave elastography include slow frame rate, limited sizeof the ROI, delay in image stabilization for each acquisition plane and signalattenuation in enlarged prostates, making the evaluation of the anteriortransitional zone difficult or impossible [32].Most of the current clinical elastography systems are only capable ofproducing an image that visualizes a single tissue physical parameter, suchas stiffness or viscosity, while cancerous tissues are complex and non-uniformand cannot be characterized using only one parameter [7, 55, 106]. Toaddress this limitation, more recently, multi-parametric elastography ultra-sound and its combination with multi-parametric MRI have been considered.Mohareri et al. [106] showed the potential of multi-parametric quantita-tive vibro-elastography in prostate cancer detection for the first time in aclinical study including 10 patients. Ashab et al. [7] also combined multi-parametric MRI with multi-parametric ultrasound, including B-mode andvibro-elastography images. In a study including 36 whole mount histol-ogy slides, they examined the potential improvement in cancer detection.The idea of capturing tissue stiffness has been extended to MRI as well,and has been explored as Magnetic Resonance Elastography (MRE) tech-nique [136, 137] for detection of tissue abnormalities. In MRE, an externalmechanical excitation is applied to the tissue of interest to induce tissuevibrations and it has been shown to be of value in MRI tissue characterization.Despite major advancement in the elastography technology, all theseapproaches are subject to the same intrinsic limitations: ânot all cancers arestiff, and all stiff lesions are not cancerousâ [32, 137].1.2.1.3 Doppler ImagingDoppler imaging is an alternative ultrasound-based technique for detectionof pathologic conditions. Doppler-based cancer detection methods takeadvantage of the neovascularization phenomenon in cancerous tissues; changesin cellular metabolism associated with the development of cancer leads to anincrease in blood supply to cancer lesions and therefore to neovascularizationin the malignant area [57]. A fundamental problem in Doppler-based cancerdetection is that blood is a much weaker scatterer of the ultrasound thanthe surrounding tissue. Therefore, a considerable frequency shift is requiredto separate a Doppler signal from the background signal. This challengenormally limits Doppler studies to larger vessels with high blood velocity [112].Moreover, neovascularization related to prostate cancer is usually at themicrovascular level which restricts the applicability of Doppler analysis inprostate cancer detection [114]. Today, color flow images, power Doppler61.2. Backgroundimaging, and contrast-enhanced Doppler are all used for detecting prostatecancer and assisting biopsy procedure.Initial reports on the application of Doppler imaging for prostate studiesdate back to 1990 when the conventional color flow imaging was utilized fortissue characterization [159]. In a study including 39 subjects, Potdevin etal. [133] used mean of speed in colored pixels and speed-weighted pixeldensity to locally discriminate prostate cancer. A report from Arger et al. [6]showed that these two features were not substantially different betweendiverse pathologic tissues. In another study, Tang et al. [148] selected 54patients with distinct cancer lesions on ultrasound images. They calculatedthe density of color pixels from Doppler images and used a t-test to analyzethe relationship between the density ratio and malignancy. The obtainedsensitivity of 91% in this study should be interpreted along with the largesize of ROIs. In general, according to different studies, color Doppler imagingis useful for identifying prostate cancer, but targeted biopsy based on colorDoppler imaging alone can miss many areas of cancer [120]. In a study of120 patients, a biopsy regimen consisting of both sextant and color Doppler-directed biopsies was more sensitive than sextant biopsies alone, but theimprovement in cancer detection was minimal [86]. Nelson et al. [120]suggested that the area of hyper-vascularity must be large enough to standout on the Doppler display. Thus, color Doppler may be more sensitivefor detection of clinically significant, high-grade lesions. While contrastenhancing agents can increase the intensity of Doppler signal from the micro-vessels [25, 160], legal and technical difficulties have limited the applicationof such agents.1.2.1.4 Ultrasound Time Series AnalysisTemporal Enhanced Ultrasound (TeUS) data is defined as the time seriesof RF echo signals obtained from a stationary position in a tissue withoutintentional motion of the transducer or the tissue [112] (Fig. 1.1). Since2007, temporal enhanced ultrasound data has been explored using a machinelearning framework to analyze subtle relative variations between various tissuetypes. It has been shown that features extracted from these variations, highlycorrelate with the underlying tissue structure [18]. A fundamental departurefrom prior methods is to display a likelihood map of the presence of cancer,based on machine learning, instead of using pre-defined thresholds for cancerdetection. Machine learning eliminates the need for accurate thresholding offeatures to identify cancer. Moradi et al. utilized spectral features extractedfrom ultrasound RF time series signals to distinguish between different animal71.2. BackgroundFigure 1.1: A schematic diagram of Temporal Enhanced Ultrasound (TeUS)data generation. A time series of a fixed point in an image frame, shownas a red dot, is analyzed over a sequence of ultrasound frames to determinetissue characteristics.tissue types [109]. The results of this study demonstrated that temporalultrasound data are sensitive and specific for tissue typing. Moreover, in an exvivo study, spectral time series features were applied to differentiate betweenhealthy and cancer tissues in 35 human prostate specimens [108]. The resultsfrom this study showed that the features extracted from temporal enhancedultrasound data are significantly more accurate and sensitive compared tothe best texture-based and spectral features for detecting prostate cancer.In an in vivo experiment with six radical prostatectomy patients [73], and ina retrospective study with 158 patients undergoing fusion biopsy [11, 13, 82],accurate prostate cancer detection across grades and patients has beendemonstrated. Evidence derived from the experiments to-date suggeststhat both tissue-related and ultrasound signal-related factors lead to tissuetyping information in temporal enhanced ultrasound data. These includethe cellular structure of the tissue [109] and the thermal properties of thetissue at micro-scale [36]. For the work presented in this dissertation, TeUSis the imaging modality used for data collection in prostate cancer patients.81.2. Background1.2.2 Machine Learning ApproachesFrom the computer-aided diagnosis point of view, prostate cancer detectionusing ultrasound imaging, can be automated as a classification or a clusteringtask. Over the past decades, machine learning methods are employed toautomate the process of tissue typing using handcrafted features, and morerecently, to automatically derive features that optimally represent the databased on tissue types. In this section, I focus the discussion on these twomajor approaches for deploying machine learning techniques in prostatecancer diagnosis.1.2.2.1 Feature Generation and ClassificationPreviously, manually engineered feature representations extracted fromultrasound data [72, 73, 76] have been used with shallow discriminantmodels such as linear regression models [42â44], k-Nearest Neighbours(k-NN) classifier [98], support vector machines (SVM) [45, 108], randomforests [51, 154], Bayesian classifiers [113] and multi-layer feed-forward per-ceptron networks [44, 109], to differentiate tissue types [112].Feleppa et al. [42, 43, 45] extracted the spectral-features for tissue typingin prostate cancer from a line fitted to the power spectrum of a singleframe of RF echo signal obtained by sonicating the tissue of interest. Inthis approach, after using Fourier transform to map the RF signals intothe spectral domain, they used a linear regression model to explain therelationship between frequency and amplitude in the power spectrum. Theextracted features from the regression model, such as intercept and slopewere used for differentiation between tissue types. Llobet et al. proposedthe adoption of a k-NN classifier to detect cancerous regions in transrectalultrasound B-mode images of the prostate [98] where they utilized texture-based feature extracted from the ultrasound images to train their classifier.Mohareri et al. [106] proposed a novel set of features that obtained fromvibro-elastography to classify cancerous and normal tissue and compute a can-cer probability map. These features include brightness, tissue strain, absolutevalue of tissue elasticity, relaxation time, and visco-elastic properties. Theyused a random forest classification framework to combine multi-parametricfeatures and perform tissue characterization. In a leave-one-patient-outstudy including 10 patients, they achieved area under the receiver operatingcharacteristic curve (AUC) of 0.8±0.01. Ashab et al. [7] proposed prostatetissue classification based on the combination of multi-parametric MRI andmulti-parametric ultrasound which includes texture-based features from B-91.2. Backgroundmode images and vibro-elastography features. They used LASSO regressionand Recursive Feature Elimination (RFE) for feature reduction and selection.Then, a weighted SVM was used for classification. In a study including 36whole mount histology slides, they achieved an AUC of 0.81 for cancerousROIs extracted from the peripheral zone of the prostates [7].More recently, Imani et al. [73, 75] combined extracted features fromdiscrete Fourier and Wavelet transforms of TeUS data and fractal dimensionto characterize in vivo prostate tissue using SVMs. Imani et al. [75] used ajoint Independent Component Analysis (ICA) algorithm to select the mostinformative and independent combination of features extracted from in vivoTeUS signal. Then, they trained an SVM classifier to differentiate betweennormal and cancer tissue. Ghavidel et al. [51] used spectral features of TeUSdata and performed feature selection using random forests to classify lowergrade prostate cancer from higher grades. Moradi et al. [109, 113] usedfractal dimension with Bayesian classifiers and shallow feed-forward neuralnetworks for ex vivo characterization of prostate tissue. Feleppa et al. [44]also used artificial neural networks to assess the performance of spectralfeatures from RF echo signals collected during TRUS imaging of the prostate.1.2.2.2 Hidden Markov ModelsHidden Markov models (HMMs) represent sequential data, especially timeseries, as stochastic processes through capturing repetitive patterns andmotifs in the data. HMMs are based on probabilistic dependency and henceare well suited for describing time-varying sequences. Llobet et al. [98]proposed the use of HMMs for tissue characterization in TRUS imagescollected during prostate biopsies. They modeled the echo intensity valuesalong the biopsy-needle path inside of the scanned tissue. The modeledsequences, in this case, are position-dependent rather than time-dependent.Most recently, analysis of TeUS in the temporal domain using probabilisticmethods, such as HMMs, have shown significant promise [115, 118]. Nahlawiet al. [116â118] used HMMs to model the temporal information of TeUSdata for prostate cancer detection. In a limited clinical study including 14subjects, they identified cancerous regions with an accuracy of 85%. Theirresults also indicate the importance of order in the characterization of tissuemalignancy from TeUS data [115].101.3. Proposed Solution1.2.2.3 Deep Learning ApproachesDeep learning algorithms are probabilistic models (networks) composed ofmany layers that transform input data (e.g., images) to outputs (e.g., diseasepresent/absent) while learning increasingly higher level features [61, 95]. Inthe computer-aided diagnosis literature, Deep Neural Networks (DNN) [63,95] have been recently established as a powerful machine learning paradigm.The main advantage of DNNs is their ability to discover informative represen-tation of data which contributes to feature engineering originally performedsolely by human experts. Our group previously examined ConvolutionalNeural Networks (CNNs) to combine temporal and spatial information fromTeUS data to detect high-risk prostate cancer. Imani et al. [74] used amachine learning framework consisting of the weighted combination of twoCNNs. One network takes the information in the temporal sequence as inputwhile the other network uses the spatial arrangement of the ultrasound data.Then, they use the fusion of mp-MRI and TeUS for characterization of highergrade prostate cancer. In a leave-one-patient-out evaluation including 14patients, they achieved the Area Under the ROC Curve (AUC) of 0.86 forcancerous regions with GSâ¥3+3, and AUC of 0.89 for those with GSâ¥3+4.Besides being computationally expensive, large-scale data is required foreffective deep learning and to ensure proper generalization. Further, thedifficulty in explaining the operation of deep neural networks has led toimperceptive and repetitive, yet successful use of deep learning methods as ablack-box in many areas. The focus of this thesis is on probabilistic modelingof spectral and temporal aspect of TeUS data using Deep Belief Networks(DBN) [62] and Recurrent Neural Networks (RNNs) [50], respectively. Inaddition, methodologies are devised to interpret the internal representationslearned by deep models to derive insights about tissue dependent featuresresiding in TeUS.1.3 Proposed SolutionDespite many years of research to improve prostate cancer diagnosis, conven-tional ultrasound-guided biopsy has low sensitivity, leading to significant ratesof under-treatment and over-treatment. Augmentation the biopsy processwith ultrasound-based tissue typing techniques that analyze the spectrumand texture of TRUS, or measure mechanical properties of tissue in responseto external excitation is useful. However, individual methods have not shownsuccess in multi-center clinical studies, as it is proven difficult to establish acancer-specific tissue property consistent across the population [102].111.3. Proposed SolutionT2 KtransMultiparametric MRIPCa SuspectTransrectal USTemporal US dataMR-TRUS fusionSystematic BiopsyFusion BiopsyProposed Decision Support ModelUS Targeted BiopsyBiopsy ProcedurePre-ProcedureIntra-ProcedureFigure 1.2: A schematic diagram of TeUS-based workflow for prostate biopsy.The focus of this thesis is to advance prostate cancer diagnosis throughthe development of a deep learning solution for real-time analysis of tem-poral enhanced ultrasound image. The proposed solution will be trainedto automatically analyze backscattered ultrasound signals from the tissueover a multitude of TRUS frames and extract tissue-dependent features todiscriminate between different cancer grades. The proposed solution can bedeployed as a decision support model for patient-specific targeting duringbiopsy; it can display cancer likelihood maps on B-mode ultrasound images,in real-time, to indicate the presence of cancer. Moreover, its fusion withMRI, where available, can be used to improve biopsy targeting (Figure 1.2).A fundamental assumption in previous models for TeUS is the availabilityof RF data in imaging equipment; however, RF is not available on allcommercial US scanners and is usually only provided for research purposes.To overcome the challenge of accessibility of RF data, in the TeUS machinelearning framework, we propose to use a transfer learning method [131]to transfer knowledge between RF and B-mode data within an ultrasoundscanner. Transfer learning is a method that can compensate for the diversitybetween datasets by learning from a source dataset (RF time series data)and applying the knowledge to a target dataset (B-mode time series data).This approach exploits the common information between the two domains.Common elements between two data domains can be features of the data orparameters of the models that are built using the data [131]. Deep learningmethods, such as DBN, have an essential characteristic that makes themwell suited for transfer learning: they can identify abstract features thatperform well generalizing across various domains [21, 53]. The model isgenerated from RF or B-mode TeUS before the biopsy procedure and we can121.3. Proposed Solutionuse the generated model to extract real-time tissue-dependent informationfrom the temporal enhanced ultrasound signals. The proposed solutionleverages the standard TRUS imaging pipeline (Figure 1.2), hence does notrequire modifications to the established clinical workflow. We expect thatthis technology provides a versatile cross-institutional solution for accuratedetection of prostate cancer and patient-specific biopsy targeting.1.3.1 ObjectivesAs explained before, the extensive heterogeneity in morphology and pathologyof prostate adenocarcinoma are challenging factors in obtaining an accuratediagnosis using different imaging modalities. Providing tissue-specific in-formation during TRUS-guided biopsies can assist with targeting regionswith high probability of being cancerous. The analysis of TeUS reveals adifference in response between benign and malignant tissues as reported byseveral previous studies. The objectives of this thesis are:â¢ Objective 1. Detection of prostate cancer using TeUS data.The first step towards improvement of TRUS-based targeted biopsy is todevelop an accurate ultrasound-based tissue typing method. Deep learn-ing techniques are proposed to automatically extract tissue-dependentfeatures from in vivo temporal enhanced ultrasound data. These ab-stract high-level features can be used to distinguish between differentprostate tissue types.â¢ Objective 2. Detection of high-grade prostate cancer usingTeUS data. The second step towards the improvement of the TRUS-based targeted biopsy is to develop a tissue typing method that candifferentiate among grades of the disease and its extent. The ubiquityof noise is an important issue for building computer-aided diagnosismodels for prostate cancer biopsy guidance where histopathology datais sparse and not finely annotated. A solution is proposed to alleviatethis challenge as a part of the TeUS-based prostate cancer biopsyguidance method.â¢ Objective 3. Development of a real-time TeUS-based decisionmodel, and enabling its clinical deployment. Following success-ful diagnosis and grading of prostate cancer using deep models, thethird objective is to enable its clinical deployment as a tool for im-proving detection of clinically significant prostate cancer in fusionbiopsies. Towards this goal, the ability to remove TeUS spectrum131.3. Proposed Solutionanalysis as a preprocessing step is studied by analyzing data directly inthe time domain using Recurrent Neural Networks (RNNs). This canhelp reduce computation time and accelerate cancer likelihood mapgeneration. Due to limited access to raw ultrasound data, transferlearning is devised to generate comparable performance using B-modeimages. Real-time implementation of TeUS-based decision model andits evaluation through retrospective data is another step in realizingthis objective. Finally, understanding the physical interaction of TeUSdata with tissue is not only of theoretical benefit, but can help withpractical decisions in building a decision model.1.3.2 ContributionsThis thesis develops, deploys and evaluates a deep learning framework thatuses temporal enhanced ultrasound data to output accurate prostate cancerlikelihood maps, overlaid on real-time TRUS images. The proposed solutionencapsulates the variability associated with the access of raw ultrasoundsignals in commercial scanners, and can provide complementary informationabout the grade and extent of prostate cancer. In the course of achievingthe objectives, the following contributions were made:1. Probabilistic modeling of spectral information in temporal enhancedultrasound:â An automatic feature selection framework is proposed based onDBN for spectral analysis of temporal enhanced ultrasound signalsfrom prostate tissue.â A statistically significant improvement in the accuracy of prostatecancer detection is demonstrated compared to previously publishedstudies using spectral features of TeUS signals [74, 76].â To determine the characteristics of non-cancerous and cancerouscores in TeUS data and their correlation with learned features,a feature visualization approach is developed. This approach isused to identify the most discriminating frequency components ofthe time series as learned by the classifier.2. Probabilistic modeling of temporal aspects of TeUS:â TeUS data is extensively and explicitly analyzed in temporaldomain using time-dependent probabilistic deep networks for thefirst time.141.3. Proposed Solutionâ Results indicated that RNN can identify temporal patterns in datathat may not be readily observable in spectral domain, leading tosignificant improvements in detection of prostate cancer.â Algorithms are developed for in-depth analysis and visualizationof high-level latent features of LSTM-based RNN.â A transformational finding, achieved through this analysis, isthat the most discriminating features for detection of prostatecancer can be learned from a fraction of the full TeUS time series.Specifically, in this data, less than 50 ultrasound frames wererequired to build models that accurately detect prostate cancer.This information can be used to optimize TeUS data acquisitionfor clinical translation.3. Classifying prostate cancer grade using probabilistic modeling of spec-tral aspects of TeUS data:â A novel approach for grading and detection of high-grade prostatecancer using spectral analysis of TeUS with DBN is proposed.â This approach could successfully differentiate among aggressiveprostate cancer (GSâ¥4+3), clinically less significant prostatecancer (GSâ¤3+4), and non-cancerous prostate tissues.4. Classifying prostate cancer grade and its extent based on temporalmodeling of TeUS data:â A novel approach is devised for grading and detection of high-grade prostate cancer using temporal analysis of TeUS with RNN.By encapsulating proposed ground-truth probability vectors, thissolution can also precisely estimate cancer length in biopsy cores.â The accuracy of tissue characterization is statistically significantlyimproved as compared to previously published studies [10, 12, 76].â A novel strategy is proposed for depiction of patient-specific col-ormaps for biopsy guidance including the estimated model uncer-tainty.5. Development of real-time TeUS-based decision model:â A transfer-learning approach is developed to address limited accessto raw ultrasound data on commercial scanners, and scannerdifferences in multi-center settings.151.3. Proposed Solutionâ Near real-time augmentation of live standard 2D ultrasound im-ages with cancer likelihood maps generated from the models isimplemented.â The viability of using B-mode TeUS for cancer detection is demon-strated using retrospective data. The initial assessment indicatesthat the solution is capable of providing guidance information forprostate biopsy procedures.6. Investigation of the physical phenomena underlying TeUS:â A method for visualization and interpretation of the learnedfeatures from TeUS data is presented.â Evidence derived from feature visualization points to low-frequencycomponents of TeUS as the most informative features for tissueclassification. These components potentially represent the effectof pulsation on prostate tissue microstructure in form of micro-vibrations.â The effect of micro-vibration is simulated using a medium withpreset elasticity and scatterer locations extracted from 14 whole-mount digital histopathology slides.â Results demonstrated that the distribution and micro-vibration ofscatterers could lead to tissue typing information in TeUS. Thisfinding is a major breakthrough in understanding and technicalformulation of TeUS after a decade.One million patients in North America undergo prostate biopsy annually;70% of 10 core biopsies return negative while up to 34% of the positive yieldare under-graded. The mp-MRI, as the state-of-the-art imaging techniquefor prostate cancer detection, has low accuracy in identifying cancer withintermediate-risk or low volume [89], and has a high rate of false positives,leading to a significant number of negative cores in current fusion biopsies [132,144]. Cancer likelihood maps displayed by the proposed solution can helpsystematic biopsy by improving the cancer yield and decreasing the number ofunnecessary biopsies. They also benefit fusion biopsy as they can compensatefor inaccurate co-alignment of mp-MRI, improving the accuracy of mp-MRIespecially for intermediate risk cancer. Innovative and transparent deeplearning solutions will allow the technology to work across institutions withoutbias towards the image settings, equipment or patient pool in one setting.In the long term, this technology can benefit patients by enabling periodicexamination of those under active surveillance by this technology alone.161.3. Proposed SolutionAll men who undergo prostate biopsy can take advantage of the proposedtechnology by improving early detection of prostate cancer and decreasing thenumber of unnecessary biopsies. Further, the proposed solution can benefitpatients by enabling periodic, affordable, widely available and minimally-invasive examination of those under active surveillance, where it is criticalto detect clinically significant prostate cancer early with a minimal numberof biopsies.1.3.3 Thesis OutlineThe rest of this thesis is subdivided into six chapters as outlined below:Chapter 2: Temporal Enhanced Ultrasound DataThis chapter describes the TeUS data acquired and analyzed in this thesis. Itexplains the clinical process of data acquisition and establishing ground truthprior to data analysis. Time-domain and spectral-domain representations ofTeUS signals are also described.Chapter 3: Diagnosis of Prostate Cancer Using TeUSFor accurate diagnosis of prostate cancer, in this chapter, both the spectraland temporal aspect of TeUS data are analyzed. The focus of the chapteris to distinguish two main categories of the tissues (i.e., cancer and benign)which can be modeled as a binary classification task:â Spectral analysis of TeUS using DBN: In this section, a deep learningapproach is proposed to automatically analyze the spectral aspectof temporal ultrasound data obtained from 255 cancer foci identifiedin mp-MRI. Each target is sampled in axial and sagittal planes. ADBN is trained to automatically learn the high-level latent featuresof temporal ultrasound data. A support vector machine classifieris then applied to differentiate cancer vs. benign tissue, verified byhistopathology. Results indicate that the distance between the biopsytarget and the prostate boundary, and the agreement between axialand sagittal histopathology of each target impact the classificationaccuracy. Using temporal ultrasound data in a fusion prostate biopsystudy, a high classification accuracy specifically for moderately scoredmp-MRI targets is achieved. These targets are clinically common andcontribute to the high false positive rates associated with mp-MRI forprostate cancer detection. Results suggest that temporal enhancedultrasound data combined with mp-MRI has the potential to reducethe number of unnecessary biopsies in fusion biopsy settings.171.3. Proposed Solutionâ Temporal analysis of TeUS using RNN: In this section, a deep RNNis proposed to explicitly model the temporal information in TeUS.By investigating several RNN models, it is demonstrated that LongShort-Term Memory (LSTM) networks achieve the highest accuracy indetecting prostate cancer in TeUS data. Algorithms are presented forin-depth analysis of LSTM networks. Results suggest that temporalmodeling of TeUS using RNN can significantly improve cancer detectionaccuracy over previously presented spectral analysis of TeUS.Chapter 4: Detection of High-Grade Prostate Cancer Using TeUSIn this chapter, the focus is on accurate detection of higher grade prostatecancer and the problem of detection of different prostate Gleason grades. Thistask can be modeled as a probabilistic multi-class classification task. However,there are two key challenges with the ground-truth. First, histopathologydata used for training of the models is sparsely annotated with the inevitableubiquity of noise. Second, the heterogeneity in morphology and pathology ofthe prostate itself contributes as a source of inaccuracy in labeling. Thesechallenges are addressed through:â Spectral analysis of TeUS for prostate cancer grading: Histopathologicalgrading of prostate cancer reports the statistics of cancer distributionin a biopsy core. In this section, a coarse-to-fine classification approach,similar to histopathology reporting, is proposed that uses statisticalanalysis and deep learning to determine the distribution of aggressivecancer in ultrasound image regions surrounding a biopsy target. Thisapproach consists of two steps; in the first step, high-level latent featuresthat maximally differentiate benign from cancerous tissue are learned.In the second step, the statistical distribution of prostate cancer gradesin the space of latent features is modeled.â Temporal analysis of TeUS for prostate cancer grading: In this section,a solution is proposed to alleviate the challenge of noisy and sparselabels in building computer-aided diagnosis models for prostate cancerbiopsy guidance. Specifically, prior knowledge from the histopathologyis embedded as soft labels in a probabilistic model. These soft labels areused as a replacement for the sparse and noisy labels for the training ofan RNN-based model. This information is then applied to detect thegrade of cancer and also to estimate the length of cancer in the target.Chapter 5: Decision Support System for Prostate Biopsy Guid-ance181.3. Proposed SolutionIn the search for an accurate and practical technique for prostate cancerdiagnosis with TeUS in the clinical setting, in this chapter the focus is on twotechnical challenges: first, the accessibility of raw RF data in commercialscanners and second, real-time processing of RF/B-mode TeUS data usingtemporal analysis of TeUS with RNN.â Transfer learning from RF to B-mode TeUS spectral features: Ina preliminary attempt, a method is presented for prostate cancerdetection using TeUS data obtained either from RF ultrasound signalsor B-mode images. For the first time, it is demonstrated that byapplying domain adaptation and transfer learning methods, a tissueclassification model trained on TeUS RF data (source domain) canbe deployed for classification using TeUS B-mode data alone (targetdomain) where both data are obtained on the same ultrasound scanner.This is a critical step for clinical translation of tissue classificationtechniques that primarily rely on accessing RF data since this imagingmodality is not readily available on all commercial scanners in clinics.â Transfer Learning from TeUS RF to B-mode Using RNN: In this section,as a part of an unified software framework for real-time analysis ofultrasound data stream using a deep learning solution, an RNN-basedtransfer learning solution is proposed. It is demonstrated that prostatecancer detection using near real-time analysis of RF and B-mode TeUSdata and deep learning is feasible. The proposed software system allowsfor depiction of live 2D ultrasound images augmented with patient-specific cancer likelihood maps that have been calculated from TeUS.The performance of the proposed system with data obtained in twoindependent retrospective clinical studies is evaluated.Chapter 6: Investigation of Physical Phenomena Underlying TeUSIn Chapter 4, TeUS is used to address the problem of grading prostate cancer.The method involves capturing high-level latent features of TeUS with adeep learning approach followed by distribution learning to cluster aggressivecancer in a biopsy core. In this hypothesis-generating study, a deep learningbased feature visualization method is utilized as a means to obtain insightinto the physical phenomenon governing the interaction of temporal ultra-sound with tissue. Evidence derived from the deep learning-based featurevisualization pointed to low-frequency components of TeUS as the mostinformative features for tissue classification. These components potentiallyrepresent the effect of pulsation on prostate tissue microstructure. As aresult, in chapter, mechanical micro-vibrations of scatterers in phantoms191.3. Proposed Solutionwith various scatterer distributions, reflecting benign and cancerous tissue,derived from digital histopathology data are simulated. It is demonstratedthat micro-vibrations of scatterers can be captured by low-frequency spectralfeatures of TeUS, similar to in vivo results. These observations togetherwith previous results suggest that the distribution and micro-vibration ofscatterers can lead to tissue typing information in TeUS.Chapter 7: Conclusion and Future WorkThis chapter includes a short summary of the thesis followed by a discussionon the limitations of the proposed methods and suggestions for future work.20Chapter 2Temporal EnhancedUltrasound DataData will talk to you if you are willing to listen.â Jim BergesonTemporal Enhanced Ultrasound or TeUS is defined as the time series ofultrasound RF or B-mode frames captured from insonification of tissue overtime, without intentionally moving the tissue or the ultrasound probe. As it isshown in Fig. 1.1, the tissue response to this prolonged insonification consistsof reflected ultrasound echo-intensity values. This chapter presents the TeUSdata used for building and evaluation of our tissue typing framework, whichis described in Chapter 3 and Chapter 4 for detection and grading of prostatecancer. Then, we deploy the same data to develop the TeUS-based decisionsystem in Chapter 5. In Section 2.1, we describe the data acquisition processfollowed by a summary of the method used to establish a ground truth forcharacterizing prostate cancer tissue in Section 2.2. Section 2.3 explainsthe process of Region of Interest (ROI) selection and data augmentation.Finally, we describe the temporal and spectral representation of TeUS datain Section 2.3.1 and Section 2.3.2, respectively. Aslo, for further evaluation ofour method, we performed a second case study that we explain in Section 2.4.This chapter is partly adapted from the following papers [11, 13]: 1) Shekoofeh Azizi,Farhad Imani, Bo Zhuang, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Nishant Uniyal,Baris Turkbey, Peter Choyke, Peter Pinto, Bradford Wood, Parvin Mousavi, and PurangAbolmaesumi. Ultrasound-based detection of prostate cancer using automatic featureselection with deep belief networks. In Medical Image Computing and Computer AssistedIntervention (MICCAI), pages 70â77. Springer, 2015, and 2) Shekoofeh Azizi, Farhad Imani,Sahar Ghavidel, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke,Peter Pinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Detection ofprostate cancer using temporal sequences of ultrasound data: a large clinical feasibilitystudy. International Journal of Computer Assisted Radiology and Surgery, 11(6):947â956,2016212.1. Data Acquisition2.1 Data AcquisitionData were obtained with a Philips iU22 ultrasound scanner in fusion prostatebiopsy procedures where the biopsy target locations were identified usingmp-MRI information, and the biopsy was guided by TRUS. The study wasapproved by the ethics review board of the National Cancer Institute (NCI),National Institutes of Health (NIH) in Bethesda, Maryland. One hundred andthirty-two (132) enrolled subjects provided informed consent to participate.Every subject underwent preoperative mp-MRI examination with threepulse sequences: T2-weighted, Diffusion Weighted Imaging (DWI), andDynamic Contrast Enhanced (DCE) imaging. Before the biopsy procedure,suspicious lesions were identified using mp-MRI (Fig. 2.1). Two independenthighly experienced genitourinary radiologists (B.T. and P.L.C.) with 8 and 14years of experience, interpreted and scored suspicious lesions according to apreviously published protocol [153]. Each radiologist assigned an overall scorein the range from 1 (no cancer) to 5 (aggressive cancer) to a suspicious area.The consensus scores were grouped into three descriptors of âlowâ (scoreof â¤ 2), âmoderateâ (score of 3) and âhighâ (score of â¥ 4), and referred toas the MR suspicious level assigned to the area. These scores are based onfindings on each mp-MRI sequence using previously described criteria [144],which indicate both the presence of prostate cancer and tumor grade. Thestandardized PI-RADS criteria were not in use for this study.At the beginning of the procedure, a 3D ultrasound volume of the prostatewas reconstructed by obtaining a series of ElectroMagnetically (EM) tracked2D TRUS images. The identified mp-MRI lesions were delineated on the T2-weighted MR image as the biopsy targets. Then, using UroNav MR-US fusionsystem (Invivo Inc., a subsidiary of Philips Healthcare), T2-weighted MRimages were registered with the 3D TRUS volume of the prostate [103, 162].Following the registration of TRUS and MR volumes, the target locationsfor biopsy were transformed into the EM coordinate frame. A clinician thennavigated in the prostate volume towards the MR-identified target. TRUStransducer was held steady for about 5 seconds to acquire 100 frames ofTRUS B-mode and RF time series data from the target, followed by firingthe biopsy gun to acquire a tissue sample. The Endocavity curved probe(Philips C9-5ec) with the frequency of 6.6 MHz was used for data acquisition.Two cores from axial and sagittal imaging planes were obtained per targetlocation, respectively. Hereafter, we refer to these cores as âaxial sampleâand âsagittal sampleâ, respectively. TeUS data was only recorded from theprimary lesion in the axial imaging plane to minimize disruption to theclinical workflow. Table 2.1 shows the details of equipment and imaging222.2. Histopathology LabelingFigure 2.1: UroNav MR/US fusion system: The identified mp-MRI lesionswere delineated on the T2-weighted MR image as the biopsy targets. Thetarget location is shown by the green point along the projected needle pathin the ultrasound image.parameters used for TeUS data collection.2.2 Histopathology LabelingHistopathology reports include the Gleason Score (GS) and the percentagedistribution of prostate cancer in the axial and sagittal samples from eachtarget. The GS is the most common system to describe the level of abnormal-ity in the prostate tissue. Gleason grades range from 1 (resembling normaltissue) to 5 (aggressive cancerous tissue). Gleason score is reported as a sumTable 2.1: Details of equipment and imaging parameters used for TeUS datacollection.Ultrasound Machine Philips iU22Ultrasound Probe Philips C9-5ecProbe Array Curved LinearCentral Frequency 6.6 MHzMRI Scanner 3.0-Tesla MRI scanner (Philips Achieva)MRI coil Endorectal coil (BPX-30)232.2. Histopathology Labeling67%33%BenignCancerous(a)67%20%12%BenignGS of 3+4 or lowerGS of 4+3 or higher(b)67%10%9%3%7%4%BenignGS 3+3GS 3+4GS 4+3GS 4+4GS 4+5(c)6%71%10%13%LowModerateHighUnknown(d)Figure 2.2: Statistics of histopathology and MR readings in our TeUS dataset:Histopathology reports include the Gleason Score (GS) and the percentagedistribution of prostate cancer. The MR scores were grouped into threedescriptors of âlowâ, âmoderateâ and âhighâ, and referred to as the MRsuspicious level assigned to the area.of the grades of the two most common patterns in a tissue specimen [39]. Inour dataset, 83 biopsy cores are cancerous with GS 3+3 or higher, where31 of those are labeled as clinically significant cancer with GS â¥ 4+3. Theremaining 172 cores are non-cancerous and include benign or fibromusculartissue, chronic inflammation, atrophy and Prostatic Intraepithelial Neoplasia(PIN) [12]. Figure 2.2 and Table 2.2 shows the distribution of different Glea-son score and MR reading in the dataset. In our dataset, 70% of the coreshave moderate MR suspicious level where 67% of the cores were detected asnon-cancerous after the biopsy.242.3. Preprocessing and Region of InterestTable 2.2: Gleason score distribution in the first retrospective TeUS dataset.Gleason Score Benign CancerousNumber of Cores 172 83Gleason Score Benign GSâ¤3+4 GSâ¥4+3Number of Cores 172 52 31Gleason Score Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 GS 4+5Number of Cores 172 26 26 5 21 52.3 Preprocessing and Region of InterestDue to scattering phenomena, ultrasound imaging techniques do not provideaccurate information about the location of every individual cells. The echoesreflected from such small objects in soft tissues are scattered in all directionsrather than solely in the direction back to the transducer [60]. Thus, togenerate and to annotate TeUS signal, a group of RF or B-mode values (nota single value), corresponding to areas known as Region of Interest (ROI) isconsidered. For each biopsy target, we analyze an area of 2Ã 10 mmÃmmaround the target location in the lateral and axial directions, respectively.This area is along the projected needle path in the ultrasound image andcentered on the target location. To register the captured target location onB-mode image during biopsy procedure to the corresponding RF image, weuse the distance map generated using the scan conversion parameter at NIH.Figure 2.3 shows an example of such a distance map, where the dark blueis showing the target location and the color spectrum from blue to yellowis showing farther distance from the target. We divide this region to 80equally-sized ROIs of size 0.5Ã 0.5 mmÃmm (see Fig. 2.4). Considering1540 m/sec as the propagation speed of ultrasound signal in soft tissue and3 cycles per pulse, the axial resolution of the system is around 0.2 mm. EachROI includes 27-55 RF samples based on the depth of imaging and the targetlocation. We also augment the data by creating ROIs using a sliding windowof size 0.5 Ã 0.5 mm Ãmm over the target region, which results in 1,536ROIs per target (see Fig. 2.4). We use histopathology outcome to assign abinary y â {0, 1} or non-binary label y to each ROI. We will further discussthe details of ground-truth label assignment and the corresponding obstaclesin the coming chapters.252.3. Preprocessing and Region of Interest(a) (b)(c) (d)Figure 2.3: Example of distance maps and their corresponding B-mode andRF frames: (a) RF distance map, (b) RF frame, (c) B-mode distance map,(d) B-mode frame. The dark blue is showing the target location and the colorspectrum from blue to yellow is showing farther distance from the target.10 mmAxialTimeTime0.5 mm0.5 mm0.5 mmSliding2 mm10 mm0.1mmFigure 2.4: Preprocessing and ROI selection: the target region is divided to80 ROIs of size 0.5 mmà 0.5 mm and then a sliding window is used for thedata augmentation.262.4. Complementary Second Retrospective TeUS Study2.3.1 Time-domain Representation of TeUSLet CT = {(x(i),y(i))}|CT |i=1 represent the collection of all labeled ROIs sur-rounding a target core, where |CT | = 80, x(i) represents the ith TeUS sequenceof the core, and y(i) indicates the corresponding label. For each ROI, wegenerate a sequence of TeUS data, x(i) = (x(i)1 , ..., x(i)T ), T = 100 by averagingover all time series values within a given ROI of an ultrasound frame andsubtracting the mean value from the given time series. We call x(i) thetime-domain representation of TeUS.2.3.2 Spectral-domain Representation of TeUSLet CS = {(f (i),y(i))}|CS |i=1 represent the collection of all labeled ROIs sur-rounding a target core, where |CS | = 80, f (i) represents the ith TeUS spectralcomponent of the core, and y(i) indicates the corresponding label. To obtainthe spectral-domain representation of TeUS, f (i), we compute the spectrumof TeUS data obtained from each biopsy target and in each region of in-terest. For this purpose, we take the Discrete Fourier Transform (DFT)of all zero-mean ultrasound time series corresponding to the RF/B-modesamples in each ROI, normalized to the frame rate. Then, we calculatethe mean absolute values of the Fourier transforms of the RF/B-mode timeseries in each ROI. Finally, each ROI is represented by 50 positive frequencycomponents, f (i) = (f(i)1 , ..., f(i)F ), F = 50.2.4 Complementary Second Retrospective TeUSStudyTo further assess the developed solutions in this thesis, we performed asecond independent MRI-TRUS fusion biopsy study. This study also followsthe exact same protocol as the previous study, however, the dataset includesonly TeUS B-mode data. The study was approved by the institutional ethicsreview board and all subjects provided informed consent to participate. Sixsubjects were enrolled in the study where they underwent preoperative mp-MRI examination prior to biopsy, to identify the suspicious lesions. Like theprevious study, the procedure was performed using UroNav MR-US fusionsystem and the data were acquired using Philips iU22 ultrasound scanner.From each MRI-identified target locations, two biopsies were taken, one inthe axial imaging plane and one in the sagittal imaging plane. Only TeUSB-mode data were recorded for each target to minimize disruption to the272.4. Complementary Second Retrospective TeUS StudyTable 2.3: Gleason score distribution in the second retrospective TeUS study.Gleason Score Benign CancerousNumber of Cores 7 14Gleason Score Benign GSâ¤3+4 GSâ¥4+3Number of Cores 7 5 9Gleason Score Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 GS 4+5Number of Cores 7 2 3 2 1 6clinical workflow. We use the histopathology labeling of the cores as theground-truth to assess the accuracy of the guidance system in detecting thecancerous lesions. This study resulted in 21 targeted biopsy cores with theGleason score distribution as explained in Table 2.3. These two datasetshave different distribution of the cancer grades. The second study includesmore cases with higher grade and aggressive lesions. In this study 67% of thecores are cancerous and 43% of the cores are labeled as clinically significantcancer with GS â¥ 4+3. In contrast, as Fig. 2.2 shows, for the first studythese ratios are 33% and 12%, respectively.Extraction of the target location, selection of the region of interest,histopathology labeling, and generation of TeUS representation also has beendone with the exact same method that we introduced in the previous section.We use this data, in Chapter 5 and in Section 5.3 to evaluate the proposedRNN-based model for prostate cancer detection using TeUS B-mode data.28Chapter 3Detection of Prostate CancerUsing TeUSThere are things I canât force. I must adjust. There are timeswhen the greatest change needed is a change of my viewpoint.â Denis Diderot3.1 IntroductionThe early diagnosis of prostate cancer plays an important role in the choiceand the success of treatments [108]. As discussed in Chapter 1, over thepast decades, several ultrasound-based techniques have been proposed forcharacterizing cancerous tissue [112]. The clinical uptake of these methods hasbeen slow, mainly due to the large variability of the tissue characterizationresults. The primary sources of such variability are the heterogeneouspatient population and cancer grades [10]. Moreover, these methods do notgenerally report clinically sufficient specificity and sensitivity to ensure thatall cancer cases can be captured by ultrasound analysis. In the search for aThis chapter is partly adapted from the following papers [9, 11, 13]: 1) Shekoofeh Azizi,Farhad Imani, Bo Zhuang, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Nishant Uniyal,Baris Turkbey, Peter Choyke, Peter Pinto, Bradford Wood, Parvin Mousavi, and PurangAbolmaesumi. Ultrasound-based detection of prostate cancer using automatic featureselection with deep belief networks. In Medical Image Computing and Computer AssistedIntervention (MICCAI), pages 70â77. Springer, 2015, 2) Shekoofeh Azizi, Farhad Imani,Sahar Ghavidel, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke,Peter Pinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Detection ofprostate cancer using temporal sequences of ultrasound data: a large clinical feasibilitystudy. International Journal of Computer Assisted Radiology and Surgery, 11(6):947â956,2016, and 3) Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, Jin TaeKwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto, Bradford Wood, ParvinMousavi, and Purang Abolmaesumi. Deep recurrent neural networks for prostate cancerdetection: Analysis of temporal enhanced ultrasound. IEEE Transactions on MedicalImaging, 2018293.1. Introductionmore accurate and pragmatic ultrasound-based solution for prostate cancerdiagnosis, in this chapter, we focus on the advancement of probabilisticmodeling of temporal enhanced ultrasound as a recent promising tissuetyping technique. This chapter is subdivided into two parts aiming to modelspectral and temporal aspect of TeUS data by exploiting suitable deeplearning approaches. Over the course of probabilistic modeling of TeUS data,we centered our attention to interpretation of the internal representationslearned by our models to derive insights about tissue dependent featuresresiding in TeUS.As we have discussed in Section 1.2.2.1, in previous implementationsof temporal enhanced ultrasound technology [72, 82, 154], features wereheuristically determined from the spectral/wavelet analysis of the ultrasounddata. In addition to challenges associated with defining features that arecorrelated with the underlying tissue properties, it is also difficult to determinethe best combination of those features for effective tissue typing as the numberof features increases. The lack of a systematic approach for feature selectioncan lead to a so-called âcherry pickingâ of the features [72, 82, 154]. Deeplearning approaches have gained significant attention for capturing abstractrepresentations of input data [22] as a replacement for feature selection,and have been successfully used in medical image analysis [94, 157]. Inthe first part of this chapter, we exploit Deep Belief Networks (DBN) [22]to automatically learn a high-level feature representation from spectralcomponents of temporal ultrasound data that can detect prostate cancer.Subsequently, we use the high-level features in a supervised classification stepbased on Support Vector Machine (SVM) to generate a cancer likelihood map.Further, we investigate the factors that affect the classification accuracywithin a targeted biopsy interface. We demonstrate that this approachis an effective method for identifying both benign and cancerous biopsycores in TRUS-guided biopsy [13]. Our results indicate that TeUS cancomplement mp-MR imaging and together they can be an effective tool forcancer detection.To date, most of the efforts in our group have mainly focused on spectralanalysis as the key pre-processing step for feature extraction from TeUSdata [11]. Recently, analysis of TeUS in the temporal domain using prob-abilistic methods, such as Hidden Markov Models (HMMs), have shownsignificant promise [115, 118]. Thus, in the second part, we propose touse Recurrent Neural Networks (RNNs) [26, 50, 84] to explicitly analyzeTeUS data in the temporal domain. Specifically, we use Long Short-TermMemory (LSTM) networks [50] and Gated Recurrent Unit (GRU) [27, 29],the classes of RNNs, to effectively learn long-term dependencies in the data.303.2. Spectral Analysis of TeUS for prostate cancer diagnosis. . .. . .v: visible unitsh: hidden unitsw: weightsFigure 3.1: An illustration of a Restricted Boltzmann Machine (RBM): RBMconsists of a layer of binary stochastic visible units v, connected to a layerof stochastic hidden units h by symmetrically weighted connections W.In an in vivo study with 157 patients, we analyze data from 255 suspiciouscancer foci obtained during MRI-TRUS fusion biopsy. We achieve AUC,sensitivity, specificity, and accuracy of 0.96, 0.76, 0.98, and 0.93, respectively.Our results indicate that RNN can identify temporal patterns in the datathat may not be readily observable in spectral analysis of TeUS [11], leadingto significant improvements in detection of prostate cancer. We also presentalgorithms for in-depth analysis of LSTM networks.3.2 Spectral Analysis of TeUS for prostate cancerdiagnosis3.2.1 Background: Deep Belief NetworksSince the original DBN article was published [63], DBN has become one ofthe most important models of deep learning. It uses the generative model inthe pre-training procedure and back-propagation in the fine-tuning stage [62].This is very useful when the number of training samples is limited [90],such as medical imaging analysis where the annotation is burdensome andlimited to the target regions. In such a case, the unsupervised pre-trainingof DBN can be beneficial. DBN is also a fast learning algorithm that canfind the optimal parameters quicker [62]. Restricted Boltzmann Machines(RBMs) are the building blocks of DBN. In this section, we quickly reviewthe construction and training of DBN.3.2.1.1 Restricted Boltzmann MachineAn RBM consists of a layer of binary stochastic visible units v, connected toa layer of stochastic hidden units h by symmetrically weighted connections313.2. Spectral Analysis of TeUS for prostate cancer diagnosisW (Fig. 3.1). The joint configuration (v,h) of visible and hidden units hasan energy given by:E(v,h; θ) = ââiâvisiblebivi ââjâhiddenbjhj ââi,jvihiwij , (3.1)where θ = {bi, bj , wij} indicates parameters of the model; vi and hj are thebinary states of visible and hidden units i and j, wij are the weights, biand bj are the bias terms. Using this energy function, the network assigns aprobability to every possible feature vector at the visible units:P (v,h; θ) =1Z(Î¸)exp(âE(v,h; θ)) , (3.2)Z(θ) =âvâhE(v,h; θ) , (3.3)where Z(θ) is the normalizing constant. The network gives a probability toevery input vector via the energy function. The probability of the trainingvector can be raised by adjusting θ to lower the energy. Given a trainingvector v, the binary states h of the hidden units follow the conditionaldistribution:P (hj = 1|v) = Ï(bj +âiviwij) , (3.4)where Ï(x) = 1/(1 + exp(âx)) is the logistic/sigmoid function. Once binarystates of the hidden units h have been chosen, a reconstruction is producedby setting each vi to 1 by following the conditional distribution:P (vi = 1|h) = Ï(bi +âihjwij) , (3.5)The hidden units states are then updated once more so that they representthe features of the reconstruction. The learning of wij â W is done by amethod called Contrastive Divergence (CD) [62, 63]. The change in a weightis given by:âwij = (vihjdata â vihjreconstruction) , (3.6)where  is a learning rate. The term vihjdata is the fraction of times that thevisible unit i and the hidden units j are on together when the hidden unitsare being driven by data. vihjreconstruction is the corresponding fraction forreconstruction. Through the learning process, we can obtain the optimum323.2. Spectral Analysis of TeUS for prostate cancer diagnosisvalue of W. Also, with the same rule, the optimum values of bi and bj â Bcan be learned.The power of RBM lies in the form of reconstruction oriented learning.During reconstruction, it only uses the information in hidden units, which islearned as features from the input. If the model can recover original inputperfectly, it means that the hidden units retain enough information of theinput, and the learned weights and biases can be deemed as good measuresof the input data.3.2.1.2 Deep Belief NetworkA single hidden layer RBM is not the best way to capture the features inthe data. After the training of one RBM, the learned features can be usedas the input data for a second RBM. This kind of layer-by-layer learningsystem can be used to construct DBN. A schematic representation is shownin Fig. 3.2. The DBN is trained in two stages:1. An unsupervised pre-training phase which sets the weights of thenetwork to the approximately right neighborhood.2. A fine-tunig phase where the weights of the network are moved to thelocal optimum by back-propagation on labeled dataThe pre-training is performed from the input layer up to the output layer,following a greedy approach. The pre-training process, as described in Sec-tion 3.2.1.1 is repeated several times, layer by layer, obtaining a hierarchicalmodel in which each layer captures strong high-order correlations betweenits input units. After having greedly pre-trained all network layers [22], theparameters of the deep model, {W,B} are then refined using the labeleddata and back-propagation algorithm. For this purpose, a final logistic re-gression layer is added to the end of feature learning system. In particular,the fine-tuning stage minimizes the cross-entropy error as the loss function,LDBN :LDBN = ââiâOoilogoËi , (3.7)where oËi is the value of the ith unit of the DBN output layer, O and oi isthe ground-truth value of the corresponding labeled input data, followingthe one-hot encoding. Each output node is associated to a specific label andin the case of binary classification, we have one final node. Based on theprevious formulation, we can see, each layer of RBM in the DBN is a process333.2. Spectral Analysis of TeUS for prostate cancer diagnosisof nonlinear feature transformation. Features learned in the top layer ofthe DBN (before the logistic regression layer) are the most representativefeatures for modeling the data. It can be denoted by Hp = (hp1, hp2, ..., hpn),where p represents the top layer, and n is the number of features in the toplayer.3.2.2 Classification Framework Based on DBNThe objective is to develop a deep learning model to discriminate cancerand benign prostate regions using spectral representation of TeUS data asexplained in Section 2.3.2. Let DS = {(f (i), y(i))}|DS |i=1 represent the collectionof all labeled ROIs, where f (i) is the ith TeUS spectral feature vector andy(i) indicates the corresponding label. An individual TeUS spectral featureof length F , f (i) = (f(i)1 , ..., f(i)j , ..., f(i)F ), is composed of F = 50 frequencycomponents of f(i)j and is labeled as yi â {0, 1}, where zero and one indicatebenign and cancer outcome, respectively in histopathology. We aim to learna mapping from f (i) to y(i) in a supervised framework by using DBN toexplicitly model the spectral information in TeUS.Figure 3.2 shows a schematic diagram of our method. Our frameworkconsists of two parts. First, we use DBN [22] to automatically learn a high-level latent feature representation of the temporal enhanced ultrasound, H,that can detect prostate cancer. Second, we then use the hidden activationsof the DBN as the input learned features to an SVM classifier to generate acancer likelihood map.3.2.2.1 Automatic Feature LearningAs we explained in Section 3.2.1.2, DBN, as a generative probabilistic model,can be trained to extract a deep hierarchical representation of the inputdata. We construct a DBN by stacking many layers of RBM, each of whichconsists of a visible layer (v) and a hidden layer of neurons (h). The twolayers are connected by a matrix of weighted connections and there areno connections within a layer. RBM can be stacked by linking the hiddenlayer of one RBM to the visible layer of the next RBM. The role of anRBM is to model the distribution of its input and capturing patterns in thevisible units. Thus, DBNs offer an unsupervised way to learn multi-layerprobabilistic representations of data that are progressively deeper with eachsuccessive layer. Utilizing this representational power, we can find a latentrepresentation of the original low-level spectral features extracted from thetemporal enhanced ultrasound. For this purpose, we feed the visible layer,343.2. Spectral Analysis of TeUS for prostate cancer diagnosisTraining Dataset: D1Feature ExtractionLearned FeaturesGround-truth LabelsHistophatologyTrainingTestingTrained DBN Trained SVMLearnned Features Classifier OutputFeature ExtractionSVM ClassifierPre-trainingBack-PropagationFeature representation1st Hidden Layer2nd Hidden Layer3rd Hidden Layer4th Hidden LayerInput Layer. . .. . .. . .. . . . . .. . .1st RBM2nd RBM3rd RBM4th RBMDeep Belief NetworkTesting Dataset: D2Temporal Ultrasound Data Spectral ComponentsSpectral ComponentsTemporal Ultrasound DataFigure 3.2: An illustration of the proposed method for prostate cancerdetection. Our DBN has a layer of real-valued visible units of dimensionF = 50 and four hidden layers with 100, 50 and 6 hidden units. The red boxcontains the pre-trained DBN, and the blue box containing one neuron isadded for the fine-tuning step. The latent features are the output of the lastlayer of DBN.v, of our network with the spectral features from TeUS, f (i) â DS where|v| = F = 50. After the pre-training of DBN, as we explained in detail inSection 3.2.1.2, we use the corresponding y(i) to fine-tune the network. We seta final logistic regression layer including only a node, to represent our binarylabel. After training of DBN, given the input spectral feature f = (f1, , ..., fF ),DBN computes the hidden vector sequence Hp = (hp1, hp2, ..., hpn) in thefeature learning step, where p represents the top layer, and n is the numberof features in the top layer. This hidden vector, Hp is a function of theinput sequence f , and model parameters, ÎDBN = {θ1, ..., θp} = {W,B}, asexplained in previous section.3.2.2.2 Cancer ClassificationOur classification approach is built by an SVM layer to map the learnedfeatures from spectrum of TeUS, fDBN : f 7â Hp, to a posterior overclasses. The SVM classification model learns a distribution over classesP (y|f ,ÎDBN ,ÎSVM ) given a spectral feature vector of f = (f1, , ..., fF ), theparameter of the trained DBN, ÎDBN , and the parameter of SVM classifier,353.2. Spectral Analysis of TeUS for prostate cancer diagnosisÎSVM . The process of training an SVM as a marginal classifier is equivalentto finding the optimal hyper-plane that maximizes the perpendicular distancebetween the decision boundary and the closest data points in classes andminimizes the decision error on the training dataset [108]. Considering{H(1)p , ...,H(N)p } as training dataset consists of N learned feature vectors withthe corresponding label class of yi â {â1, 1} (i.e. unlike previous notations,negative one indicate the benign samples), the SVM training problem isequivalent to finding ÎSVM = {W, b} such that:12ÃWTW + Ci=Nâi=1Î¾(i) , (3.8)is minimized subject toyi(WTÏ(Hp(i)) + b) â¥ 1â ξ(i) , (3.9)the slack parameters, ξ(i), are added to the equations to allow for mis-classification of noisy data, and C > 0 is a penalty parameter for the errorterm that controls over-fitting. The function Ï(x) maps the data to a higherdimensional space. We us the Radial Basis Function (RBF) kernel, as justifiedin [108], which is defined as K(x, xË) = exp(âÎ³âxâ xËâ2) = Ï(x)TÏ(x).Let CS = {(f (i), y(i))}|CS |i=1 represent the collection of all labeled ROIssurrounding a target core, |CS | = 80, f (i) represents the ith TeUS sequenceof the core, and y(i) indicates the corresponding label. Using the probabilityoutput of the classifier for each ROI, we assign a binary label to each targetcore. The label is calculated using a majority vote based on the predictedlabels of all ROIs surrounding the target. For this purpose, we define thepredicted label for each ROI, y(i), as 1, when PCS (y(i)|f (i)) â¥ 0.5, and as0 otherwise. The probability of a given core being cancerous based on thecancerous ROIs within that core is:PCS =â|CS |i I(y(i) = 1)|CS | . (3.10)A binary label of 1 is assigned to a core, when PCS â¥ 0.5.3.2.2.3 Spectral Feature VisualizationTo determine the characteristics of the benign and cancerous cores in the tem-poral enhanced ultrasound data and their correlation with learned features,we propose a feature visualization approach (Fig. 3.3). This approach is used363.2. Spectral Analysis of TeUS for prostate cancer diagnosisTraining Dataset: D1Feature ExtractionGround-truth LabelsHistophatologySpectral ComponentsTemporal Ultrasound DataTrained DBNLearned Featuresi th FeatureTesting Dataset: D2BackpropagationAbsolute Value-Visualization0 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.901234567x 1010Frequency (Hz)Absolute difference of Means0 10 20 30 40 50-2-1.5-1-0.500.511.52x 1011Frequency (Hz)Absolute difference of Means0 10 20 30 40 50-1-0.500.511.5x 1011Frequency (Hz)Absolute difference of MeansFigure 3.3: An illustration of the proposed feature visualization method.to identify the most discriminative features of the time series (i.e. frequencycomponents as introduced in Section 2.3.2), as learned by the classifier. First,data is propagated through the trained DBN and the activations of the lasthidden layer, i.e. the learned latent features are computed. To examine thesignificance of an individual learned latent feature, the activations of all otherhidden units in the third layer are set zero. The activation of the non-zerolearned feature is back-propagated to the input layer. The resulting signal,displayed in the input layer as a series of frequency components, highlightsthose components that contribute to the activation of the non-zero learnedfeature. By comparing the components activated for benign and cancerouscores, we can identify those frequency ranges that are different between twotissue types. This process is performed for all latent features.3.2.3 Results and Discussion3.2.3.1 Data DivisionThe reported registration accuracy for the Philips UroNav MR/US fusionsystem is 2.4 ± 1.2mm [162]. However, mis-registration is usually moreprominent for targets close to the segmented boundary of the prostate. Forbiopsy cores taken far away from the boundary, we assume that the targetis in the center of the core. However, clinicians normally adjust the needlepenetration depth for targets that are close to the boundary, especially inthe anterior region, so that the core sample is not taken beyond the prostate.To generate our spectral model, we aim to use homogeneous prostate tissueregions with reliable ground-truth labels. Therefore, we select cores fortraining if they meet all of the following three selection criteria, similar to373.2. Spectral Analysis of TeUS for prostate cancer diagnosisTable 3.1: Gleason score distribution in TeUS test and train dataset. Tableshows the number of cores for each category.Gleason Score Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 GS 4+5DStrain 19 0 5 2 4 2DStest 153 26 21 3 17 3our previous work [13]: 1) located more than 3 mm distance to the prostateboundary in TRUS images; 2) have matching histopathology labels betweenaxial and sagittal biopsies; and 3) have a tumor length larger than 7 mm ifcancerous. We select 32 cores from 29 patients, that fulfill the above criteria,and use the temporal enhanced ultrasound data from them to generate ourmodel. These 32 training cores are labeled as dataset DStrain, where 13 coresare cancerous and 19 cores are benign. we obtain a total number of 32Ã1,536= 49,152 training samples (N = |DStrain| = 49, 152) (see Section 2.3). Theremaining 223 biopsy cores distributed as presented in Table 3.1, are used astest dataset, DStest. The test dataset, DStest, are divided into three sub-groupsbased on the distance of the target to prostate boundary and agreementbetween axial and sagittal histopathology labels [125]:1. Dataset DAtest, consisting of 156 cores from 150 patients whose targetdistance to prostate boundary (d) is â¥ 3 mm.2. Dataset DBtest, consisting of 117 cores from 91 patients in dataset DAtestthat also have the agreement in histopathology labels of axial andsagittal biopsy cores.3. Dataset DCtest, consisting of 168 cores from 115 patients whose have theagreement in histopathology labels of axial and sagittal biopsy cores.4. Dataset DÂ¯Atest, consisting of 67 cores whose target distance to prostateboundary is < 3 mm.The distribution of the histopathology labels of the DStrain and DStest issummarized in Table 3.1.3.2.3.2 Hyper-parameter SelectionThe primary step for training a deep network is to find a proper networkstructure including the number of hidden layers and the number of neurons383.2. Spectral Analysis of TeUS for prostate cancer diagnosisin each layer. Furthermore, there are various hyper-parameters such asthe learning rate, the momentum, the weight-cost and the size of eachmini-batch that have effects on the training procedure [61]. To set thesehyper-parameters, we heuristically tried different network structures so thatlowest reconstruction error with the default library [147] values for all otherhyper-parameters is obtained in the training data. Next, we followed theguidelines given by Hinton [61] to adjust other hyper-parameters in a wayto obtain lower reconstruction error in the training data. The finalizeddeep network has a real-valued visible layer with F = 50 units equal tothe number of input spectral features. Three hidden layers of the networkinclude 100, 50 and 6 hidden units, respectively. The learning rate, , isfixed at 0.001, mini-batch size is 5 and the number of passes is 100 epochs.Moreover, the momentum and the weight cost have the default values of 0.9,and 2 Ã 10â4, respectively. The discriminative fine-tuning of the DBN isperformed by back-propagation, which requires a final node for representingdesired labels of the observations. The fine-tuning step is performed with thelearning rate of 0.01 for 70 epochs and mini-batch size of 10. After completionof the learning procedure, the last hidden layer of the DBN produces thelatent feature representation. We use training dataset DStrain to obtain thelearned features from the last hidden layer of the trained DBN. Then weuse these features as inputs to a non-linear SVM classifier. We have sixlearned features corresponding to the activations of the six hidden units inthe last hidden layer. The SVM classifier uses an RBF kernel; we determinethe parameters of the classifier through a grid-search approach [76]. Forthe parameter selection (C and γ), we exhaustively searched the parameterspace C â {2â5, 2â3..., 25} and γ â {2â10, 2â8, ..., 24}. Following training,we use the SVM classifier on the test data to derive the tissue type labels foreach ROI.3.2.3.3 Classification PerformanceTo assess the performance of our method, sensitivity, specificity, and accuracywere calculated. We also report the overall performance using the area underthe receiver operating characteristic curve (AUC). Table 3.2 shows theclassification results for test dataset DStest. Our results indicate that datasetDBtest has consistently higher classification results than dataset DAtest across allMR suspicious levels. A closer look at cores in dataset DAtest also shows thatfor those samples that are farther from the prostate boundary (at least 5 mmaway) and have moderate MR suspicious level (53 cores), we achieve AUC of0.89, irrespective of mismatch between axial and sagittal histopathology. In393.2. Spectral Analysis of TeUS for prostate cancer diagnosisTable 3.2: Model performance for classification of testing cores for differentMR suspicious levels. N indicates the number of cores in each group.MR suspicious levels Dataset DAtest Dataset DBtest Dataset DCtestN AUC N AUC N AUCAll MR suspicious levels 156 0.73 117 0.77 168 0.66Moderate MR suspicious level 110 0.75 84 0.80 119 0.67High MR suspicious level 23 0.78 15 0.71 16 0.84comparison, only 26% of those cores are identified as cancerous after biopsywhich means our approach can effectively complement mp-MRI to reducethe number of false positives for those targets with moderate MR suspiciouslevel.We also perform a similar analysis for dataset DÂ¯Atest, where we obtainAUC of 0.36. There are various factors that may have contributed to thisdrop in classification accuracy, including higher registration error among mp-MRI, TRUS and histopathology for targets close to the segmented prostateboundary, and the inclusion of ultrasound signal from tissue outside theprostate [80, 81]. Moreover, based on the clinical protocol, for targets thatare close to the prostate boundary, the biopsy core is not centered on thetarget location to minimize the penetration of needle in tissue surroundingthe prostate. A more accurate ground-truth data needs to be obtainedto further validate our approach on targets that are close to the prostateboundary.Moreover, we perform analysis on dataset DAtest without the elementsof DBtest. This analysis was done on 39 cores from 34 patient whose targetdistance to the boundary is more than 3 mm and have the disagreementin histopathology labels of axial and sagittal biopsy cores. Our resultsshowed that by using axial plane histopathology as the ground-truth label,we achieved an AUC of 0.73. On the other hand, by using sagittal planehistopathology as the ground-truth label, we obtained an AUC of 0.60. Oneof the factors that may have contributed to this performance is the fact thattemporal ultrasound data is only obtained from the axial plane and no tissuetyping information is available from the sagittal plane.3.2.3.4 Choice of Training DataIn another experiment, to ensure that the classification model does notover-fit to the training data, we trained our SVM classification model using403.2. Spectral Analysis of TeUS for prostate cancer diagnosisdataset DStest in a fold validation manner. We obtained AUC of 0.71 forDAtest and AUC of 0.73 for DBtest in a leave-one-out cross-validation analysis.We also obtained AUC of 0.71 and 0.70 for DAtest in three-fold and 13-foldcross-validation analysis, respectively. The averaged AUC of leave-one-outcross-validation analysis follows our previous performance results, whichsupports the generalization of the classification model.We performed an additional sensitivity analysis by permuting the trainingand testing data. To create new training and testing sets, in each permutation,we exchanged a randomly selected cancerous or benign core in the trainingand testing data. The cores are selected from dataset DBtest. This resulted in32 different permutations given the distribution of cores in our training data.As Table 3.3 shows, on average, we achieved AUC, accuracy, sensitivity, andspecificity of 0.70, 71%, 68%, and 70%, respectively.Table 3.3: Model performance in the fold validation analysis for testing coresin datasets DAtest and DBtest.Evaluation Dataset DAtest Dataset DBtestleave-one-out cross-validation 0.71(±0.02) 0.73(±0.02)three-fold cross-validation 0.71(±0.01) 0.73(±0.07)13-fold cross-validation 0.70(±0.05) 0.72(±0.03)3.2.3.5 ColormapsFigure 3.4 shows examples of the cancer likelihood maps from dataset DBtest,derived from the output of SVM, overlaid on B-mode ultrasound image. Weuse the approach described in our earlier publication [108] for this purpose.In the colormaps, red regions belong to ROIs for which the cancer likelihoodis more than or equal to 70%. We found that with this threshold, thevisualized maps demonstrated all the major tumors in the dataset without alarge number of false positives.3.2.3.6 Analysis of Tumor SizeTo investigate the effect of the size of the tumor on our detection accuracy, weanalyzed the AUC against the greatest length of the tumor in MRI (rangingfrom 0.3 cm to 3.8 cm) for DBtest. We obtained the average AUC of 0.77 forcores with MR-tumor-size smaller than 1.5 cm, and the average AUC of 0.93for cores with MR-tumor-size larger than 2 cm. The results show our method413.2. Spectral Analysis of TeUS for prostate cancer diagnosis(a) (b)Figure 3.4: Cancer probability maps overlaid on B-mode ultrasound image,along with the projected needle path in the temporal ultrasound data andcentered on the target. The ROIs for which the cancer likelihood is morethan 70% are colored in red, otherwise they are colored in blue. Thegreen boundary shows the segmented prostate in MRI projected in TRUScoordinates, dashed line shows needle path and the arrow pointer showsthe target: (a) Correctly identified benign core; (b) Correctly identified thecancerous core.0 0.5 1 1.5 2 2.5 3Greatest Tumor Length in MRI (cm)0.760.780.80.820.840.860.880.90.920.94AreaUndertheCurve(AUC)Binary ClassificationCombinition with MRIFigure 3.5: Investigation of the effect of tumor size on accuracy. We obtainedthe average AUC of 0.77 for cores with MR-tumor-size smaller than 1.5 cm,and the average AUC of 0.93 for cores with MR-tumor-size larger than 2 cm.has higher performance for larger tumors. Figure 3.5 shows the effect of thesize of the tumor on our detection accuracy and the accuracy of the methodwhen spectral analysis of TeUS is augmented with mp-MR readings.423.2. Spectral Analysis of TeUS for prostate cancer diagnosis1 2 3 4 5 6 7 8 9 1000.20.40.60.81Frequency (Hz)Absolute difference of Means(a)1 2 3 4 5 6 7 8 9 1000.20.40.60.81Frequency (Hz)Absolute difference of Means(b)Figure 3.6: Differences of distributions between cancerous and benign tissueback projected in the input neurons: (a) corresponds to the first neuron inthe third hidden layer; (b) corresponds to the sixth neuron in the third hiddenlayer. Results are shown in the frequency range of temporal ultrasound dataanalyzed in this section. It is clear that frequencies between 0â2 Hz providethe most discriminative features for distinguishing cancerous and benigntissue.3.2.3.7 Feature VisualizationFor the feature visualization experiment, we found that feature six, corre-sponding to the hidden activity of the sixth neuron of the third layer, alongwith features one and four, are those that maximally differentiate cancerousand benign tissue, especially in the lower frequency range. Figure 3.6 showsthe visualization of distribution differences for cancerous and benign tissuerelated to the first and sixth learned features of the third hidden layer, backpropagated to the input layer.While the physical phenomenon governing temporal ultrasound/tissueinteraction is the subject of ongoing investigation in our group, severalhypotheses have been explored so far. It has been proposed that the acousticradiation force of the transmit ultrasound signal increases the temperatureand changes the speed of sound in different tissue types [36]. It has alsobeen suggested that a combination of micro-vibration of acoustic scattersin microstructures and the density of cells play a role [111]. Our resultsshowed a consistently high classification accuracy in a large dataset in thissection. These results suggested that the phenomenon is consistent for the twoindependent training and test datasets in clinical settings. Interestingly, therange of frequencies that we have identified as most discriminative between433.3. Temporal Analysis of Temporal Enhanced Ultrasoundcancerous and benign tissue (0â 2 Hz in Fig. 3.6) are also consistent withthe ranges we have observed in our previous independent studies [75, 76]. Wewill further analyze the physical phenomenon governing temporal ultrasoundin Chapter 6 and present the theoretical background of TeUS in Appendix A.Since DBN is a computationally expensive method, to optimize ourproposed method for real-time display of cancer likelihood maps, a parallelimplementation on Graphics Processing Unit (GPU) is necessary. For sucha parallel implementation, Fourier transform computation, the need forinformation transfer between GPU and CPU, and integration of the resultswith the UroNav targeted biopsy interface are the bottlenecks. Currently,the execution time for generating a cancer probability map overlaid on aB-mode ultrasound image using an Intel Core i7 CPU with 16 GB RAM isapproximately 6 minutes. Thus, an intrinsically time-dependent deep network,capable of real-time and on-fly analysis of the TeUS frames, is preferable.Such a model could mitigate the overhead computation of spectral features.In the next section, we will focus on the temporal analysis of temporalenhanced ultrasound.3.3 Temporal Analysis of Temporal EnhancedUltrasound3.3.1 Background: Recurrent Neural NetworksRNNs are a category of neural networks that are âdeepâ in temporal dimen-sion and have been used extensively in time-sequence modeling [58]. Unlike aconventional neural network, RNNs are able to process sequential data pointsthrough a recurrent hidden state whose activation at each step depends onthat of a previous step. Generally, given sequence data x = (x1, , ..., xT ), anRNN updates its recurrent hidden state ht by:ht ={0, if t = 0Ï(htâ1, xt), otherwise(3.11)where xt and ht are data values and the recurrent hidden state at time step t,respectively, and Ï(.) represents the nonlinear activation function of a hiddenlayer, such as a sigmoid or hyperbolic tangent. Optionally, the RNN mayhave an output y = (y1, , ..., yT ). In the traditional RNN model aka vanilla,the update rule of the recurrent hidden state in (3.11) is implemented as:ht = Ï(Wxt + Uhtâ1) , (3.12)443.3. Temporal Analysis of Temporal Enhanced UltrasoundInput TeUSLayer 1Layer 2. . .. . .Representation Sequence. . . hTx1 xTht-1ï³ ï³ ÏÏï³ct-1htcthtxtÏ Nonlinear activation functionPointwise Operations(ht,ct )ft it oth1yi ï {0, 1}ï³ Sigmoid functionxi = (x1, â¦, xT)LSTM cellFCFC Fully Connected LayerFigure 3.7: Overview of the proposed method. We use two layers of RNNswith LSTM cells to model the temporal information in a sequence of TeUSdata. x(i) = (x1, , ..., xT ), T = 100 is showing the ith sequence data and xt isindicating the tth time step.where W and U are the coefficient matrices of the input at the presentstep and the recurrent hidden units activation at the previous step, respec-tively. We can further expand Equation (3.12) to calculate the hidden vectorsequence h = (h1, , ..., hT ):ht = Ï(Wihxt + Whhhtâ1 + bh) , (3.13)where t = 1 to T , Wih denotes the input-hidden weight vector, Whh representsthe weight matrix of the hidden layer, and bh is the hidden layer bias vector.It has been observed that using the traditional RNN implementation,gradients decrease significantly for deeper temporal models. This makeslearning of long-term sequence data a challenging task for RNNs. To addressthis issue, other types of recurrent hidden units such as LSTM and GRUhave been proposed. As shown in Equations (3.12) and (3.13), traditionalRNN simply applies a transformation to a weighted linear sum of inputs. Incontrast, an LSTM-based recurrent layer creates a memory cell c at eachtime step whose activation is computed as:ht = otÏ(ct) , (3.14)where ot is the output gate that determines the portion of the memory cellcontent in time step t (ct) to be exposed at the next time step [50]. Therecursive equation for updating ot is:ot = Ï(Woixt + Wohhtâ1 + Wocctâ1 + bo) , (3.15)453.3. Temporal Analysis of Temporal Enhanced Ultrasoundwhere Ï(.) is the logistic sigmoid function, Woi is the input-output weightmatrix, Woh is the hidden layer-output weight matrix, and Woc is thememory-output weight matrix. The memory cell, ct, is updated by addingnew content, ct, and discarding part of the present memory:ct = it ct + ft ctâ1 , (3.16)where is an element-wise multiplication and ct is calculated as:ct = Ï(Wcixt + Wchhtâ1 + bc) , (3.17)In this equation, the W terms represent weight matrices; e.g., Wci is theinput-memory weight matrix. Input gate i, and forget gate f determine thedegree that new information is to be added and current information is to beremoved, respectively, as follows:it = Ï(Wixxt + Wihhtâ1 + Wicctâ1 + bi) ; (3.18)ft = Ï(Wfxxt + Wfhhtâ1 + Wfcctâ1 + bf ) . (3.19)All weight matrices, W, and biases, b, are free parameters that are sharedbetween cells across time. Figure 3.7 shows a graphical model of an LSTMcell. A slightly different version of LSTMs are GRUs [29] which have a fewernumber of parameters to avoid over-fitting in the lack of sufficient trainingsamples. GRUs combine the forget and input gates into a single updategate, u, and merge the cell memory and hidden state to a reset gate, r. Theactivation of ht of the GRU at time t is a linear interpolation between theprevious activation, htâ1, and the updated activation, ht:ht = (1â ut)htâ1 + htut , (3.20)where ut, the update gate at time step t, determines how much the unitupdates its activation or content. The update gate can be calculated asfollows:ut = Ï(Wuixt + Wuhhtâ1) , (3.21)where Wui is the input-update weight matrix and Wuh denotes the update-hidden weight matrix. The updated activation, ht, is computed similarly tothe traditional RNN in Equation (3.12) as follows:ht = Ï(Wpixt + Wrh(rt htâ1)) . (3.22)463.3. Temporal Analysis of Temporal Enhanced UltrasoundFinally, the reset gate, rt, is computed as:ut = Ï(Wrixt + Wrhhtâ1) . (3.23)3.3.2 Classification Framework Based on RNN3.3.2.1 Proposed Discriminative MethodOur overarching objective is to develop a deep learning model to discriminatecancer and benign prostate regions in TeUS data. Let DT = {(x(i), y(i))}|DT |i=1represent a collection of all labeled ROIs, where x(i) is the ith TeUS sequenceand y(i) indicates the corresponding label. An individual TeUS sequence oflength T , x(i) = (x(i)1 , ..., x(i)T ), is composed of echo-intensity values x(i)t foreach time step, t, and is labeled as yi â {0, 1}, where zero and one indicatebenign and cancer outcome, respectively in histopathology (see Section 2.3.1).We aim to learn a mapping from x(i) to y(i) in a supervised framework byusing RNNs to explicitly model the temporal information in TeUS. Oursequence classification approach is built with connected RNN layers followedby a softmax layer to map the sequence to a posterior over classes. Each RNNlayer includes T = 100 homogeneous hidden units (i.e., traditional/vanillaRNN, LSTM or GRU cells) to capture temporal changes in TeUS data. Themodel learns a distribution over classes P (y|x1, ..., xT ) given a time-seriessequence x1, ..., xT rather than a single, time independent input. Figure 3.7shows an overview of the proposed architecture with LSTM cells.Given the input sequence x = (x1, ..., xT ), RNN computes the hiddenvector sequence h = (h1, ..., hT ) in the sequence learning step. As discussedin Section 3.3.1, h is a function of the input sequence x, model parameters,Î, and time, t: Ï(x; Î, t). Î = {W,B} denotes the sequence learningmodel parameters, where W is the set of weights and B is the set of biasesin Eq. (3.13) for vanilla RNN cells, in Eq. (3.14) for LSTM cells, and inEq. (3.20) for GRU cells through time steps, t = 0 to t = T . All weightmatrices, W, and biases, B, are free parameters that are shared across time.The final node generates the posterior probability for the given sequence:z(i) = wTs h + bs ; (3.24)y(i)b = arg maxjS(z(i)j ), j â {0, 1}, z(i) = WTs h + bs , (3.25)where Ws and bs are the weight and bias of the fully-connected layer, S isthe softmax function, which in our binary classification case is equivalent to473.3. Temporal Analysis of Temporal Enhanced Ultrasoundthe logistic function, and y(i) indicates the predicted label. The optimizationcriterion for the network is to maximize the probability of the training labelsor equivalently, to minimize the negative log-likelihood defined as a the lossfunction. This function is the binary cross-entropy between y(i) and y(i) overall training samples, DTtrain = {(x(i), y(i))}Ni=1 â DT :L(y,y) = â 1NNâi=1[y(i) log y(i) + (1â y(i)) log(1â y(i))], (3.26)where N = |DTtrain|. During training, the loss function is minimized througha proper gradient optimization algorithm like stochastic gradient descent(SGD), root mean square propagation (RMSprop) or adaptive momentestimation (Adam) [83].3.3.2.2 Cancer ClassificationThe RNN models learn a probability distribution over classes, P (y|x1, , ..., xT ),given a time-series sequence, x1, , ..., xT . Let CT = {(x(i), y(i))}|CT |i=1 representthe collection of all labeled ROIs surrounding a target core, where CT â DTtest,|C| = 80, x(i) represents the ith TeUS sequence of the core, and y(i) indicatesthe corresponding label. Using the probability output of the classifier foreach ROI, we assign a binary label to each target core. The label is calculatedusing a majority vote based on the predicted labels of all ROIs surroundingthe target. For this purpose, we define the predicted label for each ROI,y(i), as 1, when P (y(i)|x(i)) â¥ 0.5, and as 0 otherwise. The probability of agiven core being cancerous based on the cancerous ROIs within that core is:PCT =â|CT |i I(y(i) = 1)|CT | . (3.27)A binary label of 1 is assigned to a core, when PCT â¥ 0.5.3.3.2.3 Network AnalysisTo better understand the temporal information in TeUS, we examine theLSTM gates. For this purpose, following training, we use the learned weightsand biases to regenerate the network behavior for any given sequence oflength T . First, the state of each cell is set to zero. Then, the full learningformula (Eqs. (3.14)-(3.19)) along with the model parameters Î = {W,B}are recursively applied for T = 100 time steps. A summary of the steps is483.3. Temporal Analysis of Temporal Enhanced Ultrasoundpresented in Algorithms 1 and 2. Finally, the on-and-off behavior of thehidden activation in the last layer of the network is used to analyze the highlevel learned features.Algorithm 1 Examination of the LSTM gatesInput: Trained model parameters âÎ = {W,B}â, input data âXâ, numberof time-steps âTâ, number of input sequence âNâ.Output: States activation âSâ, gates activation âGâInitialization: Set the state of each cell âinStatesâ to zero.1: for i = 0 to N do2: for t = 0 to T do3: xâ X(i, :)4: {S(i, t),G(i, t)} â Step(x, inStates(i, t),W,B)5: inState(i, t)â S(i, t)6: end for7: end for8: return S,G3.3.3 Experiments3.3.3.1 Data DivisionData is divided into mutually exclusive training, DTtrain, and test sets, DTtest.Training data is made up of 84 cores from patients with homogeneous tissueregions (See [14] for more details.). Therefore, the training data are selectedfrom biopsy cores with at least 4.0 mm of cancer in a typical core length of18.0 mm, 26 of which are labeled as clinically significant cancer with GS â¥4+3. The other half is randomly selected from all available non-cancerouscores. The test data consists of 171 cores, where 130 cores are labeled asbenign, 29 cores with GS â¤ 3+4, and 12 cores with GS â¥ 4+3. Giventhe data augmentation strategy in Section 2.3 (Fig. 2.4), we obtain a totalnumber of 84Ã1,536 = 129,024 training samples (N = |DTtrain| = 129, 024).3.3.3.2 Hyper-parameter SelectionThe performance of deep RNNs, similar to other deep learning approaches,are affected by their hyper-parameters. In practice, hyper-parameter selectioncan be constrained as a generalization-error minimization problem. Solutionsare often based on running trials with different hyper-parameter settings, andchoosing the setting that results in the best performing model (Fig. 3.8). We493.3. Temporal Analysis of Temporal Enhanced UltrasoundAlgorithm 2 Recurrent Step Function of the LSTMInput: Trained model parameters âÎ = {W,B}â, input sequence âxâ, inputstates of time step (tâ 1) âSâ.Output: States activation of the current time step (t) âStâ, gates activationof the current time step (t) âGtâ1: procedure Step(x, Stâ1, W, B)2: Woi,Woh,Woc,Wci,Wch,Wix,Wih,Wfx,Wfh,Wfc âW3: bo, bc, bi, bf â B4: htâ1, ctâ1 â Stâ15: it â Ï(Wixx+Wihhtâ1 +Wicctâ1 + bi)6: ft â Ï(Wfxx+Wfhhtâ1 +Wfcctâ1 + bf )7: ot â Ï(Woix+Wohhtâ1 +Wocctâ1 + bo)8: câ Ï(Wcix+Wchhtâ1 + bc)9: ct â it ct + ft ctâ110: ht â otÏ(ct)11: St â {ht, ct}12: Gt â {it, ft, ot, ct}13: return St,Gtoptimize the hyper-parameters through a grid search, which is an exhaustivesearch through a pre-specified subset of the hyper-parameter space of thelearning algorithm.The grid search starts with randomly partitioning the selected trainingdataset, DTtrain, into training (80%) denoted by Dtr and held-out validationsets (20%) denoted by Dval. This partitioning results in Ntr = |Dtr| =103, 219 training samples and Nval = |Dval| = 25, 805 held-out validationsamples. To guide the grid search algorithm, we track the loss, accuracy, andAUC on both Dtr and Dval. The loss is defined using Eq. (3.26) as the binarycross-entropy between the predicted label and the true label, while accuracyis the percentage of the correctly predicted labels. To stabilize learning andprevent the model from over-fitting on the training data, we use regularizationand dropout, two of the most effective proposed strategies [61]. Regularizationadds a penalty term to the loss function (Equation (3.26)) to prevent thecoefficients from getting too large. Here, we use L2 regularization in the formof Î»âÏâ22, where we search for λ as the hyper-parameter. Dropout preventsco-adaptations on training data. In each step of training, a dropout layerremoves some units of its previous layer from the network, which means thenetwork architecture changes in every training step. These units are chosenrandomly based on the probability parameter of the dropout layer as another503.3. Temporal Analysis of Temporal Enhanced Ultrasound0 20 40 60 80 100Iteration (Epochs)00.10.20.30.40.50.60.7Losssgd, lr =0.01rmsprop, lr =0.01adam, lr =0.01sgd, lr =0.0001rmsprop, lr =0.0001adam, lr =0.0001(a) LSTM0 20 40 60 80 100Iteration (Epochs)00.10.20.30.40.50.60.7Losssgd, lr =0.01rmsprop, lr =0.01adam, lr =0.01sgd, lr =0.0001rmsprop, lr =0.0001adam, lr =0.0001(b) GRU0 20 40 60 80 100Iteration (Epochs)00.10.20.30.40.50.60.7Losssgd, lr =0.01rmsprop, lr =0.01adam, lr =0.01sgd, lr =0.0001rmsprop, lr =0.0001adam, lr =0.0001(c) Vanilla RNNFigure 3.8: Comparison between optimizer performance for different RNNcells: Each curve corresponds to an RNN network structure with two hiddenlayers, batch size of 128 with the dropout rate of 0.2 and regularization termof 0.0001.hyper-parameter. We perform a grid search over the number of RNN hiddenlayers, nh â {1, 2}, batch size, bs â {64, 128}, and initial learning rate, lr â{0.01â0.0001} with three different optimization algorithms, SGD, RMSpropand Adam [83]. We also experiment with various levels of dropout rate,dr â {0.2, 0.4} and L2-regularization term (λ), lreg â {0.0001, 0.0002}. Theseresult in 96 different hyper-parameter settings for the proposed approach.All models are trained with the same number of iterations and training isstopped after 100 epochs. Models benefit from reducing the learning rateby a factor once learning stagnates [61]. For this purpose, we monitor thevalidation loss and if no improvement is observed over 10 epochs, the learningrate is reduced by lrnew = lr Ã factor, where factor = 0.9.3.3.3.3 Model Training and EvaluationOnce the optimum hyper-parameters are identified, the entire training set,DTtrain, is used to learn the final model. The loss is used as the performancemeasure for early stopping to avoid overfitting. Training is stopped if theloss as we defined in Eq. (3.26) increases or if it does not decrease after10 epochs. An absolute change of less than δ = 0.0004 is considered as noimprovement in the loss. We also record the model performance for Dval totrack its behavior in a random subset of training data.To assess the performance of our method, we report its sensitivity, speci-ficity, and accuracy in detecting cancerous tissue samples in the test data,DTtest. All cancerous target cores are considered as the positive class (labeledas 1), and non-cancerous cores as the negative class (labeled as 0). Sensitivityor recall is defined as the percentage of cancerous cores that are correctly513.3. Temporal Analysis of Temporal Enhanced Ultrasoundidentified, while specificity is the proportion of non-cancerous cores thatare correctly classified. Accuracy is the ratio of the true results (both truepositives and true negatives) over the total number of cores. The overallperformance of the models is reported using AUC. The curve depicts arelative trade-off between sensitivity and specificity. The maximum value forAUC is 1, where higher values indicate better classification performance.3.3.3.4 ImplementationWe implement the RNNs in Keras [28] using the Tensorflow [1] back-end.Training is performed on a GeForce GTX 980 Ti GPU with 6 GB of memory,hosted by a machine running Ubuntu 16.04 operating system on a 3.4 GHzIntel CoreTM i7 CPU with 16 GB of memory. Training of vanilla RNN,LSTM and GRU network structures with 100 epochs takes 1.1, 8.1 and 3.3hours, respectively. Early stopping and calculation of additional performancemetrics are implemented using Keras callbacks and the Tensorflow back-endto evaluate internal states and statistics of the model during training. Theproposed network analysis method in Algorithms 1 and 2, is implementedindependently of Keras or Tensorflow in Python 2.7, executed on CPU.3.3.4 Results and Discussion3.3.4.1 Model SelectionResults from hyper-parameter search demonstrate that network structureswith two RNN hidden layers outperform other architectures. Furthermore,for the vanilla RNN, bs = 128, lr = 0.0001; for LSTM, bs = 64, lr = 0.0001;and for GRU, bs = 128, lr = 0.01 generate the optimum models. For allmodels, dr = 0.2 and lreg = 0.0001 generate the lowest loss and the highestaccuracy for both Dtr and Dval. The learning curves for different optimizationalgorithms and initial learning rates on Dtr are shown in Fig. 3.8. Eachcurve corresponds to an RNN network structure with two hidden layers, thebatch size of 128 with the dropout rate of 0.4 and regularization term ofλ = 0.0001, where the vertical axis is the loss and the horizontal axis is thenumber of iterations. It is clear that RMSprop substantially outperformsSGD optimization for all of the RNN cell types while RMSprop and Adamhave similar performance for GRU and LSTM cells. RMSprop leads to abetter performance on our data.Learning curves of the different RNN cells using the optimum hyper-parameters are shown in Fig. 3.9. The right-vertical axis represents theloss value while the left-vertical axis shows the accuracy and AUC, and523.3. Temporal Analysis of Temporal Enhanced Ultrasound0 10 20 30 40 50 60Iteration (Epochs)0.20.30.40.50.60.70.80.91Loss0.20.30.40.50.60.70.80.91Accuracy/AUCTrain LossTrain AccuracyValidation LossValidation AccuracyValidation AUC(a) LSTM0 10 20 30 40 50 60Iteration (Epochs)0.20.30.40.50.60.70.80.91Loss0.20.30.40.50.60.70.80.91Accuracy/AUCTrain LossTrain AccuracyValidation LossValidation AccuracyValidation AUC(b) GRU0 10 20 30 40 50 60 70Iteration (Epochs)0.20.30.40.50.60.70.80.91Loss0.20.30.40.50.60.70.80.91Accuracy/AUCTrain LossTrain AccuracyValidation LossValidation AccuracyValidation AUC(c) Vanilla RNNFigure 3.9: Learning curves of different RNN cells using the optimum hyper-parameters in our search space. All of the models use the RMSprop optimizerand converge after 65±7 epochs.the horizontal axis is the number of iterations. We observe that all modelsconverge after 65±7 epochs, and GRU and LSTM cells outperform vanillaRNN cells in terms of accuracy. Comparing Fig. 3.9(a) and (b) demonstratesthat the network with GRU cells has a steeper learning curve and convergesfaster than the network with LSTM cells. One possible reason could be thefewer number of parameters to be learned in GRU cells compared to LSTMcells. Fig. 3.9 shows that the network with LSTM cells leads to a lower lossvalue and a higher accuracy.3.3.4.2 Model PerformanceTable 3.4 shows the classification results in the test dataset, DTtest, including171 target cores. Models with LSTM and GRU cells consistently achievehigher performance compared to vanilla RNN and the spectral methodproposed in [11]. A two-way paired t-test shows statistically significantimprovement in AUC (p < 0.05) with LSTM and GRU cells. Moreover, theLSTM configuration has the highest performance for detection of cancer.Using the LSTM model as the best configuration, we achieve specificityand sensitivity of 0.98 and 0.76, respectively, where we classify 31 out of41 cancerous cores correctly. Table 3.5 and Table 3.6 shows performance ofmodels for classification of cores in DTtest for different MR suspicious levelsas explained in Section 2.1. For samples with moderate MR suspiciouslevel (70% of all cores), we achieve AUC of 0.97 using the LSTM-RNNstructure. In this group, our sensitivity, specificity, and accuracy are 0.78,0.98, and 0.95, respectively. For samples with high MR suspicious level,we consistently achieve higher sensitivity result compared to those withmoderate MR suspicious level.533.3. Temporal Analysis of Temporal Enhanced UltrasoundTable 3.4: Model performance for classification of cores in the test data (N= 171).Method Specificity Sensitivity Accuracy AUCLSTM 0.98 0.76 0.93 0.96GRU 0.95 0.70 0.86 0.92Vanilla RNN 0.72 0.69 0.75 0.76Spectral [11] 0.73 0.63 0.78 0.76Table 3.5: Model performance for classification of cores in the test data forModerate MR suspicious level. N indicates the number of cores in eachgroup.MR suspicious levels Moderate MR suspicious level (N = 115)Specificity Sensitivity Accuracy AUCLSTM 0.98 0.78 0.95 0.97GRU 0.95 0.70 0.89 0.91Vanilla RNN 0.82 0.70 0.82 0.73Spectral [11] 0.70 0.78 0.80 0.80Table 3.6: Model performance for classification of cores in the test data forHigh MR suspicious level. N indicates the number of cores in each group.MR suspicious levels High MR suspicious level (N = 20)Specificity Sensitivity Accuracy AUCLSTM 0.86 0.92 0.90 0.97GRU 0.80 1.00 0.90 0.92Vanilla RNN 0.85 0.67 0.70 0.68Spectral [11] 0.83 0.90 0.85 0.953.3.4.3 Comparison with Other MethodsThe best results to-date involving TeUS are based on spectral analysis asproposed in [11] and Section 3.2. We compare our RNN-based approachwith this method as the most related work. For the fair comparison, weimplement the spectral method with the same data division as proposed543.3. Temporal Analysis of Temporal Enhanced Ultrasoundfor RNN-based method. As reported in Table 3.4, Table 3.5 and Table 3.6,the LSTM-RNN models consistently outperform spectral analysis of TeUSwith a two-way paired t-test showing statistically significant improvementin sensitivity, specificity, accuracy, and AUC (p < 0.05) for LSTM-RNN.Datasets used in a few other studies are different from the current workso a detailed comparison is not feasible. However, the proposed networkarchitectures outperform Imani et al. [74], who used cascade CNNs, andUniyal et al. [154], who applied random forests for TeUS based prostatecancer detection, both on data from 14 patients. While analysis of a single RFultrasound frame is not possible in the context of our current clinical study,previously, we have shown that TeUS is complementary to this informationand generally outperforms analysis of single RF frames [36, 108]. The bestresults reported using a single RF frame analysis [42] involve 64 subjectswith an AUC of 0.84, where they used PSA as an additional surrogate.In a recent study, Nahlawi et al. [116, 118] used Hidden Markov Models(HMMs) to model the temporal information of TeUS data for prostate cancerdetection. In a limited clinical study including 14 subjects, they identifiedcancerous regions with an accuracy of 0.85. Furthermore, we qualitativelycompare the cancer likelihood colormap resulting from LSTM-RNN with thespectral analysis of TeUS [11]. Figure 3.10 shows two examples of the cancerlikelihood maps from the test dataset. There is an observable improvementin identifying cancerous regions around the biopsy target that match thegold-standard label.3.3.4.4 Network AnalysisTo analyze the network behavior and identify LSTM cells that contributemost to differentiating between benign and cancerous tissue, we examine thefinal high-level feature representation for cancerous and benign samples. Bygenerating the difference map between the final activation of the network (htat t = 100) for TeUS data from benign and cancerous samples, we identify20 cells with the highest activation difference.In these cells, we observe the evolution of high-level feature representa-tions. Specifically, as per Algorithm 1, input TeUS data from benign andcancerous ROIs in DTtest are forward propagated in the models. Activation ofthe input gate (i(t)), the output gate (o(t)) and the cell state (c(t)) for thetop 20 active cells are studied. We observe cell states c(t) evolve over timeand gradually learn discriminative information. Moreover, the input gate i(t)evolves so that it attenuates parts of the input TeUS sequence and detectsthe important information from the sequence. Interestingly, the input gate553.3. Temporal Analysis of Temporal Enhanced Ultrasound(a) Benign (b) Cancerous, GS â¤ 3+4 (c) Cancerous, GS â¥ 4+3Figure 3.10: Cancer likelihood maps overlaid on B-mode ultrasound images,along with the projected needle path in the TeUS data, and centered onthe target. Red indicates predicted labels as cancer, and blue indicatespredicted benign regions. The boundary of the segmented prostate in MRIis overlaid on TRUS data. The arrow points to the target location. Thetop row shows the result of LSTM and the bottom row shows the result ofspectral analysis [11] for benign targets (a), and cancer targets (b) and (c).reduces the contribution of TeUS data to model learning around time step50, for both cancerous and benign samples. The evolution and attenuationpatterns of c(t) and i(t) suggest that the most discriminative features fordistinguishing cancerous and benign tissue samples are captured within thefirst half of TeUS sequence. These findings match those reported by Nahlawiet al. [116, 118].To further examine this, we evaluate the evolving behavior of LSTM-RNNby training and testing the RNN models with different TeUS sequence lengths.Figure 3.11 shows the performance of the models evaluated by AUC fordifferent TeUS sequence lengths. For each case, using the training procedureexplained in Section 3.3.3.3, we trained an RNN-based deep network with 10-100 RNN cells corresponding to TeUS length. Similar to previous observations,the vanilla RNN-based model has the lowest performance compared to GRUand LSTM based models. By increasing the length of input TeUS sequence,the performance of the models improve. However, for TeUS sequence lengthmore than 50, the improvement saturates. A two-way paired t-test showsthat, for sequence length more than 50, there is no statistically significant563.4. Conclusionimprovement in performance using the LSTM-RNN model (p > 0.05).Figure 3.11: Sequence Length effect: The most discriminative features fordetection of prostate cancer can be learned from a fraction of the full TeUStime series.3.4 ConclusionIn this chapter, we focused on the development of probabilistic models forprostate cancer using temporal and spectral analysis of TeUS data. In thefirst section, we presented an approach for classification of tissue labels ob-tained in MR-TRUS guided targeted prostate biopsy using spectral analysisof TeUS. We utilized a DBN [13] for systematic learning of discriminantlatent features from high-dimensional temporal-ultrasound-features for char-acterizing prostate cancer. We then applied an SVM classifier along withthe activation of the trained DBN to characterize the prostate tissue. In alarge clinical study including 255 TRUS-guided biopsy cores, we identifiedtwo important factors that affect our classification performance: i) distanceof the target to the segmented prostate boundary which is correlated withthe registration error between mp-MRI, TRUS, and histopathology, and ii)disagreement between the axial and sagittal histopathology results. in thissection, we built our classification model using a fixed training dataset con-sisting of TeUS data of 32 biopsy cores and assessed the performance of themodel on the remaining 223 biopsy cores. The test data is divided into threesub-groups according to the distance of the target to the prostate boundaryand agreement between axial and sagittal histopathology labels. For coresfrom targets with moderate MR suspicious level in DBtest, we achieved AUC573.4. Conclusionof 0.80, where mp-MRI has low positive predictive value.In this second part, we utilized deep RNNs to explicitly model the tempo-ral information of TeUS for detecting prostate cancer. The investigation ofseveral RNN structures showed that LSTM-based RNN can efficiently capturetemporal patterns in TeUS data with statistically significant improvementin accuracy over our previously proposed spectral analysis approach [11].We achieved AUC, sensitivity, specificity, and accuracy of 0.96, 0.76, 0.98,and 0.93, respectively. We also presented algorithms for in-depth analysisof high-level latent features of LSTM-based RNN and DBN. A transforma-tional finding, achieved through this analysis, is that the most discriminativefeatures for detection of prostate cancer can be learned from a fraction of thefull TeUS time series. Specifically, in our data, less than 50 ultrasound frameswere required to build models that accurately detect prostate cancer. Thisinformation can be used to optimize TeUS data acquisition for clinical trans-lation. Moreover, our results showed that analysis of temporal ultrasounddata is a promising technology for accurate classification of tissue labels thatwere identified in mp-MRI as suspicious and can potentially complementmp-MRI for TRUS-guided biopsy.58Chapter 4Detection of High-GradeProstate Cancer Using TeUSThe world is noisy and messy. You need to deal with the noise anduncertainty.â Daphne Koller4.1 IntroductionEarly diagnosis, and accurate grading and staging of prostate cancer playsignificant roles in the choice and the success of treatments [108]. Grading ofprostate cancer is established by histopathological analysis of the obtainedcores and staging determines the extent of the disease beyond the prostateitself. Therapeutic aspects of prostate cancer have progressed significantly,over the recent years, for patients with advanced disease [145]. However,men with indolent prostate cancer, who constitute the vast majority ofdiagnosed cases, are over-treated with the traditional options of surgery orThis chapter is partly adapted from the following papers [10, 12, 16]: 1) ShekoofehAzizi, Farhad Imani, Jin Tae Kwak, Amir Tahmasebi, Sheng Xu, Pingkun Yan, JochenKruecker, Baris Turkbey, Peter Choyke, Peter Pinto, Bradford Wood, Parvin Mousavi,and Purang Abolmaesumi. Classifying cancer grades using temporal ultrasound for tran-srectal prostate biopsy. In Medical Image Computing and Computer Assisted Intervention(MICCAI), pages 653â661. Springer, 2016, 2) Shekoofeh Azizi, Sharareh Bayat, PingkunYan, Amir Tahmasebi, Guy Nir, Jin Tae Kwak, Sheng Xu, Storey Wilson, Kenneth AIczkowski, M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean, Peter Pinto, BradfordWood, Purang Abolmaesumi, and Parvin Mousavi. Detection and grading of prostatecancer using temporal enhanced ultrasound: combining deep neural networks and tissuemimicking simulations. MICCAIâ16 Special Issue: International Journal of ComputerAssisted Radiology and Surgery, 12(8):1293â1305, 2017, and 3) Shekoofeh Azizi, PingkunYan, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Learning from noisy labelstatistics: Detecting high grade prostate cancer in ultrasound guided biopsy. In MedicalImage Computing and Computer Assisted Intervention (MICCAI). Springer, 2018594.1. Introductionradiation therapy, leading to a decline in their quality of life. The utility ofactive surveillance for the effective disease management in men with indolentprostate cancer is dependent on accurate assessment of the disease gradeand extent [39, 144, 145]. Using a sensitive imaging modality for tissue-characterization and for biopsy guidance can substantially contribute to theappropriate and adequate treatment of prostate cancer. There have been alarge number of efforts to adopt ultrasound-based tissue characterization forprostate cancer diagnosis during the biopsy procedure. Most contributionsfocus on the analysis of texture [98] and spectral features [42] within a singleultrasound frame. Elastography [33] and Doppler imaging [120] also aimto distinguish different tissue types based on their measured tissue stiffnessand blood circulation, respectively. However, accurate characterization ofaggressive from indolent prostate cancer and grading of the disease are stillopen issues [82].Over the past decade, features extracted from TeUS data have been usedin a machine learning framework to predict labels provided by histopathologyas the ground-truth. As we have also shown in Chapter 3, TeUS has beenused successfully for characterization of cancerous and non-cancerous prostatetissue in ex vivo [108, 109] and in vivo [11, 13, 73, 75, 110] studies. In thesestudies, the area under the receiver operating characteristic curve (AUC) of0.76-0.93 has been reported. TeUS has also been used to distinguish betweenvarious cancer grades in preliminary whole-mount studies [82].In this chapter, we focus on detection of higher grade prostate cancerand the problem of detection of different prostate cancer grades. From themachine point of view, this task can be modeled as a probabilistic multi-classclassification task. However, there are two key challenges with the ground-truth. First, histopathology data used for training of the models is sparselyannotated with the inevitable ubiquity of noise. Second, the heterogeneityin morphology and pathology of the prostate itself contributes as a sourceof inaccuracy in labeling. We try to address these challenges through twoapproaches. First, the learning of statistical distribution of cancer in thebiopsy core using the automatically learned feature in spectral analysis ofTeUS. Second, in a different approach, we embed the prior knowledge fromthe histopathology as the soft labels in a probabilistic model based on theanalysis of temporal aspect of TeUS. This chapter is accordingly subdividedinto two parts.604.2. Prostate Cancer Grading Using Spectral Analysis of TeUS4.2 Prostate Cancer Grading Using SpectralAnalysis of TeUSIn this section, we propose a cancer grading approach for transrectal ultrasound-guided prostate biopsy based on spectral analysis of TeUS signals. Histopatho-logical grading of prostate cancer reports the statistics of cancer distributionin a biopsy core. The approach relies on a coarse-to-fine classification, similarto histopathology reporting, and uses statistical analysis and deep learning todetermine the distribution of aggressive cancer in ultrasound image regionssurrounding a biopsy target. Our approach consists of two steps; in thefirst step, we learn high-level latent features that maximally differentiatebenign from cancerous tissue. In the second step, we model the statisticaldistribution of prostate cancer grades in the space of latent features. In astudy with 197 biopsy cores from 132 subjects, our approach can effectivelyseparate clinically significant disease from low-grade tumors and benigntissue. Further, we achieve the area under the curve of 0.8 for separatingaggressive cancer from benign tissue in large tumors.4.2.1 MaterialsIn this section, we use the data from our first study as we explained inChapter 2 and we focus on the spectral representation of TeUS. Our findingsfrom previous chapter indicate that the agreement between axial and sagittalhistopathology of each target impact the classification accuracy in spectralanalysis of TeUS. Thus, from the whole 255 cores in our dataset, we dividethe data from 197 cores, who have the agreement between axial and sagittalpathology, into the train, DStrain, and test, DStest, sets as explained in Table 4.1.For building the classification model, we need to use homogeneous tissueregions with the known gold standard. Given the potential mis-registrationbetween MR and TRUS images [11], we use biopsy cores with at least 7 mmof cancer for a typical core length of 18 mm to build our model. Trainingdata is made up of 32 biopsy cores from 27 patients with the followinghistopathology labels: 19 benign, 0 GS of 3+3, 5 GS of 3+4, 2 GS of 4+3, 4GS of 4+4 and, 2 GS of 4+5. The test data consists of 165 cores from 114patients, with the following distribution: 121 benign, 12 GS of 3+3, 14 GSof 3+4, 2 GS of 4+3, and 16 GS of 4+4.614.2. Prostate Cancer Grading Using Spectral Analysis of TeUSTable 4.1: Gleason score distribution in TeUS test and train dataset. Tablerepresents the number of cores for each category.Dataset Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 GS 4+5DStrain 19 0 5 2 4 2DStest 121 12 14 2 16 04.2.2 MethodProstate cancer grading can be viewed as a multi-class classification problemwhere the objective is to detect the prostate cancer aggressiveness level(benign or Gleason grades 3, 4 or 5) for a given area in the tissue. Training amulti-class classifier given the histopathology reports of prostate biopsy as theground-truth is non-trivial. Challenges associated with the histopathologylabels for prostate grading are:1. The ground-truth reports a measure of the statistical distribution ofcancer in a biopsy core (e.g., % of cancer in core).2. The exact location of the cancerous tissue in the core is not provided.3. As a result of partial information as explained above, the exact labelof each ROI in a core is not known.Figure 4.8 shows an illustration of these ground-truth issues. The statisticsof ROIs with various labels in a biopsy core are the only known information.In this approach, similar to the pathology reporting, we propose a coarse-to-fine classification approach to find a statistical representation of thedistribution of ROIs in various classes (benign and Gleason grades 3, or 4).Figure 4.1 shows an illustration of the proposed grading method. Thereare two major steps: (1) feature learning to extract latent features thatmaximally separate benign from cancerous tissues; and (2) distributionlearning to model the statistical distribution of prostate cancer grades in thespace of learned features.4.2.2.1 Feature LearningAs we explained in Chapter 3.2, we can use a DBN [13, 22] to automaticallylearn a high-level latent feature representation of the TeUS data that canseparate benign and cancerous lesions. In summary, the network structure624.2. Prostate Cancer Grading Using Spectral Analysis of TeUSTimeTeUS Training DataAxialDBN TrainingWhiteningGMM InitializationDistribution LearningFeature LearningTrained DBNWhiteningGaussian Mixture ModelGleason Score PredictionTimeTeUS Test DataAxialFigure 4.1: An illustration of the proposed cancer grading approach usingspectral analysis of TeUS.includes 100, 50 and 6 hidden units in three consecutive layers, where thelast hidden layer represents the latent features. In the pre-training step,the learning rate is fixed at 0.001, mini-batch size is 5, and the epoch is100. Momentum and weight cost are set to defaults of 0.9, and 2 Ã 10â4,respectively. In the discriminative fine-tuning step, we add a node to representthe labels of observations, and back-propagation with a learning rate of 0.01for 70 epochs and mini-batch size of 10 is used. Due to a limited number ofcancerous cores, we have not used any validation set in this work. In orderto test the generalization of the trained DBN, we make certain that the testdata is never used to pre-train or fine-tune the network parameters. Thetrained network maps the set of 50 spectral components for each ROI to sixhigh-level latent features. Then, we perform dimensionality reduction in thespace of the six latent features. We use Zero-phase Component Analysis [20]to whiten the features and to determine the top two eigen vectors, f1 and f2.We call this space the eigen feature space.634.2. Prostate Cancer Grading Using Spectral Analysis of TeUS4.2.2.2 Distribution LearningTo learn the distribution of different Gleason grades in the eigen featurespace, we use a Gaussian Mixture Model (GMM) [161]. The K-componentGMM is denoted by Î = {(Ïk, µk,Σk)|k = 1, ...,K}, where Ïk is the mixingweight (âKk=1 Ïk = 1), µk is the mean and Σk is the covariance matrix ofthe k-th mixture component. Starting with an initial mixture model, theparameters of Î are estimated with Expectation-Maximization (EM) [161].Since the EM algorithm is a local optimization method, it is particularlysensitive to the initialization of the model. Instead of random initialization,we use the prior knowledge from pathology to devise a simple but efficientmethod for this purpose.Figure 4.2 shows an overview of the proposed GMM initialization tech-nique. Let XH be the set of all ROIs of the cores in training data withhistopathology labels H â{ benign, GS 3+4, GS 4+3, GS 4+4 }. First, wemap the distribution of the ROIs from benign cores, Xbenign, in the eigenfeature space; we observe two distinct clusters that span histopathologylabels of normal and fibromuscular tissue, chronic inflammation, atrophy,and PIN. We use k-means clustering to separate the two clusters and con-sider the cluster with the maximum number of ânormal tissueâ ROIs asthe dominant benign cluster, and the second cluster as a representative forother non-cancerous tissue. Next, we map ROIs in the training dataset thatcorresponds to the cores with GS 4+4, XGS4+4 in the eigen feature space,to identify the dominant cluster that represents Gleason 4 pattern. Finally,we use all other ROIs from cancerous cores that correspond to GS 3+4 andGS 4+3 to determine the cluster for Gleason 3 pattern in the eigen featurespace. We denote the centroid of all clusters by C = {Cbenign, CG4, CG3,Cnoncancerous}. To initialize the K-component GMM, we set K = 4 to modelthe four tissue patterns with mean, µk, for each Gaussian component equalto the centroid of each cluster. We use the equal covariance matrices for allcomponents and set Σk to the covariance of XH . Each Ïk, k = 1, ...,K israndomly drawn from a uniform distribution between [0, 1] and normalizedbyâKk=1 Ïk.4.2.2.3 Prediction of Gleason ScoreTo predict the Gleason score of each core, we map the data from ROIs withinthat core to the eigen feature space (see Fig.4.1). Subsequently, we assigna label from {benign, G3, G4, non-cancerous} to each ROI based on itsproximity to the corresponding cluster center in the eigen feature space. To644.2. Prostate Cancer Grading Using Spectral Analysis of TeUS+ + + +Non-cancerousBenignGleason 3Gleason 4Benign GS 4+4 Benign GS 4+4 GS 3+3 GS 3+4/4+3-3 -2 -1 0 1 2-4-3-2-1012F1F2 BenignGleason 3Gleason 4Non-cancerousCnoncancerousCG4CG3Cbenign-3 -2 -1 0 1 2-4-3-2-1012F1F2 Non-cancerousGleason 4BenignCentroidsCG4CnoncancerousCbenign-3 -2 -1 0 1 2-4-3-2-1012F1F2 Non-cancerousBenignCentroidsCnoncancerousCbenignBenignFigure 4.2: An illustration of the proposed GMM initialization method.determine a GS for a test core, Y, we follow the histopathology guidelineswhere we use the ratio of the number of ROIs labeled as benign, G3 (NG3)and G4 (NG4) (e.g., a core with a large number of G4 and a small numberof G3 ROIs has GS 4+3):Y =ï£±ï£´ï£²ï£´ï£³GS 4+3 or higher, NG4 6= 0 & NG4 â¥ NG3GS 3+4 or lower , NG3 6= 0 & NG4 < NG3benign, otherwiseSince none of the cores labeled by histopathology as GS 3+3 did notmake our selection criteria for training samples, we use cores with GS 3+4and 4+3 to derive the features that describe Gleason grade 3.4.2.3 Results and DiscussionTo assess the performance of our method, we use sensitivity, specificity,and accuracy of the approach in detecting cancerous tissue samples. Weconsider all cancerous cores as the positive class and other non-cancerouscores as the negative class. Sensitivity or recall is the percentage of cancerouscores that are correctly identified while specificity is the proportion of non-cancerous cores that are correctly classified. Accuracy is the ratio of thetrue results (both true positives and true negatives) over the total numberof cores. We also report the overall performance of our approach usingthe area under the receiver operating characteristic curve (AUC). ReceiverOperating Characteristic (ROC) is a two-dimensional graph of sensitivity654.2. Prostate Cancer Grading Using Spectral Analysis of TeUSTable 4.2: Model performance for prostate cancer grading in the test datasetusing TeUS only and by integration of TeUS and mp-MRI. L is the largestlength of the tumor visible in mp-MRI.Method TeUS TeUS + mp-MRIAll cores L â¥ 2.0 cm All cores L â¥ 2.0 cmNon-cancerous vs. GS â¥ 4+3 0.69 0.80 0.82 0.89Non-cancerous vs. GS â¤ 3+4 0.62 0.63 0.67 0.81GS â¤ 3+4 vs. GS â¥ 4+3 0.61 0.67 0.71 0.71versus (1-specificity), depicting the relative trade-offs between sensitivityand specificity. Accuracy can be reported at any particular threshold on thiscurve. The maximum AUC is 1, where larger AUC values indicate betterclassification performance [40].4.2.3.1 Prostate Cancer Detection and GradingTables 4.2 and 4.3 show the classification and grading performance basedon the inter-class AUC, accuracy, sensitivity, and specificity. To investigatethe effect of the size of the tumor on our detection performance, we alsoshow the AUC versus the largest length of the tumor in MRI. This lengthranges from 0.3 cm to 3.8 cm in our dataset. As seen in Table 4.2, incores with MR-tumor-sizeâ¥ 2.0 cm, we obtain AUC of 0.80 in classifyingcores with GS â¥ 4+3 from non-cancerous cores. Moreover, we achieveAUC, accuracy, sensitivity, and specificity of 0.70, 70%, 70%, and 71%,respectively, in the detection of cancerous from non-cancerous cores withMR-tumor-sizeâ¥ 2.0 cm.Figure 4.3 shows examples of the cancer likelihood maps from test dataset,derived from the output of the proposed clustering algorithm. Cancercolormaps overlaid on B-mode ultrasound image, along with the projectedneedle path in the TeUS data and centered on the target. The red boundaryshows the segmented prostate in MRI projected in TRUS coordinates. Inthe colormaps, red and yellow regions show 0.5Ã 0.5 mmÃmm ROIs whichwe detect as Gleason grades 4 and 3, respectively. The blue areas belong tonon-cancerous ROIs.Figure 4.3(b) shows a case where the axial and sagittal histopathologyresults do not agree (see Section 3.2). The axial pathology indicates GS 3+4whereas the sagittal pathology reports the core as GS 4+4. The colormapdemonstrates using our approach, the clinician could have reoriented the664.2. Prostate Cancer Grading Using Spectral Analysis of TeUS(a) MRI lesion length = 27 mm, benigntarget(b) MRI lesion length = 36 mm, GS â¤3 + 4(c) MRI lesion length = 24 mm, GS â¤3 + 4(d) MRI lesion length = 17 mm, GS â¥4 + 3Figure 4.3: Cancer likelihood maps overlaid on B-mode US image, alongwith the projected needle path in the TeUS data and centered on the target.The ROIs for which we detect as Gleason grade of 4 and 3 are colored inred and yellow, respectively. The non-cancerous ROIs are colored as blue.The red boundary shows the segmented prostate in MRI projected in TRUScoordinates and the arrow pointer shows the target.needle to biopsy a more aggressive region.4.2.3.2 Integration of TeUS and mp-MRITo take advantage of mp-MRI information, we combine the TeUS cancerdetection results with readings from mp-MRI suspicious levels. If mp-MRIdeclares cancer suspicious level as low or high for a core, we use that predic-tion alone and report the core as benign or aggressive cancer, respectively.However, when mp-MRI declares the suspicious level as intermediate (70%of all cores in our data), we use predictions based on TeUS. For tumors withL â¥ 2.0 cm, the integrated approach leads to an AUC of 0.76 for predictingcancer grades compared that of 0.65 using mp-MRI alone, and 0.70 usingTeUS data alone. Moreover, for classification of cores with GS â¥ 4+3 fromnon-cancerous cores, the combined AUC is 0.82. The results indicate both674.2. Prostate Cancer Grading Using Spectral Analysis of TeUSTable 4.3: Model performance for classification of cancerous vs. non-cancerous cores in the test dataset using TeUS only and Integration ofTeUS and mp-MRI. L is the greatest length of the tumor visible in mp-MRI.Evaluation TeUS TeUS + mp-MRIAll cores L â¥ 2.0 cm All cores L â¥ 2.0 cmAccuracy 65% 70% 68% 70%Sensitivity 62% 70% 63% 70%Specificity 67% 71% 67% 72%AUC 0.65 0.70 0.72 0.76TeUS and integration of TeUS with mp-MRI have higher performance forlarger tumors.4.2.3.3 Sensitivity AnalysisPerformance in Anatomical Zones: Figure 4.4 (top) summarizes thebiopsy target locations, distribution, and histopathology outcomes for thetest data. The prostate region is divided into anterior/posterior, and cen-tral/peripheral zones for the base, midgland, and apex regions. In our testdata, 34% (19 out of 56 biopsies) of all cancerous cores were in the centralregion where 24% (25 out of 109 biopsies) were in the peripheral region. Fig-ure 4.4 (middle) shows the predictions of prostate cancer grades using TeUSin different anatomical zones. Although more biopsies were performed in theperipheral zone, a higher portion of positive biopsies was observed in thecentral zone. In the central zone, we can differentiate between non-canceroustargets and clinically significant cancer (GS â¥ 4 + 3) with an AUC of 0.80.The bottom row depicts prostate cancer grading performance integratingTeUS and MRI information.Choice of the Training Data: We also analyzed the sensitivity of ourapproach to the choice of training data. To create new training and testsets we permute our original data where, in each permutation, we exchangea randomly selected cancerous or benign core in the primary training andtest data. This results in 32 different data divisions. As Fig. 4.5 shows, theaverage AUC of the sensitivity analysis follows our previous performanceresults, which supports the generalization of the proposed model.Size of the Training Data: We also investigate the effect of training data684.2. Prostate Cancer Grading Using Spectral Analysis of TeUS(a) Base (b) Mid-gland (c) ApexFigure 4.4: Target location and distribution of biopsies in the test data.Light and dark gray indicate central and peripheral zones, respectively. Thepie charts show the number of cores and their histopathology. The size ofthe chart is proportional to the number of biopsies (in the range from 1 to25), and the colors dark red, light red and blue refer to cores with GSâ¥ 4 + 3,GSâ¤ 3 + 4 and benign pathology, respectively. The top, middle, and bottomrows depict histopathology results, TeUS prediction, and integration of TeUSand MRI, respectively.size on the grading performance by gradually increasing the size of the datasetfrom 16 to 56 cores. When choosing new training samples to add, we usebiopsy cores with at least 4.0 mm of cancer for a typical core length of 18 mm(given the potential mis-registration between MR and TRUS images [11] it isa prudent step). Otherwise, we randomly select an equal number of benign694.2. Prostate Cancer Grading Using Spectral Analysis of TeUSFigure 4.5: Model performance for prostate cancer grading using spectralanalysis of TeUS and distribution learning in the test dataset and permutationset.Figure 4.6: Model performance for different sizes of training dataset usingspectral analysis of TeUS and distribution learning.and cancerous cores. Figure 4.6 shows the AUC, of differentiation betweenprostate cancer and non-cancerous tissue, versus the size of the dataset. Ingeneral, there is an increasing trend for the AUC and our approach has ahigher performance using more training samples.Number of Features: It has been shown that two features (f1 and f2) arevery effective in classification. We further analyze the effect of including anyof the six features in the model over the final results. Figure 10 shows the704.3. Temporal Analysis of TeUS for prostate cancer grading(a) (b)Figure 4.7: Model performance versus the number of features that we usedto generate the final model: (a) For all of the cores; (b) for cores withMR-tumor-sizeâ¥ 2.0 cm. Decreasing the number of features improves themodel performance.AUC of differentiation between cancerous and non-cancerous tissues, versusthe number of features that we used to generate the final GMM model. Toinvestigate the effect of the size of the tumor on our detection performance,we also show the AUC versus the largest length of the tumor in MRI. Ingeneral, decreasing the number of features used for model generation increasesthe model performance. In particular, as seen in Fig. 4.7(b) in cores withMR-tumor-sizeâ¥ 2.0 cm, this trend is more clear. One possible explanationis that increasing the number of features leads to more complexity, whichcan not be learned using a small number of samples in each category.4.3 Temporal Analysis of TeUS for prostatecancer gradingDespite promising results in detecting high grade prostate cancer usingspectral analysis of TeUS and distribution learning, accurate characterizationof aggressive lesions from indolent ones still requires refinement. We need tohave a more precise grading methods based on temporal analysis of TeUSwhich can enable real-time depiction of cancer likelihood map during thebiopsy procedure. In addition, as we discussed in the past sections, thegoodness of models built based on all the above analyses depends on detailed,noise-free annotations of ground-truth labels from pathology. Histopathologydata used for training of the computer-aided diagnosis models are sparsely714.3. Temporal Analysis of TeUS for prostate cancer gradingBenign GS 3+3 GS 3+4 GS 4+3 GS 4+4[0, 0, 0] [0, 1, 0] [0, 1, 1] [0, 1, 1] [0, 0, 1] [Benign G3 G4]Figure 4.8: Illustration of noisy and not finely annotated ground-truth label.The exact location of the cancerous ROI in the core, the ratio of the differentGleason grade, and the exact location of the Gleason grades are unknown andnoisy. The bottom vectors show one of the possible multi-label binarizationapproaches.annotated with the inevitable ubiquity of noise.As we explained in previous sections, histopathology reports include thelength of cancer in the biopsy core and a Gleason Score (GS) [120]. The GSis reported as a summation of the Gleason grades of the two most commoncancer patterns in the tissue specimen. Gleason grades range from 1 (normaltissue) to 5 (aggressive cancerous tissue). The histopathology reports ameasure of the statistical distribution of cancer in the cancer foci. Theground-truth is noisy and not finely annotated to show the exact locationof the cancerous tissue in the core (Fig. 4.8). Therefore, the exact grade ofeach ROI in a core is not available while the overarching goal is to determinethe ROI-level grade of the specimen.In this section, we propose a method to directly alleviate the challengeof sparse and noisy histopathology ground-truth labels to improve TeUS-based prostate biopsy guidance. Specifically, we embed the prior knowledgefrom the histopathology as the soft labels in a two-stage model, to leveragethe problem of diverse label noise in the ground-truth. We then use thisinformation to accurately detect the grade of cancer and also to estimate thelength of cancer in the target. Additionally, we create a Bayesian probabilisticversion of our network, which allows evaluation of model uncertainty thatcan lead to any possible misguidance during the biopsy procedure. In an invivo study with 155 patients, we analyze data from 250 suspicious cancer fociobtained during fusion biopsy. We achieve the average area under the curve724.3. Temporal Analysis of TeUS for prostate cancer gradingof 0.84 for cancer grading and mean squared error of 0.12 in the estimationof tumor in biopsy core length.4.3.1 Method4.3.1.1 Discriminative ModelLet DT = {(x(i),y(i))}|DT |i=1 represent the collection of all ROIs, where x(i) isthe ith TeUS sequence with length T and is labeled as y(i) correspondingto a cancer grade. The objective is to develop a probabilistic model todiscriminate between cancer grades using noisy and not well-annotated datain DT . For this purpose, we propose a two-stage approach to consider thediverse nature of noise in the ground-truth labeling: benign vs. all grades ofcancer and the mixture of cancer grades. The goal of the first stage is to minethe data points with non-cancerous tissue in the presence of possible noisewhere several theoretical studies have shown the robustness of the binaryclassification accuracy to the simple and symmetric label noise [47]. Thegoal of the second stage is to learn from the noisy label statistic in cancerouscores by suppressing the influence of noise using a soft label (Fig. 4.9). Inthe heart of the approach, we use deeply connected RNN layers to explicitlymodel the temporal information in TeUS followed by a fully connected layerto map the sequence to a posterior over classes. Each RNN layer includesT = 100 homogeneous hidden units (i.e., traditional/vanilla RNN, LSTM orGRU cells) to capture temporal changes in data. Given the input sequencex = (x1, , ..., xT ), RNN computes the hidden vector sequence h = (h1, , ..., hT )in the sequence learning step. This hidden vector, h is a function of theinput sequence x, model parameters, Î, and time.Stage 1: Detection of Benign Samples: Let y(i)b â {0, 1} indicate thecorresponding binary label for x(i), where zero and one indicate benign andcancer outcome, respectively. We aim to learn a mapping from x(i) to y(i)bin a supervised manner. After the sequence learning step, the final nodegenerates the posterior probability for the given sequence:y(i)b = arg maxjS(z(i)j ), j â {0, 1}, z(i) = WTs h + bs , (4.1)where Ws and bs are the weight and bias of the fully-connected layer and Sis the softmax function, which in our binary classification case is equivalent tothe logistic function, and y(i)b indicates the predicted label. The optimizationcriterion for the network is to minimize the binary cross-entropy between734.3. Temporal Analysis of TeUS for prostate cancer gradingcancerT Framesx1x2xTh1h2hTTeUSDense(GS, Lencancer)Sequence LearningMeanground-truth probability vectorCostk-way SoftmaxDcancertrainPcancerFigure 4.9: Overview of the second stage in the proposed method: the goalof this stage is to assign a pathological score to a sample. To mitigate theproblem of imperfect and noisy labels, we embed the length of cancer in theground-truth probability vector as a soft label.y(i)b and y(i)b over all training samples.Stage 2: Grading of Cancerous Samples: The goal of this stage is toassign a pathological score to Dcancertrain = {(x(i), y(i)b ) â DTtrain | y(i)b = 1}Ni=1.Here, unlike the first stage, we are facing a multi-label classification taskwith sparse labeling (Fig. 4.8). The histopathology reports include twoinformative parts:1. Gleason score which implies any of the possible labels for all ROIswithin a core, â¦ â {Benign,G3, G4}. In this representation, all or atleast two of these patterns can happen at the same time (Fig. 4.8). Aswe explained before, we can also interpret a give Gleason score as ameasure distribution of each grade in a core.2. In our dataset, we also have access to the measured length of canceroustissues (Len) in a typical core length (Lentypical) of 18.0 mm.We propose a new approach for ground-truth probability vector gener-ation, enabling the soft labeling instead of the traditional label encoding744.3. Temporal Analysis of TeUS for prostate cancer gradingmethods. For this purpose, using Dcancertrain the output of sequence learningstep h is fed into a k-way softmax function, which produces a probabilitydistribution over the k possible class labels (k = 3). Suppose Len(i) repre-sents the length of cancer for the core that x(i) belongs to. The ground-truthprobability vector of the ith ROI is defined as p(i) = [p(i)1 , ..., p(i)k ]. To es-timate these probabilities we define the normalized cancer percentage asC(i) = Len(i)/Lentypical (C(i) â [0, 1]). For k = 3:p(i) =[p(i)1 = (1âC(i)), p(i)2 = Ï ÃC(i), p(i)3 = (1â Ï)ÃC(i)], (4.2)where Ï is the cancer regularization factor to control the inherent ratiobetween pattern G3 and G4 in a way that for the cores with GS 3+4 label,Ï be greater than 0.5 to imply a higher probability of having pattern G3than the G4 and vice-versa. For ROIs which originate from the cores withGS 3+3 or GS 4+4 readings, Ï is set to 1 and 0, respectively. Then, the costfunction to be minimized is defined as:J =1|Dcancertrain |Nâi=1Kâk=1(p(i)k â p(i)k )2 , (4.3)where p(i) = [p(i)1 , ..., p(i)k ] is the predictive probability vector.4.3.1.2 Cancer Grading and Tumor in Core Length EstimationSuppose CT = {(x(i), y(i)b )}|CT |i=1 represent the collection of all labeled ROIssurrounding a target core, where CT â DTtest, |CT | = 80, x(i) represents the ithTeUS sequence of the core, and y(i)b indicates the corresponding binary label.Using the probability output of the first stage model for each ROI, we assigna binary label to each target core. The label is calculated using a majorityvoting based on the predicted labels of all ROIs surrounding the target. Wedefine the predicted label for each ROI, yb(i), as 1, when P (y(i)b |x(i)) â¥ 0.5,and as 0 otherwise. The probability of a given core being cancerous basedon the cancerous ROIs within that core is Pb =â|CT |i=1 I(y(i) = 1)/|CT |. Abinary label of 1 is assigned to a core, when Pb â¥ 0.5. For the cores withprediction of the cancer, we use the output the second stage model to bothpredict the cancer length and determine a GS for the test core. Supposep(i)m =[p(i)1 , p(i)2 , p(i)3]represents the predictive probability output of ith TeUS754.3. Temporal Analysis of TeUS for prostate cancer gradingsequence in the second stage. We define the average predictive probabilityas:Pm = 1|CT ||CT |âi=1p(i)m . , (4.4)Following the histopathology guidelines, to determine a GS for a canceroustest core, Y, we define the core as âGS 4+3 or higherâ when P(3)m â¥ P(2)mand otherwise as âGS 3+4 or lowerâ. Furthermore, based on Equation (4.4)and 4.2, we can estimate the predicted length of cancer for this core as:LenC = (1âPm(1))Ã Lentypical , (4.5)4.3.1.3 Model Uncertainty EstimationWe also aim to estimate the model uncertainty in the detection of cancer forthe areas outside the cancer foci, where the annotation is not available. Thekey for estimating model uncertainty is the posterior distribution P (Î|D),also referred to a Bayesian inference [48]. Here, we follow the idea in [48] toapproximate model uncertainty using Monte Carlo dropout (MC dropout).Given a new input x(i), we compute the model output with stochasticdropouts at each layer. That is, randomly dropout each hidden unit withcertain probability p. This procedure is repeated B times, and we obtain{yâ(1)b , ..., yâ(B)b }. Then, the model uncertainty can be approximated by thesample variance:1BBâj=1(yâ(j)b â yËâ(j)b )2 , (4.6)where yËâ(j)b is the average of yâ(j)b values.4.3.2 Experiments and Results4.3.2.1 Data Division and Model SelectionData is divided into mutually exclusive patient sets for training, DTtrain, andtest, DTtest. Training data is made up of 80 randomly selected cores frompatients with homogeneous tissue regions where the number of cancerous andnon-cancerous cores are equal. The test data consists of 170 cores, where 130cores are labeled as benign, 29 cores with GS â¤ 3+4, and 12 cores with GS â¥4+3. Given the data augmentation strategy in Section 2.3, we obtain a total764.3. Temporal Analysis of TeUS for prostate cancer gradingTable 4.4: Model performance for classification of cores in the test data (N= 170). AUC1, AUC2 and AUC3 refer to detection of Benign vs. GSâ¤3+4,Benign vs. GSâ¥4+3, and GSâ¤3+4 vs. GSâ¥4+3, respectively.Method AUC1 AUC2 AUC3 Average AUCLSTM 0.96 0.96 0.86 0.93GRU 0.92 0.92 0.84 0.89Vanilla RNN 0.76 0.76 0.70 0.74BL-1 0.96 0.96 0.68 0.86BL-2 0.75 0.68 0.58 0.67BL-3 0.82 0.84 0.65 0.77LSTM + GMM-Clustering 0.60 0.74 0.69 0.68DBN + GMM-Clustering 0.68 0.62 0.60 0.63number of 80Ã1,536 = 122,880 training samples (N = |DTtrain| = 122, 880).We use 20% of Dtrain data as the held-out validation sets (DTval) to performthe grid search over the the number of RNN hidden layers, nh â {1, 2}, batchsize, bs â {64, 128}, and initial learning rate, lr â {0.01â0.0001}, and cancerregularization factor, Ï, with three different optimization algorithms, SGD,RMSprop and Adam. Results from hyper-parameter search demonstratethat network structures with two RNN hidden layers outperform otherarchitectures. Furthermore, for the vanilla RNN, bs = 128, lr = 0.0001; forLSTM, bs = 64, lr = 0.0001; and for GRU, bs = 128, lr = 0.01 generate theoptimum models. For all models, dr = 0.2, lreg = 0.0001 generate the lowestloss and the highest accuracy in Dval. Also, Ï = 0.7 for GS 3+4 and Ï = 0.3for GS 4+3 result in the highest performance. After model selection, we usethe whole DTtrain for training a model for the first stage and Dcancertrain for thesecond stage model.4.3.2.2 Comparative Method and BaselinesWe use standard evaluation metrics as prior approaches [10, 11] to quantifyour results. We assess the inter-class area under the receiver operatingcharacteristic curve (AUC) for detection of Benign vs. GSâ¤3+4 (AUC1),Benign vs. GSâ¥4+3 (AUC2), and GSâ¤3+4 vs. GSâ¥4+3 (AUC3). Table 4.4shows the performance comparison between the proposed approach and thefollowing baselines. To substantiate the proposed soft ground-truth labelin the second stage of our approach, we replace p(i) with the labels frommulti-label binarization as shown in Fig. 4.8 (BL-1). Also, to justify thenecessity of a two-stage approach to tackle the noise, we use the labels from774.3. Temporal Analysis of TeUS for prostate cancer gradingmulti-label binarization (BL-2) and the weighted version (BL-3) in a singlestage approach; after the sequence learning step we feed the output to a fully-connected layer with a 3-way softmax function. To generate the weightedversion of multi-label binarization labels, for GS 3+4, the encoded vector isdefined as [0, 0.7, 0.3], and for GS 4+3 the encoded vector is [0, 0.3, 0.7]. Wehave also implemented the GMM-clustering method proposed in previoussection for the current data division [11]. We have used the learned featurevector from Deep Belied Network (DBN) method [11] and our best RNNstructure (LSTM) to feed the proposed GMM-clustering method. The resultssuggest that the proposed strategy using both LSTM and GRU cells canlead to a statistically significant improvement in the performance (p < 0.05),which is mainly due to a superior performance of our proposed approach inthe separation of GSâ¤3+4 from GSâ¥4+3. It is worthwhile mentioning thatcore-based approaches like multi-instance learning and traditional multi-classclassification are not feasible due to the small number of samples. Also, inthe lack of a more clean and reliable dataset, direct modeling of the noiselevel is not pragmatic [47].4.3.2.3 Tumor in Core Length EstimationFigure 4.10 shows the scatter plot of the reported tumor in core length inhistopathology vs. the predicted tumor in core length using LSTM cells.The graph shows the correlation between the prediction and histopathologyreport (correlation coefficient=0.95). We also calculate the Mean SquaredError (MSE) as the measure of our performance in cancer length estimationwhere we achieve MSE of 0.12 in the estimation of tumor length.4.3.2.4 Cancer Likelihood ColormapsFigure 4.11(a) shows an example of a cancer likelihood map for biopsyguidance derived from the output of the proposed two-stage approach. Fig-ure 4.11(b) shows the corresponding estimated uncertainty map generatedfrom the proposed uncertainty estimation method (p = 0.5, B = 100). Un-certainty is measured as the sample variance for each ROI and normalized tothe whole prostate region uncertainty. The level of uncertainty is color-codedusing a blue-red spectrum where the blue shows a low level of uncertaintyand the dark red indicates the highest level of uncertainty. The uncertaintycolormap along with the cancer likelihood map can be used as an effectivestrategy to harness the possible misguidance during the biopsy.784.4. ConclusionFigure 4.10: Scatter plot of the reported tumor in core length in histopathol-ogy vs. the predicted tumor in core length.4.4 ConclusionDetermining the aggressiveness of prostate cancer can help reduce the cur-rent high rate of over-treatment in patients with indolent cancer. In thischapter, we proposed methods for detection of higher grade prostate cancerusing spectral and temporal analysis of TeUS data. In the first part, usingthe data from an in vivo TRUS-guided prostate biopsy study, temporalenhanced ultrasound data was used to differentiate between clinically lesssignificant prostate cancer (GSâ¤3+4), aggressive prostate (GSâ¥4+3) andnon-cancerous prostate tissues. We utilized an unsupervised distributionlearning approach to address the challenges related to ground-truth labelingin prostate cancer grading. First, we learned the differentiating featuresfor detection of cancerous and non-cancerous prostate tissue, and then thestatistical distribution of prostate cancer grades was modeled using a GMM.We showed that we could successfully differentiate among aggressive prostatecancer (GSâ¥4+3), clinically less significant prostate cancer (GSâ¤3+4), andnon-cancerous prostate tissues. Furthermore, the combination of temporalenhanced ultrasound and mp-MRI had shown the potential to outperformeither modality alone in the detection of prostate cancer. An AUC of 0.8was achieved for separation of aggressive prostate cancer from non-canceroustissue for tumors that were larger than 2.0 cm in their greatest dimensionin MRI. Additionally, suspicious levels from MRI, when added to TeUSinformation, led to an AUC of 0.89 for classification of aggressive prostatecancer from non-cancerous tissue for tumors larger than 2.0 cm in MRI.794.4. Conclusion(a) Cancer likelihood map (b) Corresponding uncertainty mapFigure 4.11: (a) Cancer likelihood maps overlaid on B-mode US image, alongwith the projected needle path in the TeUS data (GS â¥ 4 + 3) and centeredon the target. The ROIs of size 0.5Ã 0.5 mmÃmm for which we detect theGleason grade of 4 and 3 are colored in red and yellow, respectively. Thenon-cancerous ROIs are colored as blue. (b) The red boundary shows thesegmented prostate in MRI projected in TRUS coordinates and the arrowpointer shows the target.[blue=low uncertainty, red=high uncertainty]In the second part, we tried to address the problem of sparse and noisyhistopathology-based ground-truth labels by employing the ground-truthprobability vectors as soft labels. These soft labels were estimated byembedding the prior histopathology knowledge about the length of cancerin our two-stage model. The results suggested that soft labels can help thelearning process by suppressing the influence of noisy labels and can be usedto accurately estimate the length of the suspicious cancer foci. Furthermore,possible misguidance in biopsy is highlighted by the proposed uncertaintymeasure. Future work will be focused on the analysis of the source of theuncertainty and integrate the proper solution in the framework.80Chapter 5Decision Support System forProstate Biopsy GuidanceYouâre either part of the solution or youâre part of the problem.â Eldridge Cleaver5.1 IntroductionConventional ultrasound images, referred to as Brightness-mode (B-mode),are generated following envelope-detection of back-scattered radio frequency(RF) signals from tissue. B-mode gray-scale images are subjected to variousnon-linear processing steps, which are used to improve the visualization ofthe displayed image for assessment by physicians [142]. RF signals containinformation about tissue microstructure at spatial scales much smaller thanthe conventional B-mode imaging resolution [127, 142]. Therefore, froma data processing perspective, using back-scattered RF signals with richerinformation content for tissue characterization can positively affect diagnosticdecisions [36, 42, 112, 127]. We have shown that, TeUS RF can be usedeffectively for characterization of prostate cancer. In the in vivo studiesdiscussed in Chapter 3 and Chapter 4, we have achieved the areas underreceiver operating characteristic curve (AUC) of 0.8-0.96 for grading andThis chapter is partly adapted from the following papers [14, 15]: 1) Shekoofeh Azizi,Parvin Mousavi, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey,Peter Choyke, Peter Pinto, Bradford Wood, and Purang Abolmaesumi. Transfer learningfrom RF to B-mode temporal enhanced ultrasound features for prostate cancer detection.International Journal of Computer Assisted Radiology and Surgery, 12(7):1111â1121, 2017,and 2) Shekoofeh Azizi, Nathan Van Woudenberg, Samira Sojoudi, Ming Li, Sheng Xu,Emran M Abu Anas, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, Baris Turkbey, PeterChoyke, Peter Pinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Towarda real-time system for temporal enhanced ultrasound-guided prostate biopsy. InternationalJournal of Computer Assisted Radiology and Surgery, pages 1â9, 2018.815.1. Introductiondiagnosis of prostate cancer [73, 75] using spectral and temporal analysis ofTeUS. Despite the promising results of TeUS for prostate cancer diagnosis,development and deployment of TeUS as well as all other ultrasound RF-based methods for computer-aided diagnosis deal with a common challenge:RF data is not available on all commercial ultrasound scanners and is usuallyonly provided for research purposes. To address this challenge, we aim toidentify an intermediate method that enables the use of B-mode data fromconventional scanners for tissue characterization with TeUS.B-mode images are a result of multiple non-linear processing steps ap-plied to ultrasound RF signal including demodulation, non-linear intensitymapping, compression and filtering, which are specific to every scanner andare usually not disclosed. The lack of detailed information about the process-ing pipeline makes RF reconstruction from B-mode data almost impossible.Previous research has focused on the estimation of the parameters of com-pression and other non-linear operations to revert B-mode image generationand reconstruct RF data [142]. However, this approach cannot be easilyadopted for clinical applications as results need to be generated in real-timefor different scanner types and various settings [142].To overcome the challenge of accessibility of RF data, in the TeUS machinelearning framework, we propose to use a transfer learning method [131] totransfer knowledge between TeUS RF and B-mode data within an ultrasoundscanner. Transfer learning is a method that can compensate for the diversitybetween datasets by learning from a source dataset (RF time series data)and apply the knowledge to a target dataset (B-mode time series data).This approach exploits the common information between the two domains.Common elements between two data domains can be features of the data orparameters of the models that are built using the data [131]. Transfer learninghas been applied to integrate data from multiple centers for characterization ofplaque in carotid arteries [156], for protocol-invariant models for segmentationof brain MRI images [155], and for MRI data harmonization [105]. Transferlearning combined with deep learning has shown tremendous potential incomputer vision as well [135]. In case of limited training examples, onecan leverage pre-trained deep networks from alternate large scale data (e.g.,ImageNet [87]) in a transfer learning framework to extract features andemploy them for the task at hand. Transfer learning has also been usedfor computer-aided detection, where information from images of naturalscenes has been successfully transferred for detection of lung lesions [143].Deep learning methods, such as Deep Belief Networks (DBN) and RecurrentNeural Networks (RNNs), have an essential characteristic that makes themwell suited for transfer learning: they can identify abstract features that825.2. Transfer Learning From TeUS RF to B-mode Spectral Featuresperform well generalizing across various domains [21, 53].In the search of an accurate and practical technique for prostate cancerdiagnosis with TeUS in clinical setting, in this chapter we focus on twotechnical challenges: first, the accessibility of raw RF data in commercialscanners and second, the real-time processing of both TeUS RF and B-mode ultrasound data using temporal analysis of TeUS with RNN. Thischapter is subdivided into two parts aiming to address these challenges.In the first section, in a preliminary attempt, we present a method forprostate cancer detection using spectral analysis of TeUS data obtainedeither from RF ultrasound signals or B-mode images. For the first time,we demonstrate that by applying domain adaptation and transfer learningmethods, a tissue classification model trained on TeUS RF data (sourcedomain) can be deployed for classification using TeUS B-mode data alone(target domain) where both data are obtained on the same ultrasound scanner.Towards a real-time system for temporal enhanced ultrasound-guided prostatebiopsy, in the second part, we propose a RNN-based transfer learning solutionwhich is solely work with TeUS B-mode data streaming from an ultrasoundscanner. We demonstrate the efficacy of the framework in two case studies.We show that prostate cancer detection using near real-time analysis of RFand B-mode TeUS data and deep learning is feasible.5.2 Transfer Learning From TeUS RF to B-modeSpectral FeaturesIn this section, in a feasibility study, we present a method for prostatecancer detection using ultrasound TeUS data obtained either from RadioFrequency (RF) ultrasound signals or B-mode images. we demonstrate thatby applying domain adaptation and transfer learning methods, a tissue clas-sification model trained on TeUS RF data (source domain) can be deployedfor classification using TeUS B-mode data alone (target domain), where bothdata are obtained on the same ultrasound scanner. This is a critical step forclinical translation of tissue classification techniques that primarily rely onaccessing RF data since this imaging modality is not readily available on allcommercial scanners in clinics. The proof-of-concept is provided for in vivocharacterization of prostate cancer using TeUS B-mode data, where differentnon-linear processing filters in the pipeline of the RF to B-mode conversionresult in a distribution shift between the two domains. Our in vivo studyincludes data obtained in MRI-guided targeted procedure for prostate biopsyfrom 84 subjects. We achieve a comparable area under the curve using TeUS835.2. Transfer Learning From TeUS RF to B-mode Spectral FeaturesRF and B-mode data for medium to large cancer tumor sizes in biopsy cores(> 4 mm). Our result suggests that the proposed adaptation technique issuccessful in reducing the divergence between TeUS RF and B-mode data.5.2.1 MaterialsIn this section, we use both annotated TeUS data from regions around thetarget location and also un-annotated TeUS data from the whole biopsy scanand inside the prostate boundary in ultrasound scan.5.2.1.1 Unlabeled DataAs we mentioned in Section 2.3.2, in spectral representation of TeUS, eachROI is represented by F = 50 spectral features over the positive frequencyrange of 0 â 10 Hz. Unlike our typical biopsy region analysis, here, the RFspectral features from the ROIs of the entire imaging plane of the prostateand in scan level make up the unlabeled source dataset, DSUS , while thecorresponding B-mode spectral features make up the unlabeled target dataset,DSUT , where DSU = DSUT âª DSUS . Both DSUS and DSUT have an equal numberof samples. In total, DSU includes 5,255,680 unlabeled training samples,consisting of 2,627,840 RF and 2,627,840 B-mode time series data.5.2.1.2 Labeled DataFor building the classification model, we randomly select an equal number ofbenign and cancerous cores, from the samples with at least 4.0 mm of cancerfor a typical core length of 18 mm. Our labeled dataset consists of 84 biopsycores with the following histopathology label distribution: Benign: 42 cores;GS 3+3: 2; GS 3+4: 14; GS 4+3: 3; GS 4+4: 18; and, GS 4+5: 5 cores.Similar to the unlabeled dataset, we extract spectral features from RF andB-mode time series of each ROI. The labeled source dataset, DSLS , includesRF spectral features of ROIs selected from the above 84 cores and the targetdataset DSLT , includes the corresponding B-mode spectral features, whereDSL = DSLT âª DSLS . DSL includes 2Ã6,720 labeled training samples, whereboth DSLS and DSLT datasets have an equal number of samples of 6,720.5.2.2 MethodsAs discussed above, RF data is not readily accessible in all commercialultrasound scanners. To potentially enable large-scale clinical deployment ofTeUS-based tissue typing, we propose to use B-mode time series data (instead845.2. Transfer Learning From TeUS RF to B-mode Spectral Features. . .. . .. . .Trained Deep Belief Network (DBN)PretrainingFine-tuning with SGD. . .. . .. . .Feature ExtractionFeature ExtractionRF time series: DLSB-mode time series: DLTPCA WhiteningFeature ExtractionFeature ExtractionRF time series: DUSB-mode time series: DUTPCA Whitening20 ROIsTarget Shared SVM ModelGround-truth LabelsHistophatologyUnsupervised Domain AdaptationSupervised ClassificationDivergence reduction between two distribution Benign vs. Cancer?Figure 5.1: An illustration of the proposed approach for domain adaptationbetween RF and B-mode time series data in TeUS framework.of RF time series) along with a domain adaptation method. Figure 5.1 showsa block diagram of the solution. Our approach consists of two major steps:1. Unsupervised learning of a common latent feature space using DSU in aDBN training framework for the spectral analysis of TeUS.2. Supervised learning of a shared support vector machine (SVM) classifierusing the labeled dataset, DSL for tissue typing.We will discuss in detail these two steps and the intuition behind them inthe following sections.855.2. Transfer Learning From TeUS RF to B-mode Spectral Features5.2.2.1 Unsupervised Domain AdaptationIn the proposed transfer learning setting, both the source (RF time series)and the target (B-mode time series) data are represented as spectral features.However, the distributions of the data in these two domains are different.We assume that, due to the intrinsic similarity of RF and B-mode time seriesdomains, it is possible to derive a common feature representation to minimizedomain divergence.For this purpose, we start our unsupervised domain adaptation witha layer of Principal Component Analysis (PCA) to whiten the spectralfeatures and align the source and target PCA subspaces. PCA providesa projection of each input vector to a low-dimensional coordinate vector,implicitly defining a low-dimensional hyperplane in input space near whichinformation density is likely to be concentrated. We keep d = 11 top eigenvectors that retrain 99% of the variance of both TeUS RF and B-mode datain DSU â RF=50 [21, 131]. Since, even after PCA projection, we keep a fairlylarge number of eigen directions, the main effect of this step is to smooth theinput shared distribution by eliminating some of the variations involved inthe least globally varying directions. Furthermore, by using whitening alongwith PCA projection, features are normalized to a unit variance [21] whichis a key step in a successful learning process. Following the PCA whiteningstep, we train a shared DBN using both source and target data in the PCAsubspace to minimize the domain divergence. The proposed DBN networkstructure includes two cascaded Restricted Boltzmann Machines (RBM) withd = 11 visible input units and the same number of hidden units in the lastlayer which represent the latent common features space (see Figure 5.1).Given the two domains, DSUS and DSUT , there are two factors to be takeninto consideration for unsupervised common representation learning betweenthe source and target data:- The first is to minimize the reconstruction error for both source andtarget domain data, as we explained in Section 3.2.1.2.- The second is to minimize the divergence between the source and targetdistributions. For this purpose as recommended in literature [21], weuse the KullbacâLeibler (KL) divergence as the measure of divergence.We start with unsupervised pre-training of DBN with the greedy layer-wise procedure where the goal is to minimize the reconstruction error for theinput data, and each layer is trained as an RBM by contrastive divergence [62].In the next step, we fine-tune the DBN using Stochastic Gradient Descent(SGD), where the goal is to minimize:865.2. Transfer Learning From TeUS RF to B-mode Spectral Features- The cross-entropy between DSUT and the output of the shared networkfor DSUS .- The cross-entropy between DSUS and the output of the network forDSUT .Let f¯(i)S indicates the ith RF TeUS sample in DSUS and f¯ (i)T is the corre-sponding ith B-mode TeUS sample in DSUT after PCA projection. Also, o¯(i)Sand o¯(i)T indicate the output of the DBN for f¯(i)S and f¯(i)T , respectively. Thesize of the visible and the last hidden unit in our DBN is set to be the sameand equal to d = 11. Thus, all f¯(i)S , f¯(i)T , o¯(i)S , and o¯(i)T are â Rd=11. Basedon the Equation 3.7 in Section 3.2.1.2, the fine-tuning stage minimizes theabove explained cross-entropy errors as the loss function, LDiv:LDiv = ââiâDSUSf¯(i)T log o¯(i)S ââiâDSUTf¯(i)S log o¯(i)T , (5.1)Using the proposed network structure, we could use target samples asthe ground-truth for the source sample and vice versa. The intuition behindour proposed approach is that a loss function defined by cross-entropy isclosely related to the KullbacâLeibler (KL) divergence between the sourceand target domain [21], which is widely used to solve domain adaptationproblems. KL divergence or relative entropy is a non-symmetric measureof divergence between two probability distributions, where minimizing theKL divergence of the network output for the source and target instancesaims to draw the two distributions of the source and target domains similarin the common feature space. Minimizing the symmetrical version of KLdivergence between the source and target distribution is equal to minimizingcross entropy between the source and target distributions in the commonfeature space and vice versa [164]. By minimizing cross-entropy betweenthe source and target instances in the input and the output latent-featurespaces, we modify the optimization problem to learn a model that can bebetter generalized for both source and target with less diversity between thetwo domains.This approach can be simplified, by just considering the model thathave to work with only, target data (here TeUS B-mode data). In this case,instead of minimizing the symmetrical version of KL divergence, the goal isto reduce the cross-entropy only between the target domain as the input ofthe network and the source domain as the preferred output of the network.Thus, in the plain version of the approach, the loss function in Equation 5.1is only contained the second term.875.2. Transfer Learning From TeUS RF to B-mode Spectral Features5.2.2.2 Supervised ClassificationAfter unsupervised training of the deep network, we use the labeled dataset,DSL, for supervised training of a shared classifier to separate cancerous frombenign ROIs. We use the trained DBN to find the latent shared featuresfor B-mode and RF time series data in DSL. We train a Support VectorMachine (SVM) classifier with Radial Basis Function (RBF) kernel. SVMhas been previously demonstrated to differentiate between various tissuetypes using TeUS data in ex vivo [108] and in vivo [13, 76] studies with highaccuracy. In addition to the binary output of the classifier, we estimate thelikelihood of the ROIs to be cancerous [13]. More detail about SVM can befound at Section 3.2.5.2.2.3 Baseline ClassificationIn addition to domain adaptation as a solution for the accessibility of invivo TeUS RF data for prostate cancer detection, we explore other plausiblesolutions as baselines for comparative analysis versus the proposed method.These include direct deployment of the trained DBN network with TeUS RFdata in [11, 13] and test on TeUS B-mode data (BL-1), and train a DBNmodel using the method proposed in [11] with TeUS B-mode data and teston the same data type (BL-2).5.2.2.4 GeneralizationTo remove any bias that could be potentially introduced in training/adap-tation, we randomly divide the data into independent subsets of training,validation, and testing. Specifically, the test data was generated by fullyexcluding data of half of the biopsy cores from DSL and DSU . This new intro-duced data division include, 42 biopsy cores in the test dataset, DStest. Forthe remaining 42 cores, we will follow the procedure that we explain earlierfor training and validation.5.2.3 Results and DiscussionTo evaluate the proposed approach, we conducted a series of experimentsto assess the performances of the unsupervised domain adaptation andclassification.885.2. Transfer Learning From TeUS RF to B-mode Spectral Features5.2.3.1 Unsupervised Domain AdaptationDBN Parameters: As we mentioned earlier, to find a common featurespace between RF and B-mode time series, we train a shared DBN betweenDSUS and DSUT . We used the MATLAB library of Tanaka et al. [147] tobuild and train the DBN. The training step requires setting the valuesof numerical meta-parameters such as the learning rate (LR), momentum,weight-cost, initial values of the weights, number of hidden units, and mini-batch size (BS). For this purpose, we randomly divide DSU to training (80%)and validation (20%) datasets. Initially, we set the DBN structure includingthe number of hidden layers and nodes configuration, as well as the numericalmeta-parameters to the default values of the DBN library [147]. The DBNstructure consists of two RBM layers with d = 11 visible and hidden unitsin the first and last layers. In the proposed architecture, we need to setthe number of hidden neurons for the first layer, n (see Fig. 5.1), as well asthe learning rate and mini-batch size. The purpose of training is to reduceerror to a reasonably low value in as few epochs as possible. We heuristicallysearch for n, so that cross entropy between a sample and its reconstructionthrough DBN is minimized in both the training and validation datasets in asfew epochs as possible. Figure 5.2(a) shows the cross-entropy loss functionfor different numbers of hidden neurons using 250 iterations in pre-trainingand training steps. Since the lowest error is obtained with n = 44, thefinalized DBN is composed of a real-valued visible layer with 11 units, andtwo hidden layers consisting of 44 and 11 hidden units, respectively. Thelearning rate and mini-batch size are set similarly with a coarse search asshown in Fig. 5.2. Figure 5.2(b) shows the cross-entropy loss function fordifferent numbers of hidden neurons using 250 epochs, where we have thelowest errors for the LR = 0.2. As it is shown in Fig. 5.2(c), we also achievedthe lowest error for the BS = 10. The momentum and the weight cost valuesdo not change from the default values (0.9, 2Ã 10â4). In the fine-tuning step,we ran 250 epochs with a fixed learning rate of 0.2, and a mini-batch size of10. After completion of the learning procedure, the last hidden layer of theDBN produces the latent common feature representation.Domain Alignment: Figure 5.3 shows the divergence between RF and B-mode time series data in DSL prior to (top row) and after (bottom row) domainalignment using the shared DBN. This figure depicts a normal distributionwhich fitted to the histogram of the top three features obtained followingPCA whitening. It is obvious that the proposed domain adaptation methodcan effectively align features in common learned feature space. Moreover,895.2. Transfer Learning From TeUS RF to B-mode Spectral Features0 50 100 150 200 250Iteration (Epoch)3.33.43.53.63.73.83.944.1Cross-EntropyLoss11 Neurons22 Neurons33 Neurons44 Neurons(a)0 50 100 150 200 250Iteration (Epoch)3.33.353.43.453.53.553.63.653.73.753.8Cross-EntropyLossLR = 0.2LR = 0.1LR = 0.01(b)0 50 100 150 200 250Iteration (Epoch)3.33.353.43.453.53.553.63.65Cross-EntropyLossBS = 10BS = 20BS = 40(c)Figure 5.2: Learning curve for DBN training based on the cross-entropy:(a) for first hidden layer size. (b) for different learning rates (LR). (c) fordifferent mini-batch size (BS). In a coarse search for the meta-parameters weachieved the lowest cross entropy loss with n = 44, LR = 0.2, and BS = 10.we use the Subspace Disagreement Measure (SDM) [54] to evaluate thedomain difference [30, 46]. In a subspace with dimension d, the domaindifference is calculated as 4D = âdi=1 SDM(i). For perfectly alignedfeature distributions, SDM(i) = 0 . So, the lower the 4D is, the better thedomains are aligned. 4D decreases from 0.708 to 0.449 for RF and B-modetime series data in d = 11 feature space as a result of domain adaptation.Aligning domains helps compensate for the domain distribution shift, thusmaking the decision models trained on the aligned source space comparablewith the source-aligned target domain. As a result, we can validate the per-formance of our domain adaptation method indirectly using the classificationaccuracy obtained in supervised training step with aligned source and targetdata.5.2.3.2 Supervised ClassificationData Division: To determine the sensitivity of our methodology to thechoice of the training labeled-data, we create different partitions of trainingand testing datasets from DSL in a hold-out validation setting. For eachnew pair of datasets, we randomly select x% of cores in the labeled datasetas the training data for the supervised training of the SVM and the other100%â x% cores are used as the testing data. We generate three datasetsincluding RF and B-mode time series data with x = 50%, 75%, and 85% ofthe total 84 biopsy cores in DSL.Classification Performance: We assess the overall performance of ourapproach using the AUC. This curve depicts relative trade-offs between905.2. Transfer Learning From TeUS RF to B-mode Spectral Features-0.1 0 0.1 0.2 0.3Feature 1020406080100120NumberofSamplesDistributaion of Feature 1B-modeRF-0.2 0 0.2 0.4 0.6Feature 2020406080100120140NumberofSamplesDistributaion of Feature 2B-modeRF-0.1 0 0.1 0.2 0.3 0.4Feature 3020406080100120140NumberofSamplesDistributaion of Feature 3B-modeRF0.85 0.9 0.95 1 1.05 1.1 1.15Feature 1050100150200250300NumberofSamplesDistributaion of Feature 1B-modeRF(a) Distribution of feature 1-3 -2 -1 0 1 2 3Feature 2050100150200250NumberofSamplesDistributaion of Feature 2B-modeRF(b) Distribution of feature 2-2 -1 0 1 2Feature 3050100150200250300350400NumberofSamplesDistributaion of Feature 3B-modeRF(c) Distribution of feature 3Figure 5.3: Distribution shift from B-mode to RF for the top three featuresbefore (top row) and after (bottom row) the shared deep network. Theproposed domain adaptation method can effectively align features and reducethe distribution shift in common learned feature space.sensitivity and specificity. The larger AUC values indicate better classificationperformance. Accuracy, sensitivity, and specificity are also measured for thetest data. For this purpose, we consider all of the cancerous cores as thepositive class and all the benign cores as the negative class. Sensitivity isthe percentage of cancerous cores that are correctly identified as cancerouscompared to the pathology results; specificity is the proportion of non-cancerous cores that are correctly classified, and accuracy is the percentageof true results (both true positives and true negatives) in the total numberof cores. For each data division, we randomly select 10 different trainingand testing datasets and report the average performance. Table 5.1 andTable 5.2 show the classification results for differentiating between cancerousand benign cores.Size of the Labeled Dataset: We investigate how the learning curveprogresses with increasing the number of training samples. This is indirectlyassessed with k-fold cross-validation increasing k from 2 to 6. Figure 5.4 andTable 5.1 show AUC, accuracy, sensitivity and specificity of the classification915.2. Transfer Learning From TeUS RF to B-mode Spectral FeaturesTable 5.1: Model performance measured by AUC for classification in differentdata divisions.Data division RF time series B-mode series50% train, 50% test 0.77 (± 0.01) 0.68 (± 0.01)75% train, 25% test 0.72 (± 0.03) 0.70 (± 0.03)85% train, 15% test 0.71 (± 0.01) 0.71 (± 0.02)Table 5.2: Model performance measured by specificity and sensitivity forclassification in different data divisions.Data division RF time series B-mode seriesSpecificity Sensitivity Specificity Sensitivity50% train, 50% test 0.69 (± 0.03) 0.70 (± 0.02) 0.70 (± 0.02) 0.72 (± 0.02)75% train, 25% test 0.69 (± 0.01) 0.79 (± 0.07) 0.73 (± 0.05) 0.76 (± 0.07)85% train, 15% test 0.75 (± 0.06) 0.78 (± 0.06) 0.73 (± 0.03) 0.76 (± 0.05)for different cross-validation settings. A two-way t-test between k = 2 andk = 6 fails to show statistically significant difference in AUC, accuracy,sensitivity and specificity (p > 0.05).Integration of TeUS and mp-MRI: In another analysis, we combine theTeUS cancer detection results with readings from mp-MRI. The combinationmethod takes advantage of both imaging modalities. If mp-MRI declarescancer suspicious level as low or high (see Section 2.1) for a core, we useits predictions alone and declare the core as benign or aggressive cancer,respectively. On the other hand, when mp-MRI declares the cancer suspiciouslevel as intermediate, we use predictions based on TeUS data. We evaluate thecombined approach with the data division explained earlier in Section 5.2.3.2.Table 5.3 and Table 5.4 show the classification performance for differentiatingbetween cancerous and benign cores using the combined method measuredby AUC, specificity, and sensitivity. The combined approach leads to anAUC of 0.81 for predicting cancer versus benign cores using RF time seriesand AUC of 0.79 using B-mode time series.5.2.3.3 Baseline ClassificationWe assess the comparative performance of the proposed method over thebaselines (Section 5.2.2.3) in terms of AUC, sensitivity, and specificity. Theperformance evaluating the metrics are discussed earlier in the section.925.2. Transfer Learning From TeUS RF to B-mode Spectral Features0.500.550.600.650.700.750.800.852-fold 3-fold 4-fold 6-foldPerformance RF time series Area Under the Curve (AUC)AccuracySensitivitySpecificity(a)0.500.550.600.650.700.750.800.852-fold 3-fold 4-fold 6-foldPerformance B-mode time seriesArea Under the Curve (AUC)AccuracySensitivitySpecificity(b)Figure 5.4: Influence of labeled dataset size in classification accuracy: perfor-mance of the method measured by AUC, accuracy, sensitivity and specificityin the k-fold cross-validation setting for (a) TeUS RF data and (b) TeUSB-mode data.Table 5.3: Performance for the combination of mp-MRI and TeUS measuredby AUC for classification in different data divisions.Data division RF time series B-mode series50% train, 50% test 0.80 (± 0.01) 0.76 (± 0.01)75% train, 25% test 0.81 (± 0.07) 0.77 (± 0.03)85% train, 15% test 0.81 (± 0.06) 0.79 (± 0.0)1Table 5.5 shows the model performance in different data divisions usingBL-1, BL-2, TeUS RF and adapted TeUS B-mode approaches. As expected,for BL-1 where we directly use B-mode data with a model trained usingRF data, the AUC is close to a random prediction. These observations showthat, while TeUS RF and B-mode data are related, the differences in theircorresponding feature distributions are significant and affect classificationresults. Moreover, for BL-2 where we used TeUS B-mode data for trainingand testing of the DBN model, the obtained AUC is 0.58. The results showthat the obtained AUCs are significantly below that of TeUS RF and TeUSB-mode data (p<0.05 for either BL-1 and BL-2 versus TeUS RF.).5.2.3.4 GeneralizationFor the 42 cores that we explained in Section 5.2.2.4, we follow the procedurementioned earlier for training and validation. Table 5.6 shows performance ofthe proposed approach in DStest. On this test set, the classification resulted inan AUC of 0.71 for predicting prostate cancer using only TeUS RF data and935.2. Transfer Learning From TeUS RF to B-mode Spectral FeaturesTable 5.4: Performance for the combination of mp-MRI and TeUS measuredby specificity and sensitivity for classification in different data divisions.Data division RF time series B-mode seriesSpecificity Sensitivity Specificity Sensitivity50% train, 50% test 0.73 (± 0.03) 0.75 (± 0.01) 0.73 (± 0.02) 0.79 (± 0.05)75% train, 25% test 0.77 (± 0.07) 0.83 (± 0.04) 0.77 (± 0.02) 0.76 (± 0.06)85% train, 15% test 0.80 (± 0.07) 0.78 (± 0.04) 0.76 (± 0.02) 0.81 (± 0.07)Table 5.5: Comparison of model performance measured by AUC usingbaselines and the proposed approach in different data divisions.Data division BL-1 BL-2 RF series B-mode series50% train, 50% test 0.50 (± 0.08) 0.58 (± 0.06) 0.77 (± 0.01) 0.69 (± 0.02)75% train, 25% test 0.49 (± 0.10) 0.61 (± 0.03) 0.72 (± 0.02) 0.70 (± 0.01)85% train, 15% test 0.51 (± 0.12) 0.60 (± 0.02) 0.71 (± 0.01) 0.71 (± 0.03)0.70 using domain adaptation with TeUS B-mode data. When integratingTeUS with MRI readings, these results were increased to AUCs of 0.76 and0.74, respectively.Figure 5.5 shows the comparative performance, measured by AUC, of theproposed method over the baselines (Section 5.2.2.3) for DStest. For BL-1,the AUC = 0.51 and it is close to a random prediction. In comparison to theTeUS RF approach, we can see a significant improvement (p < 0.05). ForBL-2, AUC = 0.59, which is significantly below that of either TeUS RF orTeUS B-mode data (p < 0.05).Table 5.6: Performance for the TeUS only and combination of mp-MRI andTeUS measured by AUC, specificity, and sensitivity for classification in thetest dataset.Data division RF time series B-mode seriesAUC Specificity Sensitivity AUC Specificity SensitivityTeUS 0.71 0.73 0.71 0.70 0.71 0.75TeUS + mp-MRI 0.76 0.78 0.76 0.74 0.79 0.76945.3. Transfer Learning From TeUS RF to B-mode Using RNNFigure 5.5: The comparative performance of the proposed method measuredby AUC over the baselines for DStest5.2.3.5 Colormaps:Figure 5.6 shows examples of the cancer likelihood maps, derived from theoutput of SVM using both TeUS RF and B-mode data, overlaid on B-modeUS image. We use the approach described in our earlier publication [11] forthis purpose. In the colormaps, red regions belong to ROIs for which thecancer likelihood is more than or equal to 50%.5.3 Transfer Learning From TeUS RF to B-modeUsing RNNTowards a real-time system for temporal enhanced ultrasound-guided prostatebiopsy, in this section, we propose an RNN-based transfer learning solutionwhich is solely working with TeUS B-mode data streaming from an ultrasoundscanner. The method is implemented as a part of a unified software frameworkdemonstrating near real-time analysis of ultrasound data stream using adeep learning solution. The detail description and additional informationabout the software framework are presented in Appendix B.At the center of the proposed software solution, a probabilistic model isresponsible for analysis of temporal enhanced ultrasound using RNN. Themodel is trained to capture tissue-dependent feature from TeUS B-mode datawhich rely on the successful knowledge transfer between TeUS RF and TeUSB-mode data in the unsupervised training step. The transfer learning anddiagnosis approach is evaluated for cancer detection accuracy on ultrasounddata obtained from a large clinical study with 255 biopsy cores from 157955.3. Transfer Learning From TeUS RF to B-mode Using RNN(a) (b)(c) (d)Figure 5.6: Cancer probability maps overlaid on B-mode US image, alongwith the projected needle path in the temporal US data and centered onthe target. The ROIs for which the cancer likelihood is more than 50% arecolored in red, otherwise they are colored as blue. The red boundary showsthe segmented prostate in MRI projected in TRUS coordinates, dashed lineshows needle path and the arrow pointer shows the target: (a)-(c) Cor-rectly identified cancerous core using RF time series data; (b)-(d) Correctlyidentified cancerous core using B-mode time series data.subjects. The proposed solution is further assessed with an independentdataset with 21 biopsy targets from six subjects. In the first study, weachieve the area under the curve, sensitivity, specificity, and accuracy of0.94, 0.77, 0.94 and 0.92, respectively, for the detection of prostate cancer.In the second study, we achieve an AUC of 0.85. Our results suggest thatTeUS-guided biopsy can be potentially effective for the detection of prostatecancer.5.3.1 MaterialsThe deep networks are generated using a dataset consisting of biopsy targetsin mp-MRI-TRUS fusion-biopsies with 255 biopsy cores from 157 subjects.We refer to this data as the first retrospective study. As we explained before,for development of our transfer learning system, we use both unlabeled data965.3. Transfer Learning From TeUS RF to B-mode Using RNNfrom the whole ultrasound scan and labeled data from the biopsy targetregion coming from the dataset explained in Chapter 2. In this section, wealso performed another small independent prostate biopsy study for furtherassessment of our approach. The details of materials and the complementarystudy can be found in following sections.5.3.1.1 Data DivisionThe data from 255 biopsy cores is divided into mutually exclusive training,DTtrain, and test sets, DTtest. Training data,DTtrain, is made up of 84 coresfrom patients with homogeneous tissue regions which we randomly selectthem. We further use the test data,DTtest, consists of 171 cores to evaluatethe trained model during the guidance system implementation, where 130cores are labeled as benign and 31 cores are labeled as cancerous with GS â¥3+3.Unlabeled Data: As we mentioned in Section 2.3.1, in temporal represen-tation of TeUS, each ROI is represented by T = 100 echo intensity value over100 consecutive ultrasound frames as x = (x1, ..., xT ). Unlike our typicalbiopsy region analysis, here, the RF time series data from the ROIs of the en-tire imaging plane of the prostate and in the scan level make up the unlabeledsource dataset, DTUS , while the corresponding TeUS B-mode sequence makeup the unlabeled target dataset, DTUT , where DTU = DTUT âª DTUS . Both DTUSand DTUT have an equal number of samples. In total, DTU includes 5,255,680unlabeled training samples, consisting of 2,627,840 RF and 2,627,840 B-modetime series data. To maintain the mutual exclusiveness between the trainand test data, for generation of above unlabeled data, we only use the coresin DTtrain.Labeled Data The labeled source dataset, DTLS , includes TeUS RF sequenceof ROIs and the target dataset DTLT , includes the corresponding TeUS B-mode sequence. Given the data augmentation strategy, we obtain a totalnumber of 129,024 training samples for both TeUS RF and TeUS B-modedata of the cores that belong to DTtrain.5.3.1.2 Complementary Second Retrospective StudyTo further assess the developed solution, we use the data that we acquired inthe second retrospective study. As we explained in Section 2.4 we performed asecond independent fusion biopsy study including six subjects. In this study,only TeUS B-mode data were recorded for each target to minimize disruption975.3. Transfer Learning From TeUS RF to B-mode Using RNNTable 5.7: Gleason score distribution in the second retrospective clinicalstudy.GS Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 GS 4+5Number of Cores 7 2 3 2 1 6to the clinical workflow. This study resulted in 21 targeted biopsy cores withGS distribution as explained in Table 5.7. We use the histopathology labelingof the cores as the ground-truth to assess the accuracy of the guidance systemin detecting the cancerous lesions.5.3.2 MethodsThe deep networks are generated mainly based on the methods that wepresented in our earlier works and previous sections. We give a brief overviewof these methods. For a detailed description of the models, the reader mayrefer to Section 3.2 and Section 5.2 [10, 14].An individual TeUS sequence of length T , x(i), is composed of echo-intensity values x(i)t for each time step, t, and is labeled as y(i) â {0, 1},where zero and one indicate benign and cancer biopsy outcome, respectively.We aim to learn a mapping from x(i) to y(i) in using RNNs to model thetemporal information in TeUS B-mode data.For this purpose, first, we use the plain version of the unsuperviseddomain adaption approach presented in [14] to find a common feature spacebetween TeUS RF and TeUS B-mode data. Let x(i)r indicates the ith RFTeUS sample in DTUS and x(i)b is the corresponding ith B-mode TeUS samplein DTUT . Also, h(i)b indicate the output of the RNN for x(i)b . Here, we use alayer of RNN with Long Short-Term Memory (LSTM) cells [50], thus, thesize of input and output is equal to T = 100. Based on the Equation 5.1in Section 5.2, to find the common feature space between the source anddomain, the unsupervised training stage minimizes the cross-entropy errorsas the loss function, LDiv:LDiv = ââiâDTUTx(i)r log h(i)b , (5.2)Given an input TeUS B-mode sequence x(i)b = (x(i)1 , , ..., x(i)T ), our RNNcomputes a hidden vector sequence h(i)b = (h(i)1 , , ..., h(i)T ) in the common985.3. Transfer Learning From TeUS RF to B-mode Using RNNfeature space between the RF and B-mode data within a domain adaptionstep. After the unsupervised domain adaptation step, our discriminativeRNN architecture is built with LSTM cells [50], where each cell, maintains amemory over time. We use two layers of LSTMs with T = 100 hidden unitsto capture temporal changes in data. Following these layers, we use a fullyconnected layer to map the learned sequence to a posterior over binary classesof benign and cancer tissue in a supervised classification step. This final nodegenerates a predicted label, y(i), for a given TeUS B-mode ROI sequence,x(i)b â DTLT . The training criterion for the network is to minimize the lossfunction as the binary cross-entropy between y(i) and y(i) over all of thetraining samples where we use Root Mean Square Propagation (RMSprop)optimizer. As recommended, using a training-validation setting, we performa grid search to find the optimum hyperparameter in our search space. Oncethe optimum hyperparameters are identified (the number of RNN hiddenlayers = 2, batch size = 64, learning rate = 0.0001, dropout rate = 0.4, andthe regularization term = 0.0001), the entire training set, DTtrain, is used tolearn the final classification model.5.3.3 Results and Discussion5.3.3.1 Classification model validationTo evaluate the proposed RNN-based approach, we used clinical data fromtwo retrospective studies to simulate the flow of information across differentmodules. To validate the accuracy of the classification model, we use DTtest.We use sensitivity, specificity, and accuracy in detecting cancerous tissuesamples to report the validation results. We consider all cancerous cores as thepositive class and other non-cancerous cores as the negative class. Sensitivityis defined as the ratio of cancerous cores that are correctly identified whilespecificity is the ratio of non-cancerous cores that are correctly classified.Accuracy is the ratio of the correctly identified results over the total numberof cores. We also report the overall performance of our approach usingthe AUC. Table 5.8 shows the model performance for classification of coresin DTtest for different MR suspicious levels. For samples of moderate MRsuspicious level (70% of all cores), we achieve an AUC of 0.94 using theLSTM-RNN. In this group, our sensitivity, specificity, and accuracy are 0.75,0.96, and 0.93, respectively. In comparison, only 26% of all of the coresidentified in mp-MRI are cancerous after biopsy which means our approachcan effectively complement mp-MRI during the guidance procedure to reducethe number of false positives for those targets with moderate MR suspicious995.3. Transfer Learning From TeUS RF to B-mode Using RNNFigure 5.7: Guidance interface implemented as part of a 3D Slicer module:cancer likelihood map is overlaid on B-mode ultrasound images. Red indicatespredicted labels as cancer, and blue indicates predicted benign regions. Theboundary of the segmented prostate is shown with white and the green circleis centered around the target location which is shown in the green dot.level.Table 5.8: Model performance for classification of cores in the test data fromthe first retrospective study for different MR suspicious levels. N indicatesthe number of cores in each group.Method Specificity Sensitivity Accuracy AUCAll of the biopsy cores (N = 171) 0.94 0.77 0.92 0.94Moderate MR suspicious level (N = 115) 0.96 0.75 0.93 0.94High MR suspicious level (N = 20) 0.85 0.98 0.95 0.955.3.3.2 System assessmentFigure 5.7 shows the guidance interface implemented as part of a 3D Slicermodule running on the client machine. In order to evaluate the performanceof the proposed solution, other than the subjective evaluation of guidancevisualization, the accuracy of the target detection and the run-time aremeasured.Guidance accuracy: To further assess the developed approach, we per-formed a second independent MRI-TRUS fusion biopsy study. This study1005.3. Transfer Learning From TeUS RF to B-mode Using RNNTable 5.9: Run-time of the steps of the prostate guidance system averagedover N = 21 trials with data from the second retrospective study (given asmean±std).Host Machine Operation Average time (s)TeUS-Client Guidance Visualization 0.40 (± 0.11)TeUS-Server Classification 1.66 (± 0.32)Segmentation 0.12 (± 0.17)resulted in 21 targeted biopsy cores with GS distribution as explained inTable 5.7. We achieve an AUC, sensitivity, specificity, and accuracy of 0.85,0.93, 0.72 and 0.85, respectively, for the fusion biopsy targets. Our resultsshow that the only miss-classified cancerous target is GS 3+3 with the tumorin core less than 0.4 cm.Run-time: The run-time was measured for all parts of the workflow (classi-fication, segmentation, and guidance visualization) using a timer log providedby the open-source Visualization Toolkit. The TeUS-client computer featuredan 2.70 GHz Intel CoreTM i7-6820 (8 CPUs) processor with 64 GB of RAM,Windows 10 Enterprise N, Visual Studio 2015, 3D Slicer 4.3.1, OpenMP 2.0,and Intel MKL 11.0. The TeUS-server was hosted by a computer running anUbuntu 16.04 operating system and a 3.4 GHz Intel CoreTM i7 CPU with16 GB of RAM, equipped with GeForce GTX 980 Ti GPU with 2816 CUDAcores and 6 GB of memory.Run-time results are summarized in Table 5.9. Averaged over all N =21 trials in the second fusion biopsy study, the total time spent on TeUS-client for guidance visualization and post-processing is 0.40 ± 0.11 seconds.Since the classification and segmentation modules run simultaneously, theTeUS-server run-time is constrained by the classification task run-time of abatch of 100 ultrasound data frames, which is 1.66 ± 0.32 second. Withinthe context of prostate biopsy guidance workflow, this means an addition ofonly about 20 sec to a 20 min procedure. We consider this near real-timeperformance sufficient for the requirements of this clinical procedure.5.3.4 Discussion and comparison with other methodsThe best result to-date involving TeUS B-mode data (AUC = 0.7) is basedon spectral analysis using DBN as the underlying machine learning frame-work [11, 13]. Comparing our LSTM-RNN approach with that method as1015.4. Conclusionthe most related work, a two-way paired t-test shows statistically significantimprovement in AUC (p < 0.05). Furthermore, using the RNN-based frame-work simplifies the real-time implementation of the guidance system. Whileanalysis of a single RF ultrasound frame is not feasible in the context ofour current clinical study, as we did not access to the transducer impulseresponse for calibration, previously, we have shown that analysis of TeUSsignificantly outperforms the analysis of a single RF frame [36, 108]. Thebest results reported using a single RF frame analysis [45] involve 64 subjectswith an AUC of 0.84, where they used Prostate Specific Antigen (PSA) asan additional surrogate.The performance of mp-MRI for detection of prostate cancer has beenestablished in a multi-center, 11-site study in the UK [3], consisting of 740patients, to compare the accuracy of mp-MRI and systematic biopsy againsttemplate prostate mapping biopsy (TMP-biopsy) used as a reference test.The findings of the study were that for clinically significant cancer, mp-MRIwas more sensitive (93%) than the systematic biopsy (48%), but much lessspecific (41% for mp-MRI vs. 96% for systematic biopsy). In a prospectivecohort study of 1003 men undergoing both targeted and standard biopsy [144],targeted fusion biopsy diagnosed 30% more high-risk cancers vs. standardbiopsy, with an area under the curve of 0.73 vs. 0.59, respectively, basedon whole-mount pathology. TeUS shows a balance of high specificity andsensitivity across all cancer grades while enabling an ultrasound-only basedsolution for guiding prostate biopsies.5.4 ConclusionIn this chapter, we tried to address the challenge of accessibility of RFdata in the commercial ultrasound scanners. In the first part, we proposeda novel method for unsupervised domain adaptation of RF (source) andB-mode (target) time series data. We worked towards the objective ofmaking the method maximally discriminative for prostate cancer detectionon both source and target domain by leveraging: (1) a shared deep networktrained using relatively large amount of unlabeled TeUS data from sourceand target, and (2) a shared classifier using a few labeled examples fromboth domains. We demonstrated that in presence of a distribution shiftbetween RF and B-mode data, a transfer learning method can compensatefor the divergence between the distribution and enable TeUS B-mode-basedtissue classification as an alternative to TeUS RF-based approach for theprostate cancer detection, where RF data may not be easily accessible on a1025.4. Conclusioncommercial ultrasound scanner. The need for domain adaptation is supportedby a recent paper [30], which also reported that even within RF data mode,small changes to the scattering environment result in significant shifts in thedistribution of features. The proposed approach is successful in factoringout the effect of domain shift in TeUS data by adapting cross-entropy lossbetween the two domains at the common shared feature space. In an in vivostudy including 84 biopsy targets, we achieved the AUC of 0.70 and accuracyof 0.73 using TeUS B-mode data. By using TeUS RF data, the AUC andaccuracy are 0.72 and 0.73, respectively.We considered this study, as a feasibility investigation to take advantageof knowledge transfer methods in ultrasound-based diagnosis and interventiontechniques. This approach can be used as a versatile pre-trained networkto reduce divergence between these two domains, which can be fine-tunedfor a given clinical task. An extension of the approach has been used in thesecond part, to capture variations in TeUS RF and B-mode data in real-time using temporal analysis of TeUS with RNNs. We have to demonstratethat accuracy of the integrated system on clinical ultrasound devices thatonly provide B-mode data is comparable to those in bench-top prototypesin the laboratory. The system was validated using retrospective in vivodatasets including both TeUS RF and B-mode data of 255 biopsy targetcores obtained in a large clinical study during mp-MRI-TRUS fusion-biopsy.For the validation data, we achieved an AUC of 0.94, and sensitivity andspecificity of 0.77, and 0.94 respectively. The integrated system was thenevaluated with an independent in vivo dataset obtained during mp-MRI-TRUS fusion-biopsy from six subjects and includes only B-mode TeUS of 21biopsy targets. We achieved an AUC of 0.85 for this dataset, and sensitivityand specificity of 0.93 and 0.72, respectively. The average sensitivity of thesystem considering both studies is 82% where 18% of cancerous cores (10out of 55) were not identified.From a clinical perspective, the motivation of the work is to enable real-time assessment of TeUS for prostate cancer detection. The immediate goalof the current work is to demonstrate the viability of this approach usingretrospective data with known ground-truth so that we can optimize thesystemâs performance and understand how such system would fit within thestandard clinical workflow. The promising results of this initial assessment in-dicate that our proposed TeUS-based system is capable of providing guidanceinformation for the prostate biopsy procedure. Future work should focuson prospective evaluation and feasibility assessment of the biopsy guidancesystem.103Chapter 6Investigation of PhysicalPhenomena UnderlyingTeUSThe particular aspect of time that Iâm interested in is the arrow of time:the fact that the past is different from the future. We remember the past butwe donât remember the future.â Sean M. Carroll6.1 IntroductionTemporal Enhanced Ultrasound (TeUS) is a novel non-invasive imagingparadigm that captures information from a temporal sequence of backscat-tered ultrasound Radio-Frequency (RF) or B-mode image data obtained froma fixed tissue location. This technology has been shown to be effective for clas-sification of various in vivo [11, 13, 73, 75, 110] and ex vivo [108, 109] tissuetypes including prostate cancer from benign tissue. A comparison of TeUSwith the analysis of power spectrum of RF data for tissue characterizationshowed that TeUS and RF spectral analysis compliment each other [73, 108],This chapter is partly adapted from the following papers [10, 18]: 1) Shekoofeh Azizi,Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, Guy Nir, Jin Tae Kwak, Sheng Xu, StoreyWilson, Kenneth A Iczkowski, M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean,Peter Pinto, Bradford Wood, Purang Abolmaesumi, and Parvin Mousavi. Detection andgrading of prostate cancer using temporal enhanced ultrasound: combining deep neuralnetworks and tissue mimicking simulations. MICCAIâ16 Special Issue: International Journalof Computer Assisted Radiology and Surgery, 12(8):1293â1305, 2017, and 2) ShararehBayat, Shekoofeh Azizi, Mohammad I Daoud, Guy Nir, Farhad Imani, Carlos D Gerardo,Pingkun Yan, Amir Tahmasebi, Francois Vignon, Samira Sojoudi, et al. Investigation ofphysical phenomena underlying temporal enhanced ultrasound as a new diagnostic imagingtechnique: Theory and simulations. IEEE Transactions on Ultrasonics, Ferroelectrics, andFrequency Control, 2017.1046.1. Introductionwith TeUS showing higher overall AUC. More recently, as we have shownin previous chapters, TeUS RF and B-mode can successfully be used fordiagnosis and characterization of prostate cancer and its extent [9â14, 16].While the physical phenomenon governing temporal ultrasound/tissueinteraction is the subject of ongoing investigation in our group, severalhypotheses have been explored so far. Our previous studies have indicatedtwo primary phenomena that influence TeUS:1. Changes in tissue temperature due to acoustic absorption: It has beenproposed that the acoustic radiation force of the transmit US signalincreases the temperature and changes the speed of sound in differenttissue types [36].2. Micro vibrations of tissue due to physiological vibration: It has also beensuggested that a combination of micro-vibration of acoustic scatters inmicro-structures and the density of cells play a role [111].To optimize TeUS for clinical translation, we aim to investigate thephysical processes that govern US-tissue interaction during the acquisition ofTeUS data [109]. We previously investigated changes in tissue temperatureduring TeUS data acquisition, using a numerical model [36]. The resultsdemonstrated that changes in tissue temperature, which affect the speedof sound, can be used for tissue characterization. However, even withexaggerated image settings of high frame rate and acoustic power that do notmatch clinical imaging conditions, classification results were substantiallybelow those achieved in our ex vivo and in vivo studies [36].Our results from Chapter 3 and Chapter 4 showed a consistently highclassification accuracy in a large dataset in this thesis. The preliminaryfeature visualization results from Chapter 3 shows that frequencies between0 â 2 Hz provide the most discriminative features for distinguishing can-cerous and benign tissue. These results suggest that the phenomenon isconsistent for the two independent training and test datasets in clinical set-tings. Interestingly, the range of frequencies that we have identified as mostdiscriminative between cancerous and benign tissue (0â 2 Hz in Fig. 6.5) arealso consistent with the ranges we have observed in our previous independentstudies [75, 76].In this section, we investigate the second phenomenon related to tissuemicro vibrations as the main source of the tissue typing capabilities of TeUSwith clinical image settings. Possible sources of tissue micro vibration includeexternal, low-amplitude environmental vibrations, and internal physiologicalmotion such as pulsation due to the heart beat [10, 19]. In design and1056.2. Spectral Feature Visualizationdeployment of deep neural networks through this thesis, we centered our focuson the internal representations learned by our models to derive insights abouttissue dependent features residing in TeUS. In this section, we extend ourprevious spectral feature visualization [12] to further explore the underlyingphysical phenomena governing TeUS. We present a method for visualizationand interpretation of the learned features from spectral analysis of TeUSdata for prostate cancer grading. In the next step, as suggested in ourprevious studies, we simulate the effect of micro-vibration as a potentialsource of TeUS tissue typing properties, using nuclei locations extracted from14 whole-mount histopathology slides.6.2 Spectral Feature Visualization6.2.1 MaterialsIn this section, we use the data from the first study as we explained inChapter 2 and we focus on the spectral representation of TeUS. From thewhole 255 cores in our dataset, we divide the data from 197 cores, into thetrain, DStrain, and test, DStest, sets as explained in Section 4.2. For buildingthe trained DBN model, we use DStrain along with the distribution learningapproach that we explained in Section 4.2. The test data, DStest, consists of165 cores from 114 patients, with the following distribution: 121 benign, 12GS of 3+3, 14 GS of 3+4, 2 GS of 4+3, and 16 GS of 4+4.6.2.2 MethodologyTo determine the characteristics of the non-cancerous and cancerous tissuesamples in the TeUS data and their correlation with the learned features, wepropose a feature visualization approach for the network that we discussedin Section 4.2. We use this approach to identify the most discriminativefrequency components of the TeUS RF data as learned in the feature learningstep (See Sections 4.2.2.1). Figure 6.1 shows an overview of the proposedmethod. First, TeUS test data, DStest, is propagated through the trained DBN,and the activations of the last hidden layer (i.e., the learned latent features)are computed. We then use the GMM along with the learned features, asexplained in Section 4.2 to assign a prostate cancer grades to each ROIs ofthe test dataset. To examine the significance of the ith (i = 1, .., 6) individuallearned latent feature in the detection of different grades, the activations ofall other five hidden units in the third layer are set to zero. The activationof the non-zero learned feature is back-propagated to the input layer. The1066.2. Spectral Feature VisualizationWhiteningTrained DBNGleason Grade PredictionLearned FeaturesithFeatureBackpropagationAbsolute DifferenceGaussian Mixture ModelTest Data Feature VisualizationFigure 6.1: An illustration of the proposed feature visualization method.resulting reconstructed signal, displayed in the input layer as a series offrequency components, highlights those components that contribute to theactivation of the non-zero learned feature. By comparing the componentsactivated for ROIs labeled as GS pattern of 3, 4 as well as non-canceroustissue in Section 4.2, we can identify those frequency ranges that are differentbetween two tissue types. This process is performed for all the six latentfeatures.The results of feature visualization (Fig. 6.5(a)) suggest that frequenciesbetween 0â 2 Hz provide the most discriminative features for distinguishingbetween Gleason pattern 3 and 4 as well as non-cancerous tissue samples (SeeSection 6.4). Moreover, the identified frequency range is also consistent withthe ranges that we have observed in our previous independent studies [11, 75,76]. Our results to-date suggest that tissue micro-vibration, possibly due topulsation from the heartbeat (â¼ 1.2 Hz), is a key contributor to the tissuetyping capability of TeUS. To further examine this hypothesis, we continueour study with a histopathology mimicking simulation as explained in thenext section.1076.3. Histopathology Mimicking SimulationFinite Element Model SimulationsDigital Pathology DatasetUS Simulations(Field II)Nuclei Location ExtractionTeUS GenerationFeature ExtractionFigure 6.2: Pathology mimicking simulations framework.6.3 Histopathology Mimicking SimulationA primary source of the observed backscattered ultrasound signal from thetissue has been associated with the scattering from the cell nuclei [8, 69].Prostate cancer primarily presents as changes in tissue microstructure wheredifferent density, size and spatial arrangement of nuclei are observed [35, 71].Subtle differences in scattering distribution results in significant changes inthe back-scattered signal [68]. We hypothesize that the induced tissue micro-motion due to blood pulsation can cause different US scattering patternsin various microstructures of the prostate, which in turn can be capturedby TeUS for prostate cancer grading. Figure 6.2 shows an overview of oursimulation design to explore this effect.6.3.1 Digital Pathology DataWe use a digital pathology dataset [71] to investigate this hypothesis in ahistopathology-based simulation framework. Our dataset includes 14 digitalhistology slides of prostate cancer patients [71]. We extract the positions ofnuclei from these slides where the locations of cancerous cells are markedby an expert pathologist. We divide the digitized slides into blocks of2Ã 2 mmÃmm, with a resolution of 0.5 µm/pixel. Here, a single scatteringpoint is considered at each nucleus location segmented from the pathologyslides.6.3.2 Numerical Simulation DesignFor the simulation of mechanical micro-vibrations, we design tissue-mimickingphantoms which are generated using digital histology slides. We extract thelocations of nuclei in the whole-mount digital slides. As we mentioned before,we divide each slide into blocks of 2Ã 2 mmÃmm. We then generate a 2D1086.3. Histopathology Mimicking SimulationFigure 6.3: ROI selection and nuclei-based scatterer generation process: (a)Sample of histopathology slide [70], where the red boundary depicts thecancer area; (b) digitized slide overlaid on the histopathology slide, wheregreen and red areas represent the benign and cancer regions, respectively.The selected ROIs are shown by black squares; (c) extracted nuclei positionsin the selected ROIs; left: a cancer region, right: a benign region; (d) theextracted positions of nuclei from each ROI is embedded in an FEM model.model of nuclei positions based on their coordinates in the blocks. A totalnumber of 42 blocks including 14 with GSâ¥4, 14 with GS=3 and 14 fromnon-cancerous tissue types is selected. We place each block at the center ofa homogeneous linear viscoelastic phantom as shown in Fig. 6.2 and Fig. 6.3.The size of the phantom is 5 Ã 5 Ã 5 cm3, and the elasticity and viscosityare set to 25 kPa and 2.15 Pa.s, respectively [92].To capture micro-motion due to an external excitation source, we usea Finite Element Model (FEM) simulation in COMSOL Multiphysics 5.2(Burlington, MA, USA). The external vibration source is defined as a si-nusoidal wave with the frequency of 1 Hz and amplitude of 100 µ m. Wepulsate the inferior surface of the defined tissue-mimicking phantom withthis low-frequency signal. The cellsâ nuclei, which are embedded in the tissuephantom, are displaced in our FEM as a result of the applied mechanicalvibration.6.3.3 TeUS SimulationFor each block, we simulate TeUS data using Field II [77] from the dis-placement data generated by the FEM. In US simulation, the speed of the1096.4. Experiments and Results(a) (b)Figure 6.4: (a) A sample whole-mount histopathology slide of the prostate.Different regions of cancer and benign tissue are shown in the pathologyslide. (b) The corresponding simulated B-mode ultrasound image.sound, the sampling frequency, and the probe frequency are set to 1540 m/s,80 MHz and 6.6 MHz, respectively. We use 40 active elements in transmitand receive aperture while the imaging focal point is set at 2 cm. The first RFframe is created based on the initial positions of scatterers; this is followed bysequential US simulations given FEM displacement information over 2.5 s togenerate TeUS data. We subdivide each block to four 1Ã 1 mmÃmm ROIsalong the lateral and axial directions. It results in 168 ROIs with 56 ROIsof GSâ¥4, GS=3 and non-cancerous tissue, each. Finally, we generate thespectral features for each ROI as explained in Section 2.3. Figure 6.4 showsa sample whole-mount histopathology slide and the corresponding simulatedB-mode ultrasound image. A more detailed description of the methods canbe found in [18, 19].6.4 Experiments and Results6.4.1 Feature Visualization ResultsFor the feature visualization experiment, by subtracting the distributionsof GS pattern 3, 4 and benign samples in the input layer (Section 6.2),we found that feature one, corresponding to hidden activity of the firstneuron of the third layer, along with features four and five, are those thatmaximally differentiate between different prostate cancer grades, especially inlower frequency range. Figure 6.5(a) shows the visualization of distributiondifferences for GS patterns 3, 4 and benign tissues related to the first learned1106.4. Experiments and Results0 1 2 3 4 5 6 7 8 9 Frequency (Hz)00.20.40.60.81MeanofAmplitudeDifferenceBenign vs. GS3Benign vs. GS4GS3 vs. GS4(a)0 1 2 3 4 5 6 7 8 9Frequency (Hz)00.20.40.60.81MeanofAmplitudeDifferenceBenign vs. GS3Benign vs. GS4GS3 vs. GS4(b)Figure 6.5: (a) Differences of distributions between cancerous tissues withGleason patterns 3 and 4 as well as benign tissues back projected in theinput neurons corresponds to the first neuron in the third hidden layer; (b)Spectral difference of the simulated TeUS in benign and different cancertissues.features of the third hidden layer, back propagated to the input layer. Theresults of feature visualization suggest that frequencies between 0 â 2 Hzprovide the most discriminative features for distinguishing between cancerousand non-cancerous tissues. The identified frequency range is also consistentwith the ranges that we have observed in our previous independent in vivostudies [75, 76].6.4.2 Simulation ResultsFigure 6.5(b) depicts the spectral difference of the simulated TeUS usingField II, for GS3, GS4 and benign ROIs extracted from digital pathologyslides. The simulation results also show a noticeable amplitude differencesbetween benign and cancerous tissue regions in the lower frequency range.Figure 6.6 depicts the distribution of the power spectrum of TeUS at1 Hz excitation frequency (Fig. 6.6(a)) and its first harmonic (Fig. 6.6(b))across benign, GS3 and GS4 ROIs. The distributions in both frequencies arestatistically significantly different between benign and cancerous ROIs (allp < 0.001 using a paired t-test).1116.5. Conclusion(a) (b)Figure 6.6: (a) Distribution of the power spectrum in the frequency spectrumof simulated TeUS data at the excitation frequency, (b) Distribution of thepower spectrum in the frequency spectrum of simulated TeUS data at thefirst harmonic of the excitation frequency.6.5 ConclusionEvidence derived from our deep learning-based feature visualization pointedto low-frequency components of TeUS as the most informative featuresfor tissue classification. These components potentially represent the effectof pulsation on prostate tissue microstructure. As a result, we simulatedmechanical micro-vibrations of scatterers in phantoms with various scattererdistributions, reflecting benign and cancerous tissue, derived from digitalhistopathology data. We demonstrated that the micro-vibrations of scattererscould be captured by low-frequency spectral features of TeUS, similar to ourin vivo results. These observations together with our previous results suggestthat the distribution and micro-vibration of scatterers could lead to tissuetyping information in TeUS.112Chapter 7Conclusion and Future WorkIt is difficult to say what is impossible, for the dream of yesterday is thehope of today and the reality of tomorrow.â Robert H. GoddardThe ultimate diagnosis of prostate cancer is through histopathologyanalysis of prostate biopsy, guided by either Transrectal Ultrasound (TRUS),or fusion of TRUS with mp-MRI. One million patients in North Americaundergo TRUS-guided prostate biopsy annually. Among this cohort, 70% of10 core biopsies return negative while up to 34% of the positive yield are under-graded. Computer-aided diagnosis models for detection of prostate cancerand guidance of biopsy involve both ultrasound and mp-MRI-based tissuecharacterization. mp-MRI has high sensitivity in detection of prostate lesionsbut low specificity, hence, limiting its utility in detecting disease progressionover time. Ultrasound-based tissue characterization methods focus on theanalysis of texture and spectral features within a single ultrasound frame,Doppler imaging, and elastography. Temporal Enhanced Ultrasound (TeUS),involving a time-series of ultrasound RF/B-mode frames captured frominsonification of tissue over time, has enabled the depiction of patient-specificcancer likelihood maps. Despite promising results in detecting prostatecancer, accurate characterization of aggressive lesions from indolent ones isan open problem and requires refinement.7.1 Conclusion and SummaryIn this thesis, in an effort to improve TeUS-based tissue characterization, wedeveloped methods and algorithms to automate diagnosis of prostate cancerand to accurately characterize aggressive lesions. A deep learning frame-work was devised, deployed and evaluated using TeUS data, to overlay nearreal-time cancer likelihood maps on TRUS images in a clinical environment.The proposed solution encapsulates variability associated with access to rawultrasound signals in commercial scanners. This approach also providescomplementary information about the grade and extent of prostate cancer.1137.1. Conclusion and SummaryThe algorithms and methods were developed using data obtained through astudy that was approved by the ethics review board of the National CancerInstitute (NCI), National Institutes of Health (NIH) in Bethesda, Maryland.In Chapter 2, the clinical process of TeUS data acquisition and establishinghistopathology ground truth for the data is described. Through this thesis,the focus was on both time-domain and spectral-domain representation ofTeUS signals as presented in Chapter 2.In Chapter 3, frameworks were proposed for diagnosis of prostate cancerusing TeUS. In this chapter, both the spectral and temporal aspect of TeUSdata were modeled to find a more accurate technique for prostate cancerdetection. The focus of the chapter was to distinguish between cancerousand normal tissue types represented as a binary classification task. A DBN-based approach was proposed to automatically analyze the spectral aspect oftemporal ultrasound data obtained from 255 cancer foci identified in mp-MRI.Also, deep RNN were proposed to explicitly model the temporal informationin TeUS. By investigating several RNN models, it was demonstrated thatLong Short-Term Memory (LSTM) networks achieve the highest accuracy indetecting prostate cancer in TeUS data. Algorithms were presented for featurevisualization using DBN and LSTM networks. The focus of In Chapter 4 wason detection of higher grade prostate cancer and the problem of separationbetween different prostate grades. An approach was presented to alleviatethe challenge of noisy and sparse labels in building computer-aided diagnosismodels for prostate cancer biopsy guidance. Towards realizing a practicalsolution for prostate cancer diagnosis using TeUS data in the clinical setting,in Chapter 5, the challenge of accessibility of raw RF data in commercialscanners was addressed. A method for knowledge transfer between TeUSRF and B-mode data was proposed. This RNN-based solution also enablednear real-time processing of RF/B-mode TeUS data. Finally, in Chapter 6,the physical phenomena underlying TeUS was investigated. In a hypothesis-generating study, to obtain insight into the physical phenomenon governingthe interaction of temporal ultrasound with tissue, a deep learning basedfeature visualization method was proposed. Evidence derived from deeplearning-based feature visualization pointed to low-frequency components ofTeUS as the most informative features for tissue classification. Simulationswere designed with mechanical micro-vibrations of scatterers in phantomswith various scatterer distributions, reflecting benign and cancerous tissue,derived from digital histopathology data.In conclusion, in this thesis, the aim was to advance TeUS-based tissuecharacterization method and enable more precise prostate biopsy guidance1147.1. Conclusion and Summarycompared to the state of the art. The work aims to help increase the detectionrates of prostate biopsy which ultimately, can lead to the reduction in theneed for repeated biopsies. Results demonstrated that TeUS is effective indifferentiating aggressive prostate cancer from clinically-less-significant dis-ease and non-cancerous tissue. The proposed solutions have the potential toestablish decision support models for patient-specific targeting during biopsyby displaying near real-time cancer likelihood maps on B-mode ultrasoundimages to indicate the presence of cancer.The contributions of this thesis are summarized as follows:â An automatic feature selection framework for spectral analysis of TeUSsignal using DBN is proposed.â Results demonstrated a statistically significant improvement in the ac-curacy of prostate cancer detection in compared to previously publishedstudies using spectral features of TeUS signals [74, 76].â A feature visualization approach is developed to determine the charac-teristics of non-cancerous and cancerous cores in TeUS data and theircorrelation with learned high-level features. This approach is used toidentify the most discriminating frequency components of the timeseries as learned by the classifier.â TeUS data is extensively and explicitly analyzed in temporal domainusing time-dependent probabilistic deep networks for the first time.â Results indicated that temporal analysis of TeUS using RNN canidentify patterns in data that may not be readily observable in spectraldomain analysis, leading to significant improvements in detection ofprostate cancer.â Algorithms were developed for in-depth analysis and visualization ofhigh-level latent features of LSTM-based RNN.â A transformational finding, achieved through this analysis, is that themost discriminating features for detection of prostate cancer can belearned from a fraction of the full TeUS time series. This informationcan be used to optimize TeUS data acquisition for clinical translation.â A novel approach for grading and detection of high-grade prostatecancer using spectral analysis of TeUS with DBN is proposed.1157.1. Conclusion and Summaryâ This approach could successfully differentiate among aggressive prostatecancer (GSâ¥4+3), clinically less significant prostate cancer (GSâ¤3+4),and non-cancerous prostate tissues.â A novel approach is devised for grading and detection of high-gradeprostate cancer using temporal analysis of TeUS with RNN. By encap-sulating proposed ground-truth probability vectors, this solution canalso precisely estimate cancer length in biopsy cores.â The accuracy of tissue characterization is statistically significantlyimproved as compared to previously published studies [10, 12].â A novel strategy is proposed for depiction of patient-specific colormapsfor biopsy guidance including the estimated model uncertainty. Themethod could highlight the possible misguidance in biopsy by usingthe uncertainty measure.â To address limited access to raw ultrasound data on commercial scan-ners, a transfer-learning strategy is developed to enable the probabilisticmodeling of TeUS B-mode data.â The viability of using B-mode TeUS for cancer detection is demon-strated using retrospective data. The initial evaluation indicates thatthe solution is capable of providing guidance information for prostatebiopsy procedures.â Near real-time augmentation of live standard 2D ultrasound imageswith cancer likelihood maps generated from the models is implemented.â A method for interpretation and visualization of the high-level learnedfeatures from TeUS data is presented. Evidence derived from featurevisualization points to low-frequency components of TeUS as the mostinformative features for tissue classification. These components poten-tially represent the effect of pulsation on prostate tissue microstructurein form of micro-vibrations.â The effect of micro-vibration is simulated using a medium with presetelasticity and scatterer locations extracted from 14 whole-mount digitalhistopathology slides.â Results showed that the distribution and micro-vibration of scattererscould lead to tissue typing information in TeUS. This finding is a majorbreakthrough in understanding and technical formulation of TeUS aftera decade.1167.2. Future Work and Suggestions for Further Development7.2 Future Work and Suggestions for FurtherDevelopmentA number of interesting areas of research can be suggested to further optimizeand improve the current solutions as follows:â¢ The immediate envisioned future direction for the current thesis is alarge inter-institution patient study to determine the accuracy of theproposed prostate cancer grading and diagnosis approaches across awide range of patient sub-populations. Access to a larger dataset wouldalso allow to further explore machine learning solutions to address theissue of noisy and sparse labels. These include:â Adding an extra noise layer to the network which adapts thenetwork outputs to match the noisy label distribution. The pa-rameters of this noise layer can be estimated as part of the trainingprocess and involve simple modifications to current training infras-tructures for deep networks [146]. Direct modeling of the noisecorresponding to the label and the ultrasound data may require asmall data set with more reliable labels [47].â Developing multi-instance learning approaches which can com-bine probabilistic modeling at ROI and core levels in the samesolution [158]. For this purpose, the use of multi-instance neuralnetworks is proposed which perform multi-instance learning in anend-to-end manner. These solutions takes bags (here cores) witha various number of instances (here ROIs) as input and directlyoutput the labels of the bags.â¢ A fundamental assumption in previous models for temporal enhancedultrasound is that data are collected in one center with specific equip-ment and image settings. However, center-specific differences in dataexist in form of variations in the imaging equipment, clinician pref-erences for image parameter settings, and the patient pool. Hence,it may not be possible to readily use a model built using data fromone center in other centers. It is recommended to use an automatictransfer learning between different image settings and different centers.The solution can learn discriminant features from ultrasound signalsin the source dataset and use a transfer learning method, similar towhat we discussed in Chapter 5, to take into account the center-specificdifferences.1177.2. Future Work and Suggestions for Further Developmentâ¢ One of the limitations of the current study is that all of the corescome from prostate regions where the presence of cancer is suspectedin mp-MRI. Future work should be focused on the evaluation of themethod for the cases that mp-MRI does not show any suspicious cancerregion. This can be achieved as part of systematic prostate biopsystudies.â¢ There has been limited research over the past few years to combineBayesian models and deep networks for estimation of uncertainty indecision-making models. However, these preliminary methods havelimitations including severe underestimate of model uncertainty [48, 93].Exploring this path and designing probabilistic models that determineâHow uncertain are we about the decision of the model?â is recommended.This concept has an special importance for any computer-aided di-agnosis system. Also, future work can be focused on the analysis ofthe source of the uncertainty and integrate the proper solution in theframework.118Bibliography[1] Mart´ın Abadi et al. Tensorflow: Large-scale machine learning onheterogeneous distributed systems. arXiv preprint arXiv:1603.04467,2016.[2] Jeffrey Abeysekera, Robert Rohling, and Septimiu Salcudean. Vibro-elastography: Absolute elasticity from motorized 3D ultrasound mea-surements of harmonic motion vectors. In International UltrasonicsSymposium (IUS), pages 1â4. IEEE, 2015.[3] Hashim U Ahmed, Ahmed El-Shater Bosaily, et al. Diagnostic ac-curacy of multi-parametric MRI and TRUS biopsy in prostate can-cer (PROMIS): a paired validating confirmatory study. The Lancet,389(10071):815â822, 2017.[4] SK Alam, EJ Feleppa, A Kalisz, S Ramchandran, RD Ennis, Freder-ick L Lizzi, C-S Wuu, and Jeffrey A Ketterling. Prostate elastography:preliminary in vivo results. In Medical Imaging, pages 339â345. Inter-national Society for Optics and Photonics, 2005.[5] Emran Mohammad Abu Anas, Saman Nouranian, S Sara Mahdavi,Ingrid Spadinger, William J Morris, Septimu E Salcudean, ParvinMousavi, and Purang Abolmaesumi. Clinical target-volume delineationin prostate brachytherapy using residual neural networks. In MedicalImage Computing and Computer Assisted Intervention (MICCAI),pages 365â373. Springer, 2017.[6] Peter H Arger, S Bruce Malkowicz, Keith N VanArsdalen, Chandra MSehgal, Anson Holzer, and Susan M Schultz. Color and power dopplersonography in the diagnosis of prostate cancer comparison between vas-cular density and total vascularity. Journal of Ultrasound in Medicine,23(5):623â630, 2004.[7] Hussam Al-Deen Ashab, Nandinee Fariah Haq, Guy Nir, Piotr Ko-zlowski, Peter Black, Edward C Jones, S Larry Goldenberg, Septimiu E119BibliographySalcudean, and Mehdi Moradi. Multimodal classification of prostatetissue: a feasibility study on combining multiparametric MRI and ultra-sound. In SPIE Medical Imaging: Computer-Aided Diagnosis, volume9414, page 94141B. International Society for Optics and Photonics,2015.[8] Shekoofeh Azizi, Sharareh Bayat, Ajay Rajaram, Emran MA Anas,Tamer Mohamed, Konrad Walus, Purang Abolmaesumi, and ParvinMousavi. 3D tissue mimicking biophantoms for ultrasound imaging:bioprinting and image analysis. In SPIE Medical Imaging: Image-Guided Procedures, Robotic Interventions, and Modeling, volume 10576,page 105761T. International Society for Optics and Photonics, 2018.[9] Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi,Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Deeprecurrent neural networks for prostate cancer detection: Analysis oftemporal enhanced ultrasound. IEEE Transactions on Medical Imaging,2018.[10] Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, GuyNir, Jin Tae Kwak, Sheng Xu, Storey Wilson, Kenneth A Iczkowski,M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean, Peter Pinto,Bradford Wood, Purang Abolmaesumi, and Parvin Mousavi. Detectionand grading of prostate cancer using temporal enhanced ultrasound:combining deep neural networks and tissue mimicking simulations.MICCAIâ16 Special Issue: International Journal of Computer AssistedRadiology and Surgery, 12(8):1293â1305, 2017.[11] Shekoofeh Azizi, Farhad Imani, Sahar Ghavidel, Amir Tahmasebi,Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi. Detectionof prostate cancer using temporal sequences of ultrasound data: a largeclinical feasibility study. International Journal of Computer AssistedRadiology and Surgery, 11(6):947â956, 2016.[12] Shekoofeh Azizi, Farhad Imani, Jin Tae Kwak, Amir Tahmasebi, ShengXu, Pingkun Yan, Jochen Kruecker, Baris Turkbey, Peter Choyke, PeterPinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi.Classifying cancer grades using temporal ultrasound for transrectalprostate biopsy. In Medical Image Computing and Computer AssistedIntervention (MICCAI), pages 653â661. Springer, 2016.120Bibliography[13] Shekoofeh Azizi, Farhad Imani, Bo Zhuang, Amir Tahmasebi, Jin TaeKwak, Sheng Xu, Nishant Uniyal, Baris Turkbey, Peter Choyke, PeterPinto, Bradford Wood, Parvin Mousavi, and Purang Abolmaesumi.Ultrasound-based detection of prostate cancer using automatic featureselection with deep belief networks. In Medical Image Computing andComputer Assisted Intervention (MICCAI), pages 70â77. Springer,2015.[14] Shekoofeh Azizi, Parvin Mousavi, Pingkun Yan, Amir Tahmasebi,Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto,Bradford Wood, and Purang Abolmaesumi. Transfer learning from RFto B-mode temporal enhanced ultrasound features for prostate cancerdetection. International Journal of Computer Assisted Radiology andSurgery, 12(7):1111â1121, 2017.[15] Shekoofeh Azizi, Nathan Van Woudenberg, Samira Sojoudi, MingLi, Sheng Xu, Emran M Abu Anas, Pingkun Yan, Amir Tahmasebi,Jin Tae Kwak, Baris Turkbey, Peter Choyke, Peter Pinto, BradfordWood, Parvin Mousavi, and Purang Abolmaesumi. Toward a real-time system for temporal enhanced ultrasound-guided prostate biopsy.International Journal of Computer Assisted Radiology and Surgery,pages 1â9, 2018.[16] Shekoofeh Azizi, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, ShengXu, Baris Turkbey, Peter Choyke, Peter Pinto, Bradford Wood, ParvinMousavi, and Purang Abolmaesumi. Learning from noisy label statis-tics: Detecting high grade prostate cancer in ultrasound guided biopsy.In Medical Image Computing and Computer Assisted Intervention(MICCAI). Springer, 2018.[17] O Basset, Z Sun, JL Mestas, and G Gimenez. Texture analysis ofultrasonic images of the prostate by means of co-occurrence matrices.Ultrasonic Imaging, 15(3):218â237, 1993.[18] Sharareh Bayat, Shekoofeh Azizi, Mohammad I Daoud, Guy Nir,Farhad Imani, Carlos D Gerardo, Pingkun Yan, Amir Tahmasebi, Fran-cois Vignon, Samira Sojoudi, et al. Investigation of physical phenomenaunderlying temporal enhanced ultrasound as a new diagnostic imagingtechnique: Theory and simulations. IEEE Transactions on Ultrasonics,Ferroelectrics, and Frequency Control, 2017.121Bibliography[19] Sharareh Bayat, Farhad Imani, Carlos D Gerardo, Guy Nir, ShekoofehAzizi, Pingkun Yan, Amir Tahmasebi, Storey Wilson, Kenneth AIczkowski, M Scott Lucia, Larry Goldenberg, Septimiu E. Salcudean,Parvin Mousavi, and Purang Abolmaesumi. Tissue mimicking simula-tions for temporal enhanced ultrasound-based tissue typing. In SPIEMedical Imaging, pages 101390Dâ101390D. International Society forOptics and Photonics, 2017.[20] Anthony J Bell and Terrence J Sejnowski. The independent componentsof natural scenes are edge filters. Vision Research, 37(23):3327â3338,1997.[21] Yoshua Bengio. Deep learning of representations for unsupervised andtransfer learning. Unsupervised and Transfer Learning Challenges inMachine Learning, 7:19, 2012.[22] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.Greedy layer-wise training of deep networks. Advances in neuralinformation processing systems, 19:153, 2007.[23] Jeremy Bercoff, Micka¨el Tanter, and Mathias Fink. Supersonic shearimaging: a new technique for soft tissue elasticity mapping. IEEETransactions on Ultrasonics, Ferroelectrics, and Frequency Control,51(4):396â409, 2004.[24] Katharina Boehm, Georg Salomon, Burkhard Beyer, Jonas Schiffmann,Kathrin Simonis, Markus Graefen, and Lars Budaeus. Shear waveelastography for localization of prostate cancer lesions and assessmentof elasticity thresholds: implications for targeted biopsies and activesurveillance protocols. The Journal of Urology, 193(3):794â800, 2015.[25] Frank K Chen, Andre Luis de Castro Abreu, and Suzanne L Palmer.Utility of ultrasound in the diagnosis, treatment, and follow-up ofprostate cancer: State of the art. Journal of Nuclear Medicine, 57(Sup-plement 3):13Sâ18S, 2016.[26] Hao Chen, Qi Dou, Dong Ni, Jie-Zhi Cheng, Jing Qin, Shengli Li, andPheng-Ann Heng. Automatic fetal ultrasound standard plane detectionusing knowledge transferred recurrent neural networks. In MedicalImage Computing and Computer Assisted Intervention (MICCAI),pages 507â514. Springer, 2015.122Bibliography[27] Kyunghyun Cho, Bart Van Merrie¨nboer, Dzmitry Bahdanau, andYoshua Bengio. On the properties of neural machine translation:Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.[28] Franois Chollet. Keras. https://github.com/fchollet/keras, 2015.[29] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and YoshuaBengio. Empirical evaluation of gated recurrent neural networks onsequence modeling. arXiv preprint arXiv:1412.3555, 2014.[30] Sailesh Conjeti, Amin Katouzian, Abhijit Guha Roy, Lo¨ıc Peter, Deb-doot Sheet, Ste´phane Carlier, Andrew Laine, and Nassir Navab. Super-vised domain adaptation of decision forests: Transfer of models trainedin vitro for in vivo intravascular ultrasound tissue characterization.Medical Image Analysis, 32:1â17, 2016.[31] Derek W Cool, Xuli Zhang, Cesare Romagnoli, Jonathan I Izawa,Walter M Romano, and Aaron Fenster. Evaluation of MRIâTRUSfusion versus cognitive registration accuracy for MRI-targeted, TRUS-guided prostate biopsy. American Journal of Roentgenology, 204(1):83â91, 2015.[32] J-M Correas, A-M Tissier, A Khairoune, G Khoury, D Eiss, andO He´le´non. Ultrasound elastography of the prostate: state of the art.Diagnostic and Interventional Imaging, 94(5):551â560, 2013.[33] Jean-Michel Correas, Anne-Marie Tissier, Ahmed Khairoune, ViorelVassiliu, Arnaud Me´jean, Olivier He´le´non, Richard Memo, andRichard G Barr. Prostate cancer: diagnostic performance of real-time shear-wave elastography. Radiology, 275(1):280â289, 2014.[34] JM Correas, A Khairoune, AM Tissier, V Vassiliu, D Eiss, O He´le´non,et al. Trans-rectal quantitative shear wave elastography: applicationto prostate cancerâa feasibility study. In Proceedings of the EuropeanCongress of Radiology, volume 10, 2011.[35] Mohammad I Daoud and James C Lacefield. Three-dimensional com-puter simulation of high-frequency ultrasound imaging of healthy andcancerous murine liver tissues. In SPIE Medical Imaging, pages 79680Hâ79680H. International Society for Optics and Photonics, 2011.[36] Mohammad I Daoud, Parvin Mousavi, Farhad Imani, Robert Rohling,and Purang Abolmaesumi. Tissue classification using ultrasound-123Bibliographyinduced variations in acoustic backscattering features. IEEE Transac-tions on Biomedical Engineering, 60(2):310â320, 2013.[37] Maarten de Rooij, Esther HJ Hamoen, Jurgen J Fu¨tterer, Jelle OBarentsz, and Maroeska M Rovers. Accuracy of multiparametric MRIfor prostate cancer detection: a meta-analysis. American Journal ofRoentgenology, 202(2):343â351, 2014.[38] Sorin M Dudea, Calin R Giurgiu, Dana Dumitriu, Angelica Chiorean,Anca Ciurea, Carolina Botar-Jid, and Ioan Coman. Value of ultrasoundelastography in the diagnosis and management of prostate carcinoma.Medical Ultrasonography, 13(1):45, 2011.[39] Jonathan I Epstein, Zhaoyong Feng, Bruce J Trock, and Phillip MPierorazio. Upgrading and downgrading of prostate cancer from biopsyto radical prostatectomy: incidence and predictive factors using themodified Gleason grading system and factoring in tertiary grades.European Urology, 61(5):1019â1024, 2012.[40] Tom Fawcett. An introduction to roc analysis. Pattern RecognitionLetters, 27(8):861â874, 2006.[41] Andriy Fedorov, Reinhard Beichel, Jayashree Kalpathy-Cramer, JulienFinet, Jean-Christophe Fillion-Robin, Sonia Pujol, Christian Bauer,Dominique Jennings, Fiona Fennessy, and Milan Sonka. 3D slicer asan image computing platform for the quantitative imaging network.Magnetic Resonance Imaging, 30(9):1323â1341, 2012.[42] EJ Feleppa, CR Porter, JA Ketterling, S Dasgupta, S Ramachandran,and D Sparks. Recent advances in ultrasonic tissue-type imaging ofthe prostate. In Acoustical Imaging, pages 331â339. Springer, 2007.[43] Ernest J Feleppa. Ultrasonic tissue-type imaging of the prostate:implications for biopsy and treatment guidance. Cancer Biomarkers,4(4-5):201â212, 2008.[44] Ernest J Feleppa. Imaging the prostate with quantitative ultrasound:Implications for guiding biopsies, targeting focal treatment, and moni-toring therapy. In Prostate Ultrasound, pages 147â161. Springer, 2015.[45] Ernest J Feleppa, Mark J Rondeau, Paul Lee, and Christopher RPorter. Prostate-cancer imaging using machine-learning classifiers:Potential value for guiding biopsies, targeting therapy, and monitoring124Bibliographytreatment. In Ultrasonics Symposium (IUS), pages 527â529. IEEE,2009.[46] Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuyte-laars. Unsupervised visual domain adaptation using subspace align-ment. In IEEE International Conference on Computer Vision, pages2960â2967, 2013.[47] BenoËÄ±t Fre´nay and Michel Verleysen. Classification in the presenceof label noise. IEEE Transactions on Neural Networks and LearningSystems, 25(5):845â869, 2014.[48] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approxi-mation: Representing model uncertainty in deep learning. In MachineLearning, pages 1050â1059, 2016.[49] J-L Gennisson, Thomas Deffieux, Mathias Fink, and Michae¨l Tanter.Ultrasound elastography: principles and techniques. Diagnostic andInterventional Imaging, 94(5):487â495, 2013.[50] Felix A Gers, Ju¨rgen Schmidhuber, and Fred Cummins. Learningto forget: Continual prediction with LSTM. Neural Computation,12(10):2451â2471, 2000.[51] Sahar Ghavidel, Farhad Imani, Siavash Khallaghi, Eli Gibson, AmirKhojaste, Mena Gaed, Madeleine Moussa, Jose A Gomez, D RobertSiemens, and Michael Leveridge. Classification of prostate cancer gradeusing temporal ultrasound: in vivo feasibility study. In SPIE MedicalImaging, pages 97860Kâ97860K. International Society for Optics andPhotonics, 2016.[52] Calin R Giurgiu, Cristian Manea, Nicolae Crisan, Catalina Bungardean,Ioan Coman, and Sorin M Dudea. Real-time sonoelastography in thediagnosis of prostate cancer. Medical Ultrasonography, 13(1):5, 2011.[53] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptationfor large-scale sentiment classification: A deep learning approach. InInternational Conference on Machine Learning, pages 513â520, 2011.[54] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flowkernel for unsupervised domain adaptation. In Computer Vision andPattern Recognition, pages 2066â2073. IEEE, 2012.125Bibliography[55] Daniel W Good, Grant D Stewart, Steven Hammer, Paul Scanlan,Wenmiao Shu, Simon Phipps, Robert Reuben, and Alan S McNeill.Elasticity as a biomarker for prostate cancer: a systematic review. BJUInternational, 113(4):523â534, 2014.[56] T Goossen and H Wijkstra. Transrectal ultrasound imaging andprostate cancer. Archivio italiano di urologia, andrologia, 75(1):68â74,2003.[57] Tjerk EB Goossen, Jean JMCH de la Rosette, Christina A Hulsbergen-van de Kaa, Geert JLH van Leenders, and Hessel Wijkstra. The valueof dynamic contrast enhanced power doppler ultrasound imaging inthe localization of prostate cancer. European Urology, 43(2):124â131,2003.[58] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speechrecognition with deep recurrent neural networks. In Acoustics, Speechand Signal Processing, pages 6645â6649. IEEE, 2013.[59] Seok Min Han, Hak Jong Lee, and Jin Young Choi. Computer-aidedprostate cancer detection using texture features and clinical featuresin ultrasound image. Journal of Digital Imaging, 21(1):121â133, 2008.[60] Wayne R Hedrick, David L Hykes, and Dale E Starchman. Ultrasoundphysics and instrumentation. Elsevier, 2005.[61] Geoffrey Hinton. A practical guide to training restricted Boltzmannmachines. Momentum, 9(1):926, 2010.[62] Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learningalgorithm for deep belief nets. Neural Computation, 18(7):1527â1554,2006.[63] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimen-sionality of data with neural networks. Science, 313(5786):504â507,2006.[64] E Holasek, LA Gans, EW Purnell, and A Sokollu. A method forspectra-color B-scan ultrasonography. Journal of Clinical Ultrasound,3(3):175â178, 1975.[65] Mohammad Honarvar, Robert Rohling, and Septimiu E Salcudean. Acomparison of direct and iterative finite element inversion techniques126Bibliographyin dynamic elastography. Physics in Medicine and Biology, 61(8):3026,2016.[66] Mohammad Honarvar, Septimiu E Salcudean, and Robert N Rohling.Vibro-elastography: direct FEM inversion of the shear wave equa-tion without the local homogeneity assumption. In Medical Imaging:Ultrasonic Imaging and Tomography, volume 9040, page 904003. Inter-national Society for Optics and Photonics, 2014.[67] Yipeng Hu, Marc Modat, Eli Gibson, Nooshin Ghavami, Ester Bon-mati, Caroline M Moore, Mark Emberton, J Alison Noble, Dean CBarratt, and Tom Vercauteren. Label-driven weakly-supervised learn-ing for multimodal deformable image registration. arXiv preprintarXiv:1711.01666, 2017.[68] John W Hunt, Arthur E Worthington, and Andrew T Kerr. Thesubtleties of ultrasound images of an ensemble of cells: simulation fromregular and more random distributions of scatterers. Ultrasound inMedicine and Biology, 21(3):329â341, 1995.[69] John W Hunt, Arthur E Worthington, Andrew Xuan, Michael C Kolios,Gregory J Czarnota, and Michael D Sherar. A model based upon pseudoregular spacing of cells combined with the randomisation of the nucleican explain the significant changes in high-frequency ultrasound signalsduring apoptosis. Ultrasound in Medicine and Biology, 28(2):217â226,2002.[70] KA. Iczkowski, KC. Torkko, GR. Kotnis, RS. Wilson, W. Huang, TM.Wheeler, AM. Abeyta, FG. La Rosa, S. Cook, PN. Werahera, andMS Lucia. Digital quantification of five high-grade prostate cancerpatterns, including the cribriform pattern, and their association withadverse outcome. American Journal of Clinical Pathology, 136(1):98â107, 2011.[71] Kenneth A Iczkowski, Kathleen C Torkko, Gregory R Kotnis,R Storey Wilson, Wei Huang, Thomas M Wheeler, Andrea M Abeyta,Francisco G La Rosa, Shelly Cook, Priya N Werahera, et al. Digitalquantification of five high-grade prostate cancer patterns, includingthe cribriform pattern, and their association with adverse outcome.American Journal of Clinical Pathology, 136(1):98â107, 2011.[72] Farhad Imani, Purang Abolmaesumi, Eli Gibson, Amir Khojaste, MenaGaed, Madeleine Moussa, Jose A Gomez, Cesare Romagnoli, D Robert127BibliographySiemens, Michael Leviridge, Silvia Chang, Aaron Fenster, Aaron DWard, and Parvin Mousavi. Ultrasound-based characterization ofprostate cancer: an in vivo clinical feasibility study. In Medical ImageComputing and Computer Assisted Intervention (MICCAI), pages 279â286. Springer, 2013.[73] Farhad Imani, Purang Abolmaesumi, Eli Gibson, Amir Khojaste, MenaGaed, Madeleine Moussa, D Robert Siemens, Aaron Fenster, AaronWard, and Parvin Mousavi. Computer-aided prostate cancer detectionusing ultrasound RF time series: In vivo feasibility study. IEEETransactions on Medical Imaging, 34(11):2248â2257, 2015.[74] Farhad Imani, Sahar Ghavidel, Purang Abolmaesumi, Siavash Khal-laghi, Eli Gibson, Amir Khojaste, Mena Gaed, Madeleine Moussa,Jose A Gomez, Cesare Romagnoli, et al. Fusion of multi-parametricmri and temporal ultrasound for characterization of prostate cancer:in vivo feasibility study. In Medical Imaging 2016: Computer-AidedDiagnosis, volume 9785, page 97851K. International Society for Opticsand Photonics, 2016.[75] Farhad Imani, Mahdi Ramezani, Saman Nouranian, Eli Gibson, AmirKhojaste, Mena Gaed, Madeleine Moussa, J Gomez, Cesare Romagnoli,M. Leveridge, and S. Chang. Ultrasound-based characterization ofprostate cancer using joint independent component analysis. IEEETransactions on Biomedical Engineering, 62(7):1796â1804, 2015.[76] Farhad Imani, Bo Zhuang, Amir Tahmasebi, Jin Tae Kwak, ShengXu, H. Agarwal, S. Bharat, N. Uniyal, I.B. Turkbey, P. Choyke, andP. Pinto. Augmenting MRI-transrectal ultrasound-guided prostatebiopsy with temporal ultrasound data: a clinical feasibility study.International Journal of Computer Assisted Radiology and Surgery,pages 1â9, 2015.[77] Jørgen Arendt Jensen. Simulation of advanced ultrasound systemsusing field ii. In IEEE International Symposium on Biomedical Imaging:Nano to Macro, pages 636â639. IEEE, 2004.[78] Kazumi Kamoi, Koji Okihara, Atsushi Ochiai, Osamu Ukimura, YoichiMizutani, Akihiro Kawauchi, and Tsuneharu Miki. The utility oftransrectal real-time elastography in the diagnosis of prostate cancer.Ultrasound in Medicine and Biology, 34(7):1025â1032, 2008.128Bibliography[79] Atul Kapoor, Aprajita Kapoor, Goldaa Mahajan, and Bholla SinghSidhu. Real-time elastography in the detection of prostate cancer inpatients with raised psa level. Ultrasound in Medicine and Biology,37(9):1374â1381, 2011.[80] Siavash Khallaghi, C Antonio Sa´nchez, Abtin Rasoulian, Saman Noura-nian, Cesare Romagnoli, Hamidreza Abdi, Silvia D Chang, Peter CBlack, Larry Goldenberg, William J Morris, et al. Statistical biomechan-ical surface registration: application to MRâTRUS fusion for prostateinterventions. IEEE Transactions on Medical Imaging, 34(12):2535â2549, 2015.[81] Siavash Khallaghi, C Antonio Sa´nchez, Abtin Rasoulian, Yue Sun,Farhad Imani, Amir Khojaste, Orcun Goksel, Cesare Romagnoli,Hamidreza Abdi, Silvia Chang, et al. Biomechanically constrainedsurface registration: Application to MRâTRUS fusion for prostate in-terventions. IEEE Transactions on Medical Imaging, 34(11):2404â2414,2015.[82] Amir Khojaste, Farhad Imani, Mehdi Moradi, David Berman, D RobertSiemens, Eric E Sauerberi, Alexander H Boag, Purang Abolmaesumi,and Parvin Mousavi. Characterization of aggressive prostate cancerusing ultrasound RF time series. In SPIE Medical Imaging, pages94141Aâ94141A. International Society for Optics and Photonics, 2015.[83] Diederik Kingma and Jimmy Ba. Adam: A method for stochasticoptimization. arXiv preprint arXiv:1412.6980, 2014.[84] Bin Kong, Yiqiang Zhan, Min Shin, Thomas Denny, and ShaotingZhang. Recognizing end-diastole and end-systole frames via deep tem-poral regression network. In Medical Image Computing and ComputerAssisted Intervention (MICCAI), pages 264â272. Springer, 2016.[85] Katharina KO¨nig, Ulrich Scheipers, Andreas Pesavento, AndreasLorenz, Helmut Ermert, and Theodor Senge. Initial experiences withreal-time elastography guided biopsies of the prostate. The Journal ofUrology, 174(1):115â117, 2005.[86] Sergey Kravchick, Shmuel Cytron, Ronit Peled, Daniel London, YosefSibi, and David Ben-Dor. Optimal combinations for detection ofprostate cancer: systematic sextant and laterally directed biopsiesversus systematic sextant and color doppler-targeted biopsies. Urology,63(2):301â305, 2004.129Bibliography[87] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenetclassification with deep convolutional neural networks. In Advances inNeural Information Processing Systems, pages 1097â1105, 2012.[88] Thomas A Krouskop, Thomas M Wheeler, Faouzi Kallel, Brian SGarra, and Timothy Hall. Elastic moduli of breast and prostate tissuesunder compression. Ultrasonic Imaging, 20(4):260â274, 1998.[89] Timur H Kuru, Matthias C Roethke, Jonas Seidenader, TobiasSimpfendo¨rfer, Silvan Boxler, K. Alammar, P. Rieker, V.I. Popeneciu,W. Roth, S. Pahernik, and H.P. Schlemmer. Critical evaluation ofmagnetic resonance imaging targeted, transrectal ultrasound guidedtransperineal fusion biopsy for detection of prostate cancer. The Jour-nal of Urology, 190(4):1380â1386, 2013.[90] Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra,and Yoshua Bengio. An empirical evaluation of deep architectures onproblems with many factors of variation. In International Conferenceon Machine Learning, pages 473â480. ACM, 2007.[91] Andras Lasso, Tamas Heffter, Adam Rankin, Csaba Pinter, Tamas Ungi,and Gabor Fichtinger. PLUS: open-source toolkit for ultrasound-guidedintervention systems. IEEE Transactions on Biomedical Engineering,61(10):2527â2537, 2014.[92] Saying Li, Min Chen, Wenchao Wang, Weifeng Zhao, Jianye Wang,Xuna Zhao, and Cheng Zhou. A feasibility study of mr elastography inthe diagnosis of prostate cancer at 3.0 t. Acta Radiologica, 52(3):354â358, 2011.[93] Yingzhen Li and Yarin Gal. Dropout inference in bayesian neuralnetworks with alpha-divergences. arXiv preprint arXiv:1703.02914,2017.[94] Shu Liao, Yaozong Gao, Aytekin Oto, and Dinggang Shen. Repre-sentation learning: a unified deep learning framework for automaticprostate mr segmentation. In Medical Image Computing and ComputerAssisted Intervention (MICCAI), pages 254â261. Springer, 2013.[95] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, ArnaudArindra Adiyoso Setio, et al. A survey on deep learning in medi-cal image analysis. arXiv preprint arXiv:1702.05747, 2017.130Bibliography[96] Frederic L Lizzi, Ernest J Feleppa, S Kaisar Alam, and Cheri X Deng.Ultrasonic spectrum analysis for tissue evaluation. Pattern RecognitionLetters, 24(4):637â658, 2003.[97] Frederic L Lizzi, Michael Greenebaum, Ernest J Feleppa, Marek El-baum, and D Jackson Coleman. Theoretical framework for spectrumanalysis in ultrasonic tissue characterization. The Journal of the Acous-tical Society of America, 73(4):1366â1373, 1983.[98] Rafael Llobet, Juan C Pe´rez-Corte´s, Alejandro H Toselli, and AlfonsJuan. Computer-aided detection of prostate cancer. internationalJournal of Medical Informatics, 76(7):547â556, 2007.[99] Julio Lobo, Ali Baghani, Hani Eskandari, Sara Mahdavi, RobertRohling, Larry Goldernberg, William James Morris, and SeptimiuSalcudean. Prostate vibro-elastography: Multi-frequency 1D over 3Dsteady-state shear wave imaging for quantitative elastic modulus mea-surement. In International Ultrasonics Symposium (IUS), pages 1â4.IEEE, 2015.[100] Carmen Maccagnano and Vincenzo Scattoni. HistoScanning. In Atlas ofUltrasonography in Urology, Andrology, and Nephrology, pages 597â604.Springer, 2017.[101] Petr Macek, Eric Barret, Rafael Sanchez-Salas, Marc Galiano, FrancoisRozet, Youness Ahallal, Joseph M Gaya, Matthieu Durant, LaurentMascle, Camilo Giedelman, et al. Prostate histoscanning in clinicallylocalized biopsy proven prostate cancer: an accuracy study. Journal ofEndourology, 28(3):371â376, 2014.[102] Petr Macek, Eric Barret, Arjun Sivaraman, Rafael Sanchez-Salas, MarcGaliano, Francois Rozet, et al. Prostate histoscanning true targetingguided prostate biopsy: initial clinical experience. World Journal ofUrology, 33(10):1475â1479, 2015.[103] Leonard Marks, Shelena Young, and Shyam Natarajan. MRI-ultrasoundfusion for guidance of targeted prostate biopsy. Current opinion inurology, 23(1):43, 2013.[104] Oliver Mattausch and Orcun Goksel. Image-based psf estimationfor ultrasound training simulation. In International Workshop onSimulation and Synthesis in Medical Imaging, pages 23â33. Springer,2016.131Bibliography[105] Hengameh Mirzaalian, Lipeng Ning, Peter Savadjiev, Ofer Pasternak,Sylvain Bouix, O Michailovich, G Grant, CE Marx, Rajendra A Morey,LA Flashman, et al. Inter-site and inter-scanner diffusion MRI dataharmonization. NeuroImage, 135:311â323, 2016.[106] Omid Mohareri, Angelica Ruszkowski, Julio Lobo, Joseph Ischia, AliBaghani, Guy Nir, Hani Eskandari, Edward Jones, Ladan Fazli, LarryGoldenberg, et al. Multi-parametric 3D quantitative ultrasound vibro-elastography imaging for detecting palpable prostate tumors. In MedicalImage Computing and Computer Assisted Intervention (MICCAI),pages 561â568. Springer, 2014.[107] Majid Mohrekesh, Shekoofeh Azizi, and Shadrokh Samavi. AcceleratingGPU implementation of Contourlet transform. In Machine Vision andImage Processing (MVIP), 2013 8th Iranian Conference on, pages328â332. IEEE, 2013.[108] Mehdi Moradi, Purang Abolmaesumi, AH Boag, Eric E Sauerbrei,DR Siemens, and Parvin Mousavi. Augmenting detection of prostatecancer in transrectal ultrasound images using SVM and RF time series.IEEE Transactions on Biomedical Engineering, 56(9):2214â2224, 2009.[109] Mehdi Moradi, Purang Abolmaesumi, and Parvin Mousavi. Tissuetyping using ultrasound RF time series: Experiments with animaltissue samples. Medical Physics, 37(8):4401â4413, 2010.[110] Mehdi Moradi, S Sara Mahdavi, Guy Nir, Edward C Jones, S LarryGoldenberg, and Septimiu E Salcudean. Ultrasound RF time series fortissue typing: first in vivo clinical results. In SPIE Medical Imaging,pages 86701Iâ86701I. International Society for Optics and Photonics,2013.[111] Mehdi Moradi, S Sara Mahdavi, Guy Nir, Omid Mohareri, A. Koup-paris, L.O. Gagnon, L. Fazli, R.G. Casey, J. Ischia, E.C. Jones, and S.L.Goldenberg. Multiparametric 3D in vivo ultrasound vibroelastographyimaging of prostate cancer: Preliminary results. Medical Physics, 41(7),2014.[112] Mehdi Moradi, Parvin Mousavi, and Purang Abolmaesumi. Computer-aided diagnosis of prostate cancer with emphasis on ultrasound-basedapproaches: a review. Ultrasound in Medicine and Biology, 33(7):1010â1028, 2007.132Bibliography[113] Mehdi Moradi, Parvin Mousavi, Philip A Isotalo, David R Siemens,Eric E Sauerbrei, and Purang Abolmaesumi. A new approach to anal-ysis of RF ultrasound echo signals for tissue characterization: animalstudies. In SPIE Medical Imaging, pages 65130Pâ65130P. InternationalSociety for Optics and Photonics, 2007.[114] Aaron Moskalik, Paul L Carson, Jonathan M Rubin, Robert L Bree,J Brian Fowlkes, Mark A Rubin, Kirk Wojno, Sargum Manley, andJames E Montie. Analysis of three-dimensional ultrasound doppler forthe detection of prostate cancer. Urology, 57(6):1128â1132, 2001.[115] Layan Nahlawi, Caroline Goncalves, Farhad Imani, Mena Gaed, Jose AGomez, et al. Models of temporal enhanced ultrasound data for prostatecancer diagnosis: The impact of time-series order. In SPIE MedicalImaging, pages 101351Dâ101351D. International Society for Optics andPhotonics, 2017.[116] Layan Nahlawi, Caroline Goncalves, Farhad Imani, Mena Gaed, Jose AGomez, et al. Stochastic modeling of temporal enhanced ultrasound:Impact of temporal properties on prostate cancer characterization.IEEE Transactions on Biomedical Engineering (TBME), 2017.[117] Layan Nahlawi, Farhad Imani, Mena Gaed, Jose Gomez, MadeleineMoussa, Eli Gibson, Aaron Fenster, Aaron Ward, Purang Abolmae-soumi, Parvin Mousavi, et al. Using hidden markov models to capturetemporal aspects of ultrasound data in prostate cancer. In EEE In-ternational Conference on Bioinformatics and Biomedicine (BIBM),2015.[118] Layan Nahlawi, Farhad Imani, Mena Gaed, Jose A Gomez, MadeleineMoussa, et al. Prostate cancer: Improved tissue characterization bytemporal modeling of radio-frequency ultrasound echo data. In MedicalImage Computing and Computer Assisted Intervention (MICCAI),pages 644â652. Springer International Publishing, 2016.[119] Shyam Natarajan, Leonard S Marks, Daniel JA Margolis, Jiaoti Huang,Maria Luz Macairan, Patricia Lieu, and Aaron Fenster. Clinicalapplication of a 3D ultrasound-guided prostate biopsy system. InUrologic Oncology: Seminars and Original Investigations, volume 29,pages 334â342. Elsevier, 2011.[120] Eric D Nelson, Craig B Slotoroff, Leonard G Gomella, and Ethan JHalpern. Targeted biopsy of the prostate: The impact of color Doppler133Bibliographyimaging and elastography on prostate cancer detection and Gleasonscore. Urology, 70(6):1136â1140, 2007.[121] Kathryn R Nightingale, Mark L Palmeri, Roger W Nightingale, andGregg E Trahey. On the feasibility of remote palpation using acousticradiation force. The Journal of the Acoustical Society of America,110(1):625â634, 2001.[122] Guy Nir, Ramin S Sahebjavaher, Piotr Kozlowski, Silvia D Chang,Ralph Sinkus, S Larry Goldenberg, and Septimiu E Salcudean. Model-based registration of ex vivo and in vivo MRI of the prostate usingelastography. IEEE Transactions on Medical Imaging, 32(7):1349â1361,2013.[123] J Alison Noble. Ultrasound image segmentation and tissue characteri-zation. Proceedings of the Institution of Mechanical Engineers, PartH: Journal of Engineering in Medicine, 224(2):307â316, 2010.[124] Yngve NygËard, Svein A Haukaas, Ole J Halvorsen, Karsten Gravdal,Jannicke FrugËard, Lars A Akslen, and Christian Beisland. A positivereal-time elastography is an independent marker for detection of high-risk prostate cancers in the primary biopsy setting. BJU International,113(5b), 2014.[125] Can O¨bek, Tu¨nkut DogËanca, Sinan Erdal, Sarper ErdogËan, and HaydarDurak. Core length in prostate biopsy: size matters. The Journal ofUrology, 187(6):2051â2055, 2012.[126] Michael L Oelze and Jonathan Mamou. Review of quantitative ul-trasound: Envelope statistics and backscatter coefficient imaging andcontributions to diagnostic ultrasound. IEEE Transactions on Ultra-sonics, Ferroelectrics, and Frequency Control, 63(2):336â351, 2016.[127] Michael L Oelze, William D OâBrien, James P Blue, and James FZachary. Differentiation and characterization of rat mammary fibroade-nomas and 4T1 mouse carcinomas using quantitative ultrasound imag-ing. IEEE Transactions on Medical Imaging, 23(6):764â771, 2004.[128] Michael L Oelze and William D OâBrien Jr. Method of improvedscatterer size estimation and application to parametric imaging us-ing ultrasound. The Journal of the Acoustical Society of America,112(6):3053â3063, 2002.134Bibliography[129] BJ Oosterveld, JM Thijssen, and WA Verhoef. Texture of B-modeechograms: 3-d simulations and experiments of the effects of diffractionand scatterer density. Ultrasonic Imaging, 7(2):142â160, 1985.[130] Leo Pallwein, Michael Mitterberger, Peter Struve, Germar Pinggera,Wolfgang Horninger, Georg Bartsch, Friedrich Aigner, Andreas Lorenz,Florian Pedross, and Ferdinand Frauscher. Real-time elastography fordetecting prostate cancer: preliminary experience. BJU International,100(1):42â46, 2007.[131] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEETransactions on Knowledge and Data Engineering, 22(10):1345â1359,2010.[132] Tobias Penzkofer, Kemal Tuncali, Andriy Fedorov, Sang-Eun Song,Junichi Tokuda, Fiona M Fennessy, Mark G Vangel, Adam S Kibel,Robert V Mulkern, William M Wells, et al. Transperineal in-bore 3-tmr imagingâguided prostate biopsy: a prospective clinical observationalstudy. Radiology, 274(1):170â180, 2014.[133] TC Potdevin, AP Moskalik, JB Fowlkes, RO Bude, and PL Carson.Doppler quantitative measures by region to discriminate prostate cancer.Ultrasound in Medicine and Biology, 27(10):1305â1310, 2001.[134] Elisabetta Rapiti, Robin Schaffar, Christophe Iselin, Raymond Miral-bell, Marie-Franc¸oise Pelte, Damien Weber, Roberto Zanetti, IsabelleNeyroud-Caspar, and Christine Bouchardy. Importance and deter-minants of Gleason score undergrading on biopsy sample of prostatecancer in a populationâbased study. BMC Urology, 13(1):19, 2013.[135] Ali Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carls-son. Cnn features off-the-shelf: an astounding baseline for recognition.In Computer Vision and Pattern Recognition Workshops, pages 806â813, 2014.[136] Ramin S Sahebjavaher, Guy Nir, Mohammad Honarvar, Louis OGagnon, Joseph Ischia, Edward C Jones, Silvia D Chang, Ladan Fazli,S Larry Goldenberg, Robert Rohling, et al. MR elastography of prostatecancer: quantitative comparison with histopathology and repeatabilityof methods. NMR in Biomedicine, 28(1):124â139, 2015.[137] Ramin Sebastian Sahebjavaher, Samuel Frew, Artem Bylinskii, LeonBeek, Philippe Garteiser, Mohammad Honarvar, Ralph Sinkus, and135BibliographySeptimiu Salcudean. Prostate MR elastography with transperinealelectromagnetic actuation and a fast fractionally encoded steady-stategradient echo sequence. NMR in Biomedicine, 27(7):784â794, 2014.[138] Septimiu E Salcudean, Daniel French, Simon Bachmann, Reza Zahiri-Azar, Xu Wen, and William James Morris. Viscoelasticity modelingof the prostate region using vibro-elastography. In Medical ImageComputing and Computer Assisted Intervention (MICCAI), pages 389â396. Springer, 2006.[139] Georg Salomon, Jens Ko¨llerman, Imke Thederan, Felix KH Chun, LarsBuda¨us, Thorsten Schlomm, Hendrik Isbarn, Hans Heinzer, HartwigHuland, and Markus Graefen. Evaluation of prostate cancer detectionwith ultrasound real-time elastography: a comparison with step sectionpathological analysis after radical prostatectomy. European Urology,54(6):1354â1362, 2008.[140] U Scheipers, K Ko¨nig, H-J Sommerfeld, M Garcia-Schu¨rmann, T Senge,and H Ermert. Sonohistologyâultrasonic tissue characterization forprostate cancer diagnostics. Cancer Biomarkers, 4(4, 5):227â250, 2008.[141] Georg Schmitz, Helmut Ermert, and Theodor Senge. Tissue-characterization of the prostate using RF ultrasonic signals. IEEETransactions on Ultrasonics, Ferroelectrics, and Frequency Control,46(1):126â138, 1999.[142] Jose´ Seabra and Joao Miguel Sanches. RF ultrasound estimation fromB-mode images. In Ultrasound Imaging, pages 3â24. Springer, 2012.[143] Hoo-Chang Shin, Holger R Roth, Mingchen Gao, Le Lu, Ziyue Xu,Isabella Nogues, Jianhua Yao, Daniel Mollura, and Ronald M Summers.Deep convolutional neural networks for computer-aided detection: Cnnarchitectures, dataset characteristics and transfer learning. IEEETransactions on Medical Imaging, (99):1â1, 2016.[144] M Minhaj Siddiqui, Soroush Rais-Bahrami, Baris Turkbey, Arvin KGeorge, Jason Rothwax, Nabeel Shakir, Chinonyerem Okoro, DimaRaskolnikov, Howard L Parnes, W Marston Linehan, et al. Comparisonof MR/US fusionâguided biopsy with US-guided biopsy for the diagnosisof prostate cancer. Jama, 313(4):390â397, 2015.136Bibliography[145] Eric A Singer, Aradhana Kaushal, et al. Active surveillance forprostate cancer: past, present and future. Current Opinion in Oncology,24(3):243â250, 2012.[146] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev,and Rob Fergus. Training convolutional networks with noisy labels.arXiv preprint arXiv:1406.2080, 2014.[147] Masayuki Tanaka and Masatoshi Okutomi. A novel inference of arestricted Boltzmann machine. In International Conference on PatternRecognition, pages 1526â1531. IEEE, 2014.[148] Jie Tang, Song Li, Junlai Li, Yukun Luo, Jianhong Xu, Yan Zhang,Xin Li, Huaiyin Shi, and Gaokui Zhang. Correlation between prostatecancer grade and vascularity on color doppler imaging: preliminaryfindings. Journal of Clinical Ultrasound, 31(2):61â68, 2003.[149] Farheen Taquee, Orcun Goksel, S Sara Mahdavi, Mira Keyes, W JamesMorris, Ingrid Spadinger, and Septimiu Salcudean. Deformable prostateregistration from MR and TRUS images using surface error driven FEMmodels. In Medical Imaging 2012: Image-Guided Procedures, RoboticInterventions, and Modeling, volume 8316, page 831612. InternationalSociety for Optics and Photonics, 2012.[150] Junichi Tokuda, Gregory S Fischer, Xenophon Papademetris, Ziv Yaniv,Luis Ibanez, Patrick Cheng, Haiying Liu, Jack Blevins, Jumpei Arata,and Alexandra J Golby. OpenIGTLink: an open network protocolfor image-guided therapy environment. The International Journal ofMedical Robotics and Computer Assisted Surgery, 5(4):423â434, 2009.[151] Po-Hsiang Tsui and Chien-Cheng Chang. Imaging local scattererconcentrations by the Nakagami statistical model. Ultrasound inMedicine and Biology, 33(4):608â619, 2007.[152] Masakazu Tsutsumi, Tomoaki Miyagawa, Takeshi Matsumura, Nat-sui Kawazoe, Satoru Ishikawa, Tatsuro Shimokama, Tsuyoshi Shiina,Naoto Miyanaga, and Hideyuki Akaza. The impact of real-time tissueelasticity imaging (elastography) on the detection of prostate cancer:clinicopathological analysis. International Journal of Clinical Oncology,12(4):250â255, 2007.[153] Baris Turkbey, Haresh Mani, Omer Aras, Jennifer Ho, Anthony Hoang,A.R. Rastinehad, H. Agarwal, V. Shah, M. Bernardo, Y. Pang, and137BibliographyD. Daar. Prostate cancer: Can mp-MR imaging help identify patientswho are candidates for active surveillance? Radiology, 268(1):144â152,2013.[154] Nishant Uniyal, Farhad Imani, Amir Tahmasebi, Harsh Agarwal,Shyam Bharat, Pingkun Yan, et al. Ultrasound-based predicationof prostate cancer in MRI-guided biopsy. In Medical Image Computingand Computer Assisted Intervention-Workshop on Clinical Image-BasedProcedures, pages 142â150. Springer, 2014.[155] Arna van Engelen, Anouk C van Dijk, Martine TB Truijman, Ronaldvanât Klooster, Annegreet van Opbroek, Aad van der Lugt, Wiro JNiessen, M Eline Kooi, and Marleen de Bruijne. Multi-center MRIcarotid plaque component segmentation using feature normaliza-tion and transfer learning. IEEE Transactions on Medical Imaging,34(6):1294â1305, 2015.[156] Annegreet Van Opbroek, M Arfan Ikram, Meike W Vernooij, andMarleen De Bruijne. Transfer learning improves supervised imagesegmentation across imaging protocols. IEEE Transactions on MedicalImaging, 34(5):1018â1030, 2015.[157] Gijs van Tulder and Marleen de Bruijne. Learning features for tissueclassification with the classification restricted boltzmann machine. InMedical Computer Vision: Algorithms for Big Data, pages 47â58.Springer, 2014.[158] Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and WenyuLiu. Revisiting multiple instance neural networks. Pattern Recognition,74:15â24, 2018.[159] M Watanabe, K Ohnishi, H Hayami, et al. Study on blood flowimages in prostatic cancer during therapy by 2d-doppler flow mapping.Japanese Journal of Medical Ultrasound, 17(1):443â444, 1990.[160] Shao Wei Xie, Hong Li Li, Jing Du, Jian Guo Xia, Yi Fen Guo,Mei Xin, and Feng Hua Li. Influence of serum PSA level, prostatevolume, and PSA density on prostate cancer detection with contrast-enhanced sonography using contrast-tuned imaging technology. Journalof Ultrasound in Medicine, 32(5):741â748, 2013.138[161] Lei Xu and Michael I Jordan. On convergence properties of the emalgorithm for gaussian mixtures. Neural Computation, 8(1):129â151,1996.[162] Sheng Xu, Jochen Kruecker, Baris Turkbey, Neil Glossop, Anurag KSingh, P. Choyke, P. Pinto, and B.J. Wood. Real-time MRI-TRUSfusion for guidance of targeted prostate biopsies. Computer AidedSurgery, 13(5):255â264, 2008.[163] Man Zhang, Priya Nigwekar, Benjamin Castaneda, Kenneth Hoyt,Jean V Joseph, Anthony di SantâAgnese, Edward M Messing, John GStrang, Deborah J Rubens, and Kevin J Parker. Quantitative charac-terization of viscoelastic properties of human prostate correlated withhistology. Ultrasound in Medicine and Biology, 34(7):1033â1042, 2008.[164] Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and QingHe. Supervised representation learning: transfer learning with deepautoencoders. In International Conference on Artificial Intelligence,pages 4119â4125. AAAI Press, 2015.139Appendix ATheoretical Background ofTemporal EnhancedUltrasoundThe characteristic model for the formulation of RF backscattered ultrasoundsignal can be expressed as [104, 128]I(x) = P (x) â S(x) + n, (A.1)where P (x), ultrasound Point Spread Function (PSF) at an arbitrary pointx, is convolved with S(x), the tissue scattering function at the same point,to generate I(x), the backscattered RF data at point (x). In this equation,n represents random noise.If we assume that (S(x)) varies locally as a function of time, a simplemodel of this effect at point x0 can be expressed as S(x0 + f(t)), where f(t)is a time varying function and t is the âslow timeâ (i.e., frame number). Forrelatively small f(t), the first order approximation of Taylor expansion of Scan be written asS(x0 + f(t)) = S(x0) +âS(x)âxâ£â£â£â£x=x0f(t), (A.2)where âSâx (x0) is the local change in the scattering function about point x0.As stated in the Chapter 6, we previously investigated a scenario wherevariations in f(t) were primarily due to induced thermal effects as a resultof acoustic absorption [36]. In this thesis, we investigate an alternativehypothesis, where variations in f(t) are a result of tissue micro vibrationsdue to physiological vibration. Let f(t) be a sinusoidal function of the form:f(t) = a(x0) sin(Ït), (A.3)where a(x0) is the amplitude of the micro vibration at point x0, and Ï is thefrequency of vibration. In a fully elastic tissue, a(x0) is inversely proportional140Appendix A. Theoretical Background of Temporal Enhanced Ultrasoundto E(Ï), the local elasticity, which is frequency dependent. In a viscoelasticmedium, a(x0) is inversely proportional to a function of E(Ï) and µ(Ï),where µ(Ï) is the local viscosity, which is also frequency dependent [163].Equation (A.2) can be rewritten asS(x0 + a(x0) sin(Ït)) = S(x0) + a(x0)âS(x)âxâ£â£â£â£x=x0sin(Ït). (A.4)Combining Equations (A.1) and (A.4), we have:I(x0, t) = (P (x) â S(x))â£â£â£â£x=x0+a(x0) (P (x) â âS(x)âx)â£â£â£â£x=x0sin(Ït) + n.(A.5)The first term of Equation (A.5) corresponds to the time-invariant com-ponent of the RF signal received from point x0. This component dependsonly on the spatial variations of the backscattering function across the propa-gation medium. It can be characterized using conventional analysis of the RFspectrum and B-mode texture. The second term corresponds to time-varyingcomponents of I(x0, t) affected by local variations in the backscatteringfunction, in slow time. Such local variations represent changes in tissuestructure, such as changes in nuclear configuration.Two important observations can be made about the second term ofthis equation: 1) In media with the same mechanical properties E and µ,spectral analysis of TeUS captures (P (x) â âS(x)âx )â£â£â£â£x=x0, which is related tospatial variaitons in the scattering function. This property can be of benefitto characterize, e.g., tissue at early stage cancer, where changes in nucleiconfiguration could dominate changes in tissue property; 2) Where thereare changes in mechanical properties, a(x0), and the scattering function, S,or the vibration frequency, Ï, TeUS captures a combined effect for tissuecharacterization.Through a series of simulations [19], we demonstrate that local changesin tissue properties, captured by a(x0) (P (x)â âS(x)âx )â£â£â£â£x=x0using, e.g., Fouriertransform of TeUS, are effective features for tissue characterization. Oursimulations include media with synthetic array of scatterers of varying dis-tances and pathology mimicking simulations based on whole-mount prostate141Appendix A. Theoretical Background of Temporal Enhanced Ultrasounddigital histopathology data. These simulations confirm our observations fromEquation (A.5) that as a result of micro vibration in the medium, TeUS candifferentiate tissue types with subtle variations in arrangement of scatterers.142Appendix BTeUS Biopsy GuidanceSystem ImplementationIn this part, we present the details of our unified software framework demon-strating real-time analysis of ultrasound data stream using a deep learningsolution. The system integrates cutting-edge machine learning softwarelibraries with 3D Slicer [41] and the âPublic software Library for UltraSoundimaging research (PLUS)â [91] to build an accessible platform. To the bestof our knowledge, this is the first system of its kind in the literature. Thisis the very first demonstration of automatic, real-time prostate segmenta-tion in ultrasound in the literature. The proposed software system allowsfor depiction of live 2D US images augmented with patient-specific cancerlikelihood maps that have been calculated from TeUS.B.1 TeUS biopsy guidance systemThe overview of the components of the guidance system is given in Fig. B.1and Fig. B.2. To allow for continuous localization of likely-cancerous tissuein US data, the architecture incorporates state-of-the-art open-source soft-ware libraries for US data streaming, data visualization, and deep learning.A client-server approach allows running computationally expensive algo-rithms [107] simultaneously, and in real-time. The system has a three-tieredarchitecture as seen in Fig. B.2: US-machine layer, TeUS-client, and TeUS-server. US-machine layer acquires data and streams it to the TeUS-clientlayer. TeUS-client is a 3D Slicer [41] extension responsible for US datamanagement, pre-processing and visualization. Ultrasound B-mode data, orB-mode and Radio Frequency (RF) data if both available, are streamed tothe TeUS-server for tissue characterization and prostate localization, andreceived back by the TeUS-client for real-time displaying in 3D Slicer. TheTeUS-server receives the US data through an OpenIGTLink network pro-tocol [150], performs segmentation for continuous tracking and localizationof the prostate, and computes cancer likelihood maps that are transferred143B.1. TeUS biopsy guidance systemFigure B.1: Overview of the biopsy guidance system. The three steps inthe guidance workflow are volume acquisition, classification and guidance.A client-server approach allows for simultaneous and real-time execution ofcomputationally expensive algorithms including TeUS data classification,and prostate boundary segmentation.back to the TeUS-client through a second OpenIGTLink. The followingsub-sections describe details of TeUS-client and TeUS-server.B.1.1 TeUS-clientThe TeUS-clientâs primary tasks are receiving streamed B-mode data andgenerating a TeUS sequence, preprocessing the data to divide it to smallerRegions of Interest (ROIs), and visualizing the cancer likelihood maps overlaidon US data. Data from ROIs are sent to the TeUS-server for analysisthrough the machine learning framework. TeUS-client receives the results asa colormap and segmented prostate boundary, and overlays this informationon the US image. The TeUS-client includes a custom C++ loadable module,TeUS guidance module, created as an extension to 3D Slicer (Fig. B.2).Ultrasound data acquisition: Once the TeUS guidance module is started,144B.1. TeUS biopsy guidance systemFigure B.2: The software system has a three-tiered architecture. Ovalsrepresent processing elements while arrows show the direction of data flow.In the US machine layer, PLUS is responsible for US data acquisition andcommunicates with the TeUS-client via the OpenIGTLink protocol. TheTeUS client layer includes TeUS guidance, an extension module within the3D Slicer framework. The TeUS-server layer is responsible for the simulta-neous and real-time execution of computationally expensive algorithms andcommunicates with TeUS-client via the OpenIGTLink protocol.an instance of PLUS [91] is initiated on the US-machine layer by runningthe PlusServer application. Next, an OpenIGTLink receiver is created thatcontinuously listens to the PlusServer to receive the US data ( Fig. B.2).Upon successful connection, the US data are displayed on the 3D Slicerwindow. Once the user issues a âStartâ signal, data acquisition is initiatedby buffering streamed B-mode images, followed by preprocessing. Dataacquisition is halted by the user through a âStopâ signal.Preprocessing: Once the data acquisition begins, a independent prepro-cessing thread is initiated to avoid freezing 3D Slicer while the TeUS ROIsare being generated. In this thread, each US image is divided to equally-sizedROIs of 0.5 mmà 0.5 mm based on the scan conversion parameters. For theith ROI, a sequence of TeUS data, x(i) = (x(i)1 , , ..., x(i)T ), T = 100 frames, isgenerated by averaging all time series within that ROI and subtracting themean value from the given time series.Communication with the TeUS-server: Following the completion ofthe preprocessing thread, the TeUS guidance module sends data from theextracted ROIs to the TeUS-server layer, using a standard OpenIGTLink145B.1. TeUS biopsy guidance systemprotocol and by forking a sender thread. In addition to this thread, theTeUS-client layer also forks a receiver thread, which creates a new connectionwith the TeUS-server using the OpenIGTLink protocol. The receiver threadthen waits for a message containing the resulting cancer likelihood colormapas well as the prostate localization information from the TeUS-server.Receiving and visualizing the cancer likelihood colormap: Upon thegeneration of results and receiving the output message from the TeUS-server,the TeUS-clientâs receiver thread picks up the TeUS moduleâs execution. Theguidance colormap and prostate boundary segmentation results are savedand 3D Slicer thread begins overlaying the information. During the guidancecolormap visualization, the segmentation information is used to localize theprostate and mask out any colormap data falling outside the boundary. Theboundary matrix is resized down to the colormapâs dimensions. Then, theguidance colormap is converted from single channel float values (ranging from0 to 1) to 3-channel RGB data (ranging from 0 to 255), with 0 being pureBlue and 1 being pure Red (The Green channel is always 0). The boundarymask and processed colormap are multiplied together, masking out any dataoutside the prostate boundary.B.1.2 TeUS ServerThe integrated system encapsulates a machine learning framework where wespecifically use two deep learning methods, implemented in Tensorflow [1].These deep networks are responsible to identify target locations using TeUSdata and a state-of-the-art automatic prostate boundary segmentation tech-nique is used to localize the prostate boundary during the prostate biopsyguidance. The TeUS-server is mainly responsible for simultaneous and real-time execution of computationally expensive deep learning models. TheTeUS-server loads, prepares and runs a Tensorflow graph [1] from a savedprotocol buffers (.pb) file. Our Tensorflow graphs include the trained modelparameters obtained from deep learning methods we will explain below. TheTeUS-server also receives the extracted TeUS ROIs from the TeUS-client,buffers them into a Tensorflow Tensor object, and after running the Tensor-flow graph, returns the result back to the TeUS-client. The TeUS-server is astandalone C++ application running on Linux, and is built from within aclone of Tensorflow and is compiled using Bazel open-source toolbox.Receiving the TeUS data: As with the TeUS-client, the TeUS-servercreates two socket-based objects: a sender and a receiver. Note that theTeUS-serverâs receiver receives the TeUS ROIs data from the clientâs sender,146B.1. TeUS biopsy guidance systemwhile the TeUS-serverâs sender sends the output colormap to the clientâsreceiver. There is no need for multi-threading on the TeUS-server sidebecause it needs to receive the frames in sequential order.Running the deep neural networks: Concurrent with the OpenIGTLinkactivity, the TeUS-server also performs a few steps to set up and run theTensorflow graphs for PCa detection and prostate boundary segmentation.The TeUS-server initializes the Tensorflow graph, by loading the cancerclassification and segmentation trained models from the protocol bufferfiles and initializes the buffer tensor, which will be filled with T = 100frames. After receiving the T th frame, the TeUS-server runs the Tensorflowsession, feeding the buffer tensor as input to the classification network graph.Simultaneously, prostate boundary segmentation graph is fed with the buffertensor as the input to generate the segmentation results. Then, TeUS-serverconverts the returned tensors into a float vector, packs the float data fromoutput vectors into a new OpenIGTLink âImageMessageâ and sends themto the TeUS-client using the TeUS-serverâs sender object.Prostate cancer classification: The deep networks are generated basedon the methods that we presented in our earlier works and from a data setconsisting of biopsy targets in mp-MRI-TRUS fusion-biopsies with 255 biopsycores from 157 subjects (here, we refer to this data as the first retrospectivestudy). We give a brief overview of these methods. For a detailed descriptionof the models, the reader may refer to [10, 14].Prostate segmentation: The segmentation deep networks are generatedbased on our earlier works [5]. The network is pre-trained on a datasetconsisting of 4,284 expert-labeled TRUS images of the prostate as explainedin [5] and further fine-tuned using manually segmented B-mode imagesobtained during mp-MRI-TRUS fusion biopsy from Dtrain subjects. Themethod is based on residual neural networks and dilated convolution atdeeper layers. The model takes an input image of the size of 224Ã224 pixelsand generates a corresponding label map of the same size. For a detaileddescription of the models, the reader may refer to [5].147
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
A machine learning framework for temporal enhanced ultrasound guided prostate cancer diagnostics
Text