{"http:\/\/dx.doi.org\/10.14288\/1.0417279":{"http:\/\/vivoweb.org\/ontology\/core#departmentOrSchool":[{"value":"Applied Science, Faculty of","type":"literal","lang":"en"},{"value":"Electrical and Computer Engineering, Department of","type":"literal","lang":"en"}],"http:\/\/www.europeana.eu\/schemas\/edm\/dataProvider":[{"value":"DSpace","type":"literal","lang":"en"}],"https:\/\/open.library.ubc.ca\/terms#degreeCampus":[{"value":"UBCV","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/creator":[{"value":"Behnami, Delaram","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/issued":[{"value":"2022-08-11T16:01:18Z","type":"literal","lang":"en"},{"value":"2022","type":"literal","lang":"en"}],"http:\/\/vivoweb.org\/ontology\/core#relatedDegree":[{"value":"Doctor of Philosophy - PhD","type":"literal","lang":"en"}],"https:\/\/open.library.ubc.ca\/terms#degreeGrantor":[{"value":"University of British Columbia","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/description":[{"value":"Heart failure (HF) is associated with poor patient outcomes and burdens healthcare systems and clinicians. Fortunately, therapeutic options are available for managing cardiac dysfunction if diagnosed early. Echocardiography (echo) can be used to assess cardiac function swiftly and detect signs or risk factors of HF. Nonetheless, echo acquisition and interpretation require extensive training and experience, leading to exceeding demand for the available clinical echo services. This thesis investigates the feasibility of machine learning (ML)-based solutions for analyzing heart function based on clinical echo data and available annotations. The goal is to automate measurements of indicators of functional diseases. We focus on guideline-aware supervised learning frameworks for assessing LV ejection fraction (EF), regional wall motion abnormality (WMA), and LV diastolic dysfunction (LVDD). We propose spatio-temporal neural networks to determine EF from echo cine loops. We utilize multi-task learning with observer variability modelling is leverage the label noise and decouple errors in different available EF labels. In the context of regional systolic function, we present an error quantification and visualization framework to evaluate the generalizability of disease-agnostic models on diseased cohorts. We validate segmentation models trained on standard populations in a WMA cohort and report global and local error metrics with weak wall segment labels. This framework enables us to further identify failure modes in trained ML models. Using the errors obtained from the weak labels, we observed that segmentation performance might become jeopardized in the presence of akinetic LV wall segments. Finally, in the most extensive study of its kind, we demonstrate the impacts of the updated clinical guidelines for diastolic function assessment based on measurements derived from echo. We propose a neural network to replicate the latest clinical guidelines for diastolic function classification and extend this model to a regression framework to obtain a novel continuous LVDD scoring system. Increasing the size and diversity of the training and test set for model training and clinical validation is critical to further developing ML-driven heart disease diagnostic tools. Future work may involve ML-based multi-chamber quantification, myocardium localization, and Doppler image analysis toward automatic disease diagnosis.","type":"literal","lang":"en"}],"http:\/\/www.europeana.eu\/schemas\/edm\/aggregatedCHO":[{"value":"https:\/\/circle.library.ubc.ca\/rest\/handle\/2429\/82322?expand=metadata","type":"literal","lang":"en"}],"http:\/\/www.w3.org\/2009\/08\/skos-reference\/skos.html#note":[{"value":"Machine Learning for Diagnosing Functional HeartDisease in EchocardiographybyDelaram BehnamiB.A.Sc., The University of British Columbia, 2014M.A.Sc., The University of British Columbia, 2016A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Electrical and Computer Engineering)The University of British Columbia(Vancouver)August 2022\u00a9 Delaram Behnami, 2022The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Machine Learning for Diagnosing Functional Heart Disease in Echocar-diographysubmitted by Delaram Behnami in partial fulfillment of the requirements for thedegree of Doctor of Philosophy in Electrical and Computer Engineering.Examining Committee:Purang Abolmaesumi, Professor, Electrical and Computer Engineering, UBCSupervisorRobert Rohling, Professor, Electrical and Computer Engineering and Departmentof Mechanical Engineering, UBCSupervisory Committee MemberZhen Jane Wang, Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberRoger Tam, Professor, Biomedical Engineering and Radiology, UBCUniversity ExaminerShahriar Mirabbasi, Professor, Electrical and Computer Engineering, UBCUniversity ExaminerKayvan Najarian, Professor, Computational Medicine and Bioinformatics, Emer-gency Medicine, and Electrical Engineering, University of MichiganExternal ExamineriiAbstractHeart failure (HF) is associated with poor patient outcomes and burdens healthcaresystems and clinicians. Fortunately, therapeutic options are available for manag-ing cardiac dysfunction if diagnosed early. Echocardiography (echo) can be usedto assess cardiac function swiftly and detect signs or risk factors of HF. Nonethe-less, echo acquisition and interpretation require extensive training and experience,leading to exceeding demand for the available clinical echo services.This thesis investigates the feasibility of machine learning (ML)-based solu-tions for analyzing heart function based on clinical echo data and available anno-tations. The goal is to automate measurements of indicators of functional diseases.We focus on guideline-aware supervised learning frameworks for assessing LVejection fraction (EF), regional wall motion abnormality (WMA), and LV diastolicdysfunction (LVDD). We propose spatio-temporal neural networks to determineEF from echo cine loops. We utilize multi-task learning with observer variabil-ity modelling is leverage the label noise and decouple errors in different availableEF labels. In the context of regional systolic function, we present an error quan-tification and visualization framework to evaluate the generalizability of disease-agnostic models on diseased cohorts. We validate segmentation models trainedon standard populations in a WMA cohort and report global and local error met-rics with weak wall segment labels. This framework enables us to further identifyfailure modes in trained ML models. Using the errors obtained from the weaklabels, we observed that segmentation performance might become jeopardized inthe presence of akinetic LV wall segments. Finally, in the most extensive study ofits kind, we demonstrate the impacts of the updated clinical guidelines for dias-tolic function assessment based on measurements derived from echo. We proposeiiia neural network to replicate the latest clinical guidelines for diastolic functionclassification and extend this model to a regression framework to obtain a novelcontinuous LVDD scoring system. Increasing the size and diversity of the trainingand test set for model training and clinical validation is critical to further develop-ing ML-driven heart disease diagnostic tools. Future work may involve ML-basedmulti-chamber quantification, myocardium localization, and Doppler image anal-ysis toward automatic disease diagnosis.ivLay SummaryEchocardiography (echo) enables real-time heart imaging and is an excellent toolfor the early detection of heart disease. However, echo interpretation requires highcompetency and is very subjective. Machine learning (ML) models can be trainedon large echo databases to perform basic image or video analysis tasks. This the-sis proposes ML-based methodologies for directly measuring critical indices fromecho. We focus on solutions that use existing labels in clinical reports and ex-plore ways to combat or leverage label uncertainties. We utilize multi-view spatio-temporal neural networks for systolic function assessment. We investigate disease-agnostic ML models\u2019 generalizability and fairness in patients with wall motionabnormalities. We also utilize neural networks to model diastolic function basedon clinical measurements. To ensure diagnostic reliability, it is imperative to trainand validate ML models on larger datasets and incorporate relevant informationsources into the designed decision support system.vPrefaceEthics ApprovalAll data used for the research in this thesis originate from retrospective clinicaldatabases in Vancouver Coastal Health, with approval from the Research EthicsBoard (REB) at the University of British Columbia (UBC), as per ethics approvalcertificate number H16-02624. The data were anonymized and stored in ethics-compliant spaces, as required.First-author PublicationsThe enclosed dissertation has been prepared based on the publications and sub-mitted manuscripts listed below, all of which benefited from the collaboration andcontributions of several scientists and researchers who co-authored them. In ad-dition, minor modifications have been made to the text in published chapters asneeded to fit the overall flow of the dissertation.Chapter 2: The studies presented in Chapter 2 have been published in:\u2022 D. Behnami, C. Luong, H. Vaseli, A. Abdi, H. Girgis, D. Hawley, R. Rohling,K. Gin, P. Abolmaesumi, and T. Tsang. Automatic detection of patients witha high risk of systolic cardiac failure in echocardiography. In Deep Learningin Medical Image Analysis and Multimodal Learning for Clinical DecisionSupport, pages 65\u201373. Springer, 2018 ([25] in Bibliography).\u2022 D. Behnami, C. Luong, H. Vaseli, H. Girgis, A. Abdi, D. Hawley, K. Gin,R. Rohling, P. Abolmaesumi, and T. Tsang. Automatic cine-based detec-tion of patients at high risk of heart failure with reduced ejection fractionviin echocardiograms. Computer Methods in Biomechanics and BiomedicalEngineering: Imaging & Visualization, 8(5):502\u2013508, 2020 ([28] in Bibli-ography).Manuscript [25] was presented in the Deep Learning in Medical Image Analy-sis and Multimodal Learning for Clinical Decision Support Workshop in Granada,Spain, at the Medical Image Computing and Computer-Assisted Intervention (MIC-CAI) 2018. Manuscript [28] was written as an extension to this study with a deeperdive into the results and analysis of challenges. D. Behnami led data selection, viewclassification and data processing, problem formulation, method development, ex-perimentation and result analysis and manuscript development. Dr. Luong con-tributed to echo acquisition and annotation, problem formulation, result analysis,and manuscript development. H. Vaseli and Dr. Abdi contributed to data retrievaland processing. Dr. Girgis contributed to the manuscript with clinical feedback.Drs. Abolmaesumi, Tsang, and Rohling supervised the project and contributed toproblem formulation and methodology development. Drs. Tsang, Luong and Gincontributed to data acquisition as part of clinical service.Chapter 3: The studies presented in Chapter 3 were published in:\u2022 D. Behnami, H. Y. A. Girgis, C. Luong, D. Hawley, R. Rohling, K. Gin,P. Abolmaesumi, and T. Tsang. Artificial intelligence for visual assessmentof left ventricular systolic function in patients with a wide range of ejectionfraction. Journal of the American Society of Echocardiography, 32:121\u2013122,06 2019 ([26] in Bibliography).\u2022 D. Behnami, Z. Liao, H. Girgis, C. Luong, R. Rohling, K. Gin, T. Tsang,and P. Abolmaesumi. Dual-view joint estimation of left ventricular ejectionfraction with uncertainty modelling in echocardiograms. In InternationalConference on Medical Image Computing and Computer-Assisted Interven-tion, pages 696\u2013704. Springer, 2019 ([27] in Bibliography).Multi-task learning results were first presented at the American Society of Echo(ASE) [26] in Portland, OR, USA, 2019. Observer variability modelling was pub-lished and presented in MICCAI 2019 and published in Shenzhen, China. In [26],D. Behnami and Dr. Girgis are co-first authors who contributed equally to theseviimanuscripts. In both versions, D. Behnami led data selection, view classificationand data processing, problem formulation, method development, experimentation,result analysis, and manuscript development. Drs. Girgis contributed to problemformulation, result analysis, and manuscript development. In [27], Dr. Liao is alsoa co-first author and contributed to observer variability modelling based on earlierwork on quality estimation. D. Behnami led the implementation and experimen-tation involved in [27] for multi-task EF modelling. H. Vaseli and Dr. Abdi con-tributed to data retrieval and processing. Dr. Luong contributed to the manuscriptwith clinical feedback. Drs. Abolmaesumi, Tsang, and Rohling supervised theproject and contributed to problem formulation, methodology and manuscript de-velopment. Drs. Tsang and Gin contributed to data acquisition as part of clinicalservice. Additionally, the observer variability modelling presented in this chaptercontributed to the patent:\u2022 P. Abolmaesumi, Z. Liao, T. Tang, and D. Behnami. Neural network imageanalysis. https:\/\/patents.google.com\/patent\/US20210365786A1, August 2021([5] in Bibliography).Chapter 4: Chapter 4 contains an exploration of the designed framework forleft ventricular wall motion analysis. Studies on generalizability of LV segmenta-tion presented in Chapter 4 has been presented at ASE 2022 in Seattle, WA, USA,in June.\u2022 D. Behnami, C. Luong, M. Jafari, N. Van Woudenberg, D. Hawley, R. Rohling,P. Abolmaesumi, and T. Tsang. Can AI-driven LV delineation withstand re-gional wall motion abnormalities in TTE echo. Journal of the AmericanSociety of Echocardiography (in press), June 2022 ([29] in Bibliography).D. Behnami and Dr. Luong are co-first authors who contributed equally to thismanuscript. D. Behnami led data selection, view classification and data process-ing, problem formulation, method development, experiments and analysis. Dr.Luong contributed to echo acquisition and annotation, problem formulation, re-sult analysis, and manuscript development. Dr. Jafari\u2019s contribution involvedpreviously-trained endocardium LV segmentation. Dr. Liao\u2019s contribution in-volved previously-trained view and quality classification. N. Van Woudenbergviiicontributed to data retrieval, anonymization and transfer, as well as model trou-bleshooting support. D. Hawley contributed to data retrieval. Drs. Abolmaesumi,Tsang and Rohling supervised the project and contributed to problem formulationand methodology development.Chapter 5: Studies in Chapter 5 were published in:\u2022 R. Jiang*, D. Yeung*, D. Behnami*, J. Jue, M. Tsang, K. Gin, C. Luong,P. Nair, H. Girgis, P. Abolmaesumi, et al. Machine learning to facilitateassessment of diastolic function by echocardiography. Canadian Journal ofCardiology, 35(10):S4\u2013S5, 2019 ([106] in Bibliography).\u2022 D. F. Yeung*, R. Jiang*, D. Behnami*, J. Jue, R. Sharma, M. Turaga, C. L.Luong, M. Y. Tsang, K. G. Gin, H. Girgis, et al. Impact of the updateddiastolic function guidelines in the real world. International Journal of Car-diology, 326:124\u2013130, 2021 ([241] in Bibliography).The authors with asterisks, including D. Behnami, are co-first authors who con-tributed equally to diastolic function manuscripts. D. Behnami contributed to theproblem formulation and ML model design, data preparation, setup, model im-plementation and troubleshooting. Drs. Yeung and Jiang delved deeper into theclinical implications of our findings. Dr. Jiang contributed to model training andperformance evaluation. Dr. Yeung contributed to clinical analysis and clinicalmanuscript development. Others contributed by providing clinical annotations aspart of clinical service.ixTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxxvi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Clinical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 Cardiac Anatomy and Function . . . . . . . . . . . . . . . . 11.1.2 Heart Disease . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Heart Disease Management and Cardiac Imaging . . . . . . . . . . . 51.2.1 Heart Imaging Modalities . . . . . . . . . . . . . . . . . . . . 61.2.2 Cardiac Assessment in Echo . . . . . . . . . . . . . . . . . . 81.2.3 High Echo Demand and Importance of Timely Diagnosis . 171.3 Towards Automated Echo Interpretation . . . . . . . . . . . . . . . . 181.3.1 Machine Learning and Echo . . . . . . . . . . . . . . . . . . 18x1.3.2 Challenges and Considerations for ML-based Echo Anal-ysis and Diagnostics . . . . . . . . . . . . . . . . . . . . . . . 191.4 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.4.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211.4.2 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221.4.3 Thesis Context and Scope . . . . . . . . . . . . . . . . . . . 251.4.4 Summary of Contributions . . . . . . . . . . . . . . . . . . . 252 Automatic Risk-based Classification of Ejection Fraction in Echo . . 302.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.1.1 Clinical Background . . . . . . . . . . . . . . . . . . . . . . 302.1.2 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . 312.1.3 Challenges of Visual EF Assessment in Echo . . . . . . . . 332.1.4 Chapter Overview . . . . . . . . . . . . . . . . . . . . . . . . 342.2 Material and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 352.2.1 Echo Clinical Database . . . . . . . . . . . . . . . . . . . . . 352.2.2 Data Preparation . . . . . . . . . . . . . . . . . . . . . . . . . 372.2.3 Neural Network Architecture . . . . . . . . . . . . . . . . . . 422.2.4 Implementation and Model Training . . . . . . . . . . . . . 442.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . 462.4 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . 482.4.1 Optimal Model Performance . . . . . . . . . . . . . . . . . . 482.4.2 Effect of Image Quality on Performance . . . . . . . . . . . 492.4.3 Phase Detection and Cine Synchronization . . . . . . . . . . 492.4.4 Extending Binary to Multi-class EF Classification . . . . . . 512.4.5 Investigating the Impacts of LV Localization Accuracy onFunction Assessment . . . . . . . . . . . . . . . . . . . . . . 513 Dual-view Joint Estimation of EF with Uncertainty Modelling in Echo 543.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.1.1 EF and Observation Variability . . . . . . . . . . . . . . . . 543.1.2 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . 573.2 Materials and Method . . . . . . . . . . . . . . . . . . . . . . . . . . 57xi3.2.1 Uncertainty Modelling . . . . . . . . . . . . . . . . . . . . . 573.2.2 EF Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.2.3 Neural Network Architecture . . . . . . . . . . . . . . . . . . 593.2.4 Data Preparation . . . . . . . . . . . . . . . . . . . . . . . . . 613.2.5 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . 633.4 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . 663.4.1 Impacts of Uncertainty Modelling . . . . . . . . . . . . . . . 663.4.2 Model Design and Performance in the Thesis and Litera-ture Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . 663.4.3 Beyond Aleatoric Modelling for EF . . . . . . . . . . . . . . 674 Machine Learning for Left Ventricular Wall Motion Analysis in Echo 684.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.1.1 Clinical Background . . . . . . . . . . . . . . . . . . . . . . 684.1.2 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . 714.1.3 Challenges of Regional Wall Motion Analysis . . . . . . . . 734.1.4 Chapter Overview . . . . . . . . . . . . . . . . . . . . . . . . 744.2 Material and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 774.2.1 RWMA Patient Cohort . . . . . . . . . . . . . . . . . . . . . 774.2.2 View Classification and Quality Quantification . . . . . . . 784.2.3 Modelling Myocardial Wall Motion . . . . . . . . . . . . . . 814.2.4 RWMA Detection via Multi-task Visual Assessment . . . . 834.2.5 Wall Motion Analysis with Weak Labels . . . . . . . . . . . 854.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . 904.3.1 Optimal Quality Views for Regional Wall Motion Analysis 904.3.2 Experiments with Direct Segmentation-free Wall MotionAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974.3.3 Endocardium Segmentation Performance . . . . . . . . . . . 984.4 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . 1144.4.1 Echo Quality for Systolic Assessment . . . . . . . . . . . . . 1144.4.2 Difficulties with Segmentation-free Wall Motion Assessment116xii4.4.3 Generalizability of Disease-agnostic ML-based Segmenta-tion on WMA Cohorts . . . . . . . . . . . . . . . . . . . . . 1174.4.4 Future Directions for Wall Motion Analysis . . . . . . . . . 1195 Automatic Diastolic Dysfunction Diagnosis from Echo-derived Pa-rameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1225.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1225.1.1 Clinical Background . . . . . . . . . . . . . . . . . . . . . . 1225.1.2 Challenges for Detecting LVDD . . . . . . . . . . . . . . . . 1245.1.3 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . 1295.1.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . 1295.2 Material and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 1305.2.1 Clinical Database . . . . . . . . . . . . . . . . . . . . . . . . 1305.2.2 Comparison of 2009 and 2016 Clinical Guidelines for LVDD1315.2.3 Modelling Diastolic Dysfunction with Neural Networks . . 1345.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1385.3.1 Impacts of Updated Clinical Guidelines on LVDD . . . . . . 1385.4 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . 1435.4.1 Significance of the Presented ML Use Case for Diastology . 1435.4.2 Comparison of Guidelines . . . . . . . . . . . . . . . . . . . 1445.4.3 ML for Image-based Diastology . . . . . . . . . . . . . . . . 1456 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . 1486.1 Thesis Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1486.1.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . 1496.2 Discussion and Future Work . . . . . . . . . . . . . . . . . . . . . . . 1516.2.1 Combatting Label Noise . . . . . . . . . . . . . . . . . . . . 1516.2.2 More Data, Fewer Problems? . . . . . . . . . . . . . . . . . 1536.2.3 Towards a Holistic ML-based Heart Disease Diagnosis Frame-work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164A Related Co-authored Publications . . . . . . . . . . . . . . . . . . . . . 194xiiiB Supporting Materials: Echo Data Retrieval Pipelines . . . . . . . . . . 196C FileMaker Clinical Reporting Interface . . . . . . . . . . . . . . . . . . 202xivList of TablesTable 2.1 Default values of EFEyeballed in DFileMaker database. . . . . . . . . 36Table 2.2 Classification accuracy was obtained based on A2C, A4C andA2C+A4C cine loops using the proposed DenseNet-based DSFEand bi-GRU. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Table 2.3 A breakdown of the EF classification accuracy based on EFrange and number of samples. . . . . . . . . . . . . . . . . . . . . 47Table 4.1 Regional Wall Motion Index (WMSI) and measurement crite-ria [127]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Table 4.2 Correspondences of alphabetical segment codes (used in thischapter for error quantification in A2C and A4C) and the con-ventional 16-segment numerical coding used by echocardiogra-phers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Table 4.3 ML-predicted mean and maximum quality scores were analyzedfor the RWMA-relevant views for n=1,145 multi-view echo ex-ams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93xvTable 4.4 Summary of overall model performance for LV segmentation,volume estimation, and EF calculation for A2C and A4C viewson ED and ES frames. Compared to the control cohort, consis-tently more significant errors were obtained, suggesting lowerperformance with the existence of regional dysfunction, mostlikely not represented in the original training set. We observeda tendency to underestimate the LV area and calculated volumein both ED and ES frames in both views. Higher errors wereobserved in predicting the ED area and volume compared to ES. 101Table 4.5 Summary of regional prediction accuracy across different sub-sets in terms of regional wall distance d\u03c6\u2aa7k, regional Dice D\u03c6\u2aa7k,overall Dice D\u03c6 and EF calculation error \u2206EF with and withoutthe presence of RWMA. Results are broken down based on A2Cand A4C views and their corresponding visible CAD regions.Notations: y expert-annotated quantity, y\u02c6 model-predicted quan-tity, \u03c6 phase (ED or ES), mask Dice D\u03c6 , number of samples nwith mean and standard deviation (\u00b5,\u03c3). . . . . . . . . . . . . . 107Table 5.1 Parameters involved in echo-based diastology. These quantitiesare used as inputs to the guideline algorithms (rule-based andneural networks) presented in this chapter to determine LVDD. . 124Table 5.2 Baseline characteristics of each cohort assessed for LV diastolicfunction and filling pressures . . . . . . . . . . . . . . . . . . . . 133Table 6.1 Summary of datasets used in this thesis compared to publicdatasets of echo and natural image and video datasets. The pub-lic natural image datasets are several orders of magnitude largerthan the echo datasets. Larger echo training sets will likely en-able further ML model development for echo-based image anal-ysis and disease detection. . . . . . . . . . . . . . . . . . . . . . . 155Table 6.2 Categorical clinical labels in DFileMaker. FileMaker interfacesnapshots are available in Figs. C.1-C.4. . . . . . . . . . . . . . . 161xviList of FiguresFigure 1.1 The cardiac anatomy includes the four heart chambers, i.e. LV,LA, RV, RA, and four valves, i.e. the mitral (bicuspid), aor-tic, tricuspid, and pulmonary valves. Systole involves the flowof oxygenated blood (path: lungs, pulmonary veins, LA, mi-tral valve, LV, aortic valve, aorta, other organs). Diastole in-volves the flow of deoxygenated blood (path: other organs,superior and inferior vena cava, RA, tricuspid valve, RV, pul-monary valve, pulmonary artery, lungs). The heart schematicwas adapted from Wikimedia Commons [229]. . . . . . . . . . . 2Figure 1.2 The cardiac cycle consists of systole (LV contraction) and di-astolic (LV relaxation and expansion). Arrows show the direc-tion of blood flow, and the Wiggers diagram shows the ven-tricular volume changes (red) and voltage changes (blue) as afunction of time. (Heart images and Wiggers diagram adaptedfrom Wikimedia Commons [230, 231].) . . . . . . . . . . . . . . 3Figure 1.3 Common cardiac imaging modalities for assessing the heartstructure and function. The left ventricle is highlighted in eachimage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6xviiFigure 1.4 The clinical workflow involved in echo acquisition and inter-pretation. Images of various views and ultrasound modes arecaptured by a sonographer and pushed to the hospital cardi-ology PACS (CPACS) after each study. Cardiologists later re-trieve these studies and review and annotate them to assess car-diac health. Finally, the annotations and diagnoses are pushedback to the CPACS database. . . . . . . . . . . . . . . . . . . . . 9Figure 1.5 Standard cross sections for 2D echo images and the corre-sponding views of the heart: a) apical two-chamber (A2C), b)apical four-chamber (A4C), c) parasternal long-axis (PLAX),and d) parasternal short-axis at the papillary muscle level (PSAX-PM). (View images adapted from Wikimedia Common). . . . . 10Figure 1.6 Categorization of cardiac indices and parameters derived fromecho. Colours represent aspects of determining these parame-ters: standard echo views, mode of ultrasound imaging, tem-poral data dimension, and measurement method. . . . . . . . . . 12Figure 1.7 Biplane Simpson\u2019s method of disks for measuring EF. LV issegmented in ED and ES frames of both A2C and A4C cineloops (a); LV volume is calculated by summing over the es-timated disk volumes (b). (Subfigure (a) from DocPlayer.net;and (b) from CardioServ.net.) . . . . . . . . . . . . . . . . . . . . 13Figure 1.8 Cardiac wall motion and thickening. The LV wall consistsof the inner and outer boundaries endocardium, pericardium(serous membrane lining), and the myocardium (muscular tis-sue). Blue arrows indicate systolic myocardial motion. . . . . . 15Figure 1.9 The main directions of deformation and strain are imposed onthe LV myocardium. . . . . . . . . . . . . . . . . . . . . . . . . . 16Figure 1.10 Normal vs. dysfunctional regional wall motion and differentseverities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17xviiiFigure 1.11 Overview of a supervised learning framework for automaticcardiac measurement extraction and heart disease prediction.For the i-th patient that visits the echo clinic at time t, i.e. Pti ,echo images X ti are acquired. Measurements Yti are derivedfrom echo manually by clinicians or automatically by trainedML models, which can then be used to determine if the patienthas a heart disease diagnosis during the visit (Dti). Chapters 2and 3 focus on X ti \u00d0\u2192 Y ti for EF. Chapter 4 investigates thegeneralizability of a model trained on an average cohort on astrictly diseased wall motion abnormality cohort to evaluatethe performance of X ti \u00d0\u2192Y ti in the presence of severe cardiacconditions. Chapter 5 investigates the feasibility of using MLto determine diastolic dysfunction based on clinical measure-mentsi.e. Y ti \u00d0\u2192Dti . . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 1.12 Overview of thesis focusing on using ML for systolic (EF andRWMA) and diastolic function assessment in echo. Each rowrepresents a hypothesis and the corresponding carried out stud-ies. Chapters 2 and 3 focus on EF estimation based on echoimages. Chapter 4 investigates visual assessment and endo-cardial segmentation for wall motion analysis in echo videos.Chapter 5 explores the feasibility of ML to automate LVDDclassification based on clinical measurements. Although thisthesis focuses not on the outcomes themselves, the potentialadverse outcomes linked to the diagnoses are highlighted toput the work in the clinical context. . . . . . . . . . . . . . . . . 24xixFigure 1.13 Overview of the INFUSE projects that utilize ML towards au-tomating echo examination. The main goals are to assist withecho acquisition and interpretation for diagnostics, and aux-iliary modules indirectly enable these goals. The bigger redbox represents the scope of this thesis, and enclosed smalleroutlined boxes represent chapters of the thesis, with collabora-tions with Asgharzadeh and Kazemi Esfehani et al. highlightedin orange and gray. Other colours represent adjacent researchby peers (blue for Taheri Dezaki, purple for Jafari and Sahe-bzamani). Items outside coloured boxes are active projects inpreliminary stages at the time of submission of this thesis. . . . 29Figure 2.1 Comparison of LV motion in ES and ED phases of PSAX (a),A2C (b) and A4C (c). Deformations, and movements of cham-bers and valves are more complex in A2C and A4C (used inecho) compared to PSAX (used in CMR), causing echo-basedLV assessment to be more difficult. . . . . . . . . . . . . . . . . 33Figure 2.2 EF can be evaluated by studying the dynamics of the heartthroughout the cardiac cycle. Low-risk (left) and high-risk(right) systolic performance are associated with sufficient andinsufficient changes in the LV volume, respectively. The solidcontours depict the ES volume (ESV), and the dashed contoursshow the ED volume (EDV). . . . . . . . . . . . . . . . . . . . . 34Figure 2.3 EF labels in DFileMaker (a), i.e. EFSimpson\u2019s and EFEyeballed,and assigned risk-based labels EFBinary used in this section(b). EFSimpson\u2019s is a manually entered value based on Simp-son\u2019s method of disks, while EFEyeballed is an estimated valuein eight fixed values. (See Table 2.1 for details on these classes.) 40Figure 2.4 The cardiac phase is extracted from ECG. Subfigure (a) showsthe PQRST complex visible in ECG and (b) shows an exampleof synchronized A2C and A4C echo cines. Cines are tempo-rally resampled between consecutive ED frames, i.e. RAxC1 toRAxC2 , and effectively synchronized. . . . . . . . . . . . . . . . . 41xxFigure 2.5 The architecture of the proposed dual-view network for risk-based classification of EF. Frame-level image features are ex-tracted for each channel, then synchronously concatenated andfed through a bi-GRU for temporal embedding (a). The ar-chitecture used in the DSFE block consists of 2D convolu-tional layers, three interconnected dense blocks, two transitionblocks and ReLU nonlinearity (b). . . . . . . . . . . . . . . . . . 45Figure 2.6 EF classification accuracy using DenseNet (DNet) and Capsu-leNet (CNet) as the spatial feature extraction and various RNNversions on A2C, A4C and synchronous A2C+A4C views. . . . 46Figure 2.7 Example results of dual-channel segmentation-free classifica-tion of high-risk vs. low-risk EF using DenseNets and bi-GRU. 53Figure 3.1 The clinical workflow for EF assessment. The workflow\u2019s darkand light purple paths illustrate the Simpson\u2019s and visual as-sessment methods, respectively. (Heart schematics: courtesyof 123 Sonography.) . . . . . . . . . . . . . . . . . . . . . . . . . 55Figure 3.2 Correlation of EF labels (EFSimpson\u2019sBiplane and EFEyeballed) inthe current echo database. Labels are very noisy, with a cor-relation coefficient of only 0.71. Red data points indicate thesamples for which the two EF labels do not agree, while theblack crosses show samples whose labels are in agreement. . . . 56Figure 3.3 Gaussian PDF characterized by (\u00b5,\u03c3) to model the observervariability in the clinical EF labels. . . . . . . . . . . . . . . . . 56Figure 3.4 A closer look at the EF dataset and labels. The variability andinherent aleatoric uncertainty in the four EF measurements ishighlighted in (a) in terms of agreement between the four la-bels in terms of R2 scores (as low as 0.58 for EFA2CSimpson\u2019s andEFA4CSimpson\u2019s). A break-down of the categorical EFBiplaneVisual labelsis provided in (b). . . . . . . . . . . . . . . . . . . . . . . . . . . 60Figure 3.5 The proposed architecture for dual-view cine-based joint es-timation of four EF labels with uncertainty modelling, whichcharacterizes each prediction as N (\u00b5,\u03c3). . . . . . . . . . . . . 61xxiFigure 3.6 Overall regression results on the test set (N=430) in terms ofcoefficient of determination (R2), MAE, and standard devi-ation \u03c3 on the regression labelsEFA2CSimpson\u2019s, EFA4CSimpson\u2019s andEFBiplaneSimpson\u2019s without and with observer variability modelling. . . 62Figure 3.7 Regression plots demonstrating the correlation of the joint EFmodel results (vertical) and the expert annotation EF labels(horizontal) on EFA2CSimpson\u2019s (blue), EFA4CSimpson\u2019s (red), and EFBiplaneSimpson\u2019s(purple). In each case, the point-estimate results are shown onthe left (subfigures a, c, e), while the results of the uncertaintymodelling is shown on the right (subfigures b, d, f). Modellingthe observer variability using the proposed method improvesthe model performance. . . . . . . . . . . . . . . . . . . . . . . . 64Figure 3.8 Confusion matrices showing test results (N=430) for four-wayclassification of EFBiplaneVisual without (left) and with (right) ob-server variability. . . . . . . . . . . . . . . . . . . . . . . . . . . . 65Figure 3.9 Results on two test samples: the network inputs cine loops(videos on the left), expert-annotated ground labels, and thenetwork\u2019s predictions, expressed in terms of (\u00b5,\u03c3) in the ta-bles. (To play the videos, open the PDF in Adobe Acrobat,Internet Explorer or other PDF viewers, enable Flash Player,and click on the snapshots.) . . . . . . . . . . . . . . . . . . . . . 65Figure 4.1 Tri-plane LV regional wall motion analysis and the 16-segmentLV model. Orthogonal cross-sectional apical views are shownin yellow (A2C), orange (A4C) and red (A3C). CAD regionsRCA, LAD, and CX are highlighted in blue, green, and pink. . 70Figure 4.2 Synchronized echo frame sequences the apical planes (left toright: A2C, A4C, and A3C) for one cycle for a patient withRWMA diagnosis in segments 4, 10, and 14 (refer to Fig. 4.1),emphasizing high data noise, which makes both clinical andautomated wall motion analysis extremely difficult. . . . . . . . 76Figure 4.3 Distribution of global systolic function labels in the WMA co-hort (n=2,910). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78xxiiFigure 4.4 Network architecture for simultaneous view classification andquality estimation in echo as proposed by [136]. A denseNetblock [94] extracts frame-wise spatial features, which are thenaggregated through an LSTM for temporal embedding. Fi-nally, the spatio-temporal feature vectors are mapped to onestandard echo view and a continuous quality score. . . . . . . . 79Figure 4.5 Systolic myocardial thickening can be modelled as a set of tu-ples (ximyo,yimyo,\u03b8 i), given the endocardial Wendo and pericar-dial wall coordinates Wperi (see Equations 4.2 and 4.3). W \u2032periis the projection of Wendo on the Wperi curve with one-to-onemapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81Figure 4.6 Network architecture in experiments with apical tri-plane di-rect systolic wall motion analysis. Inputs are synchronizedcines of A2C, A3C, and A4C views, and outputs are regional(RWM) and global (EF, GWM) labels . . . . . . . . . . . . . . . 84Figure 4.7 Study overview for investigating the generalizability of disease-agnostic endocardial segmentation models for A2C and A4C(green and red) on a clinical WMA cohort (blue box). Themodels were previously trained [102] on a dataset with an es-timated 10% WMA. . . . . . . . . . . . . . . . . . . . . . . . . . 86Figure 4.8 Schematics of dividing the endocardial wall in A2C and A4Cviews into six segments for regional assessment of segmenta-tion performance. The LV mask M is shown in white, withthe major and minor axes red and blue, respectively. SegmentsA\u2212F are determined for the ground truth and predicted LVmasks. CO is the centroid, and \u03b2\u0302 denotes the orientation of thebullet-shaped LV. . . . . . . . . . . . . . . . . . . . . . . . . . . . 88Figure 4.9 Representative frames cine loops in five views of interest gradedby the view and quality ML model. ML-predicted qualityscores correspond well with the visibility and clarity of thecardiac anatomy. . . . . . . . . . . . . . . . . . . . . . . . . . . . 91xxiiiFigure 4.10 ML model-predicted mean (a) and maximum (b) image qualityscore percentage by the view. Boxplots show the distributionpredicted mean and maximum image quality score for eachof the five views of interest. The A4C view had significantlyhigher scores as compared to all other views (p < 0.001). TheA2C view had lower scores than all other views, except thePSAX-M\/PM view (p < 0.001, A2C view vs. PSAX-M\/PMview p=1). The A3C and PSAX-M\/PM views had similar max-imum image quality scores (p=1). . . . . . . . . . . . . . . . . . 92Figure 4.11 Demographic data distributions in the WMA cohort (n=2,910). 95Figure 4.12 Breakdown of WMSI labels for segments 1-16 (shown in Fig. 4.1),indicating the distribution of regional systolic function in theWMA cohort. Description of WMSI scoring is given in Ta-ble 4.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96Figure 4.13 Typical results for direct systolic visual assessment for RWM,GWM and EF on an unseen pathologic test set (n=246) afterconvergence on the training set. With the available data, reli-able direct regional systolic assessment could not be achieved. . 98Figure 4.14 Qualitative comparison of ground-truth LV segmentation andmodel prediction on failure samples. We noted a tendency tomiss parts of the wall, especially lateral and septal walls (all)appearing in segments D and E, underestimate predicted areaand subsequently volume, fail drastically where shadows leadto blurry walls (b and e), overestimate the major to minor axislength, yielding more slim LV mask prediction. . . . . . . . . . 100Figure 4.15 Comparison of ML prediction and clinical LV localization met-rics in terms of LV centroid (a and b), the length and width(major and minor axes dimensions in c and d), and the esti-mated area and volume (e and f). The data distributions aremodelled as normal N (\u00b5,\u03c3). . . . . . . . . . . . . . . . . . . . 103Figure 4.16 Comparison of ML prediction and clinical volumetric metricsstroke volume (a) and EF (b). The data distributions are mod-elled as normal N (\u00b5,\u03c3). . . . . . . . . . . . . . . . . . . . . . . 104xxivFigure 4.17 Comparison of segmentation accuracy D\u03c6 for key cardiac phases(\u03c6 ) ED and ES, where clinical tracings are available. Perfor-mance segmentation is significantly better in ED for A2C andES for A4C. In both views, the tendency to underestimate EFis higher than to overestimate (\u2206EF > 0). Higher errors wereobserved in A4C EF estimation compared to A2C. The risk ofoverestimating EF is lower in A4C. . . . . . . . . . . . . . . . . 106Figure 4.18 Overall comparison of wall distance errors dW,\u03c6\u2aa7k for diseased orhealthy (normal) regional wall motion across all visible seg-ments k. The possible range of the wall distance error is thediagonal length of the 128\u00d7170 images (< 214 px). . . . . . . . 108Figure 4.19 Wall distance errors dW,\u03c6\u2aa7k across individual segments of A2Cand A4C for segments A-F. Regional errors are consistentlyhigher for segments marked as diseased, i.e. yRWMk > 1, com-pared to healthy. . . . . . . . . . . . . . . . . . . . . . . . . . . . 110Figure 4.20 Comparison of segmentation accuracy D\u03c6 (a) and segmentation-based EF estimation errors \u2206EF across views and number ofdysfunctional segments (yRWM). Overall segmentation and EFaccuracy are not significantly affected by the presence of RWMA.111xxvFigure 4.21 Example RWMA sample and visualization of the regional walldistance errors in an A4C view. The first and second columnsshow the image overlay of target masks and predictions, re-spectively, for ED and ES frames, illustrating that the MLmodel performs poorly around the apex in ED. The third col-umn shows the target and predictions overlay, and the thirdrow depicts the ED-ES overlay. The regional clinical labelsare shown in the bottom-left colormap for the six LV segments.Estimated LV wall motion from the target masks and predic-tions are colour-coded on the corresponding columns. Thefourth column shows colormaps highlighting the wall distanceerrors for ED and ES, as well as mean and maximum wall dis-tance errors. The relative distribution of the regional wall de-tection errors corresponds with the RWMA labels, suggestingthe model\u2019s difficulty in tracking the LV wall when RWMA ex-ists. Other clinical labels are listed on the far left. Localizationerrors for ED and ES frames are listed on the far right. . . . . . 112xxviFigure 4.22 Example RWMA sample and visualization of the regional walldistance errors in an A2C view. The first and second columnsshow the image overlay of target masks and predictions, re-spectively, for ED and ES frames. The ML model demon-strates an overall tendency to underestimate the LV area and,subsequently, volume. The third column shows the target andpredictions overlay, and the third row depicts the ED-ES over-lay. The regional clinical labels are shown in the bottom-leftcolormap for the six LV segments. Estimated LV wall motionfrom the target masks and predictions are colour-coded on thecorresponding columns. The fourth column shows colormapshighlighting the wall distance errors for ED and ES and meanand maximum wall distance errors. The relative distribution ofthe regional wall detection errors corresponds with the RWMAlabels, suggesting the model\u2019s difficulty in tracking the LV wallwhen RWMA exists. Errors in ED seem more prominent thanin ES. Other clinical labels are listed on the far left. Localiza-tion errors for ED and ES frames are listed on the far right. . . . 113Figure 5.1 The algorithm for assessing the LVDD based on the updated2016 ASE\/EACVI Guidelines [156]. . . . . . . . . . . . . . . . 126Figure 5.2 Schematics of mitral flow peak velocity estimation using pulsedDoppler on the mitral valves for determining the E\/A ratio fordiastolic function analysis. Pulsed Doppler is placed on themitral valve (a), and velocity E and A-waves are obtained (b).(Figures are inspired by diagrams in [1]). . . . . . . . . . . . . . 127Figure 5.3 Scenarios of E\/A calculated from mitral flow velocities andclinical interpretations of normal vs. abnormal diastolic bloodflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128Figure 5.4 Overview of clinical data cohorts established for the compari-son of 2009 ASE [155] and 2016 ASE\/EACVI [156] for eval-uating left ventricular diastolic function. . . . . . . . . . . . . . 132xxviiFigure 5.5 Neural network architecture for classification of diastolic dys-function based on diastology parameters, i.e. age, LV EF , LAVi,VT R, E, e\u2032lat , e\u2032sep, E\/A, E\/e\u2032 . . . . . . . . . . . . . . . . . . . . . 137Figure 5.6 Comparison of diastolic function grading based on the 2016ASE\/EACVI Guidelines [156] compared to the 2009 ASE Guide-lines [155] in the five established cohorts outlined in Table 5.2. 140Figure 5.7 Diastolic function grading is stratified by age category whenassessed using the 2009 ASE Guidelines (a) and the 2016 ASE\/EACVIGuidelines (b). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141Figure 5.8 Normalized confusion matrix for prediction of LVDD usingthe proposed neural network v.s. the ASE\/EACVI 2016 Guide-lines. The indeterminate class was excluded in this study. . . . . 142Figure 5.9 Diastolic function scores obtained by the regression LVDDmodel. A continuous output space between severely dysfunc-tional to normal function is assumed. . . . . . . . . . . . . . . . 142Figure 5.10 Example pulsed-wave Doppler on the mitral valve and extract-ing for calculating E\/A ratio. Doppler imaging is a very noisymodality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146Figure 6.1 Overview of a holistic ML-based framework for heart diag-nosis, extending beyond the scope of this thesis (1.11). Nextsteps involve prediction of outcomes for patient Pi, based onmeasurements Y ti , and predicted or diagnosed disease Dti , aswell as intervention and therapy received history of medica-tion or interventions recorded. . . . . . . . . . . . . . . . . . . . 153Figure B.1 Sequence of planned echocardiographer visit in Vancouver, BC. 197Figure B.2 Sequence of emergency echocardiographer visit in Vancouver,BC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198Figure B.3 Process for obtaining ethics and operational approvals for re-trieving retrospective clinical echocardiography data for re-search and development at UBC. . . . . . . . . . . . . . . . . . . 199Figure B.4 Echocardiography data access process. . . . . . . . . . . . . . . 200xxviiiFigure B.5 Retrieval process for the retrospective clinical echo data usedfor research and development at UBC. . . . . . . . . . . . . . . . 201Figure C.1 FileMaker reporting interface on the main page, including pa-tient information (name, ID, sex, etc.), disease indications, pa-tient lifestyle and history indications, study type and modality,text-based technician comments, etc. . . . . . . . . . . . . . . . 203Figure C.2 FileMaker reporting interface for ventricular function and struc-ture assessment, including left and right ventricular ejectionfraction (EF), systolic wall motion analysis, diastolic function,as well as hypertrophy, and thrombus. . . . . . . . . . . . . . . . 204Figure C.3 FileMaker reporting interface for atrial, aortic and pericardialassessment, including left and right atrial chamber quantifica-tion, aortic assessment, pericardial effusion, tamponade andcongenital indications. . . . . . . . . . . . . . . . . . . . . . . . . 205Figure C.4 FileMaker reporting interface for valvular assessment, includ-ing aortic valve (AV), mitral valve (MV), tricuspid valve (TV),and pulmonary valve assessment. . . . . . . . . . . . . . . . . . 206xxixGlossary2D+T Two-dimensional Time Series3D Three-dimensionalA A-wave Trans-mitral Flow Velocity in Atrial contraction (late diastole)A2C Apical Two-chamber cardiac viewA3C Apical Three-chamber cardiac viewA4C Apical Four-chamber cardiac viewA5C Apical Five-chamber cardiac viewASE American Society of EchocardiographyAxC Apical Two-chamber or Apical Four-chamber cardiac viewBCE Binary Cross-Entropybi-GRU Bi-directional Gated Recurrent UnitBMI Body Mass IndexC3D 3D Convolutional NetworkCAD Coronary Artery DiseaseConv2D 2D Convolutional Neural Network layerCCE Categorical Cross-EntropyxxxCCT Cardiac Computed TomographyCDF Cumulative Distribution FunctionCMR Cardiac Magnetic Resonance (Imaging)Cnet CapsuleNetCNN Convolutional Neural NetworksCPACS Cardiology Picture Archiving and Communications SystemConvLSTM Convolutional Long Short-term MemoryDD Diastolic DysfunctionDSFE Deep Spatial Feature ExtractionDICOM Digital Imaging and Communications in MedicineDice S\u00f8rensen-Dice CoefficientDL Deep LearningDnet\/DenseNet Densely Connected Neural NetworksDTI Doppler Tissue Imaginge\u2019 Mitral annular tissue Doppler velocityE E-wave trans-mitral inflow early filling velocityE\/e\u2019 Ratio of mitral inflow early filling velocity (E-wave) to mitral annulartissue Doppler velocityE\/A Ratio of mitral inflow early filling velocity (E-wave) to mitral inflowvelocity during atrial contraction (A-wave)EACVI European Association of Cardiovascular ImagingECE Electrical and Computer EngineeringxxxiECG ElectrocardiogramEcho Echocardiography, EchocardiogramED End-diastoleEDV End-diastolic VolumeEF Ejection FractionES End-systoleESV End-systolic VolumeFC Fully-connected Neural Network layerFDA Food and Drug AdministrationFE Feature ExtractionFOV Field of viewFR Frame RateGLS Global Longitudinal StressGRU Gated Recurrent UnitGWM Global Wall MotionGWMA Global Wall Motion AbnormalityH0 Null HypothesisHF Heart FailureHFpEF Heart Failure with Preserved Ejection FractionHFrEF Heart Failure with Reduced Ejection FractionHR Heart RateINFUSE Information Fusion in EchocardiographyxxxiiLA Left AtriumLAD Left Anterior Descending ArteryLAVi Left Atrial Volume IndexLBBB Left Annular Branch BlockLCx Left Circumflex ArteryLSTM Long Short Term MemoryLV Left VentricleLVAD Left Ventricular Assist DeviceLVEF Left Ventricular Ejection FractionLVDD Left Ventricular Diastolic DysfunctionLVH Left Ventricular HypertrophyMAC Mitral Annular CalcificationMAE Mean Absolute ErrorML Machine LearningMR Magnetic Resonance (Imaging)MSE Mean Square ErrorNLL Negative Log-likelihoodPACS Picture Archiving and Communication SystemPDF Probability Distribution FunctionPLAX Parasternal Long-axis cardiac viewPOCUS Point-of-care UltrasoundPSAX Parasternal Short-axis cardiac viewxxxiiiPSAX-A Parasternal Short-axis cardiac view at Apex levelPSAX-M Parasternal Short-axis cardiac view at the Mitral valve levelPSAX-PM Parasternal Short-axis cardiac view at Papillary Muscle levelPSAX-M\/PM Parasternal Short-axis cardiac viewa at Mid- or Papillary Musclelevelp-val p-value (in statistical tests)Px PixelR2 R2 Score (Coefficient of determinationRA Right AtriumRCA Right Coronary ArteryReLU Rectified Linear UnitResNet Residual Neural NetworkRNN Recurrent Neural NetworksROI Region of InterestRV Right VentricleRVIT Right Ventricle Inflow Tract cardiac viewRWM Regional Wall MotionRWMA Regional Wall Motion AbnormalityRWMSI Regional Wall Motion Score IndexS4C Subcoastal Four-chamber cardiac viewSGD Stochastic Gradient DescentSRI Strain Rate ImagingxxxivSTFE Spatio-temporal Feature ExtractionSTE Speckle Tracking EchoSUP Suprasternal cardiac viewSV Stroke VolumeSVM Support Vector MachineTEE Transesophageal EchocardiographyTTE Trans-thoracic EchocardiographyTR Tricuspid RegurgitationUBC University of British ColumbiaVCH Vancouver Coastal HealthVGH Vancouver General HospitalWMA Wall Motion AbnormalityWMSI Wall Motion Score IndexxxxvAcknowledgmentsI would like to acknowledge and thank my supervisor, Prof. Purang Abolmaesumi,for his invaluable supervision and support, the advisory and examining committeeProf. Rob Rohling, my clinical advisors Prof. Teresa Tsang, Dr. Christina Luong,and Dr. Hany Girgis, for their advice and expertise; and Prof. Jane Wang, Prof.Edmond Cretu, Prof. Roger Tam, Prof. Shahriar Mirabbasi, and Prof. KayvanNajarian for their feedback and guidance.I would like to additionally express my deep gratitude to Dr. Zhibin Liao, Dr.Darwin Yeung, Dr. River Jiang, Mr. Dale Hawley, Mr. Nathan Vanwoudenberg,Ms. Samira Sojoudi, Dr. Mohammad Jafari for their support, guidance and col-laboration in research; Prof. Steve Wilton, Dr. Tom Diethe, Dr. Sarah Scarfe, Mr.Ross Sheppard, Prof. Matt Yedlin, Dr. Marshall Tappen, Dr. Gabriel Pratt, Ms.Samira Sojoudi, Prof. Robin Turner, and Prof. Shahriar Mirabbasi for their valu-able support, guidance and mentorship; Prof. Leonid Sigal, Dr. Sharareh Bayat,Prof. Lutz Lampe, Dr. Babak Shadgan, Mr. David Miller, Mr. Fraser Pogue,Dr. Justin Bull, Prof. Rafeef Garbi, Prof. Tim Salcudean, for their expertise; andMs. Danielle Walker, Mr. Oluwaseun Ajaja, Ms. Kristie Henderson, Dr. AnnaMeredith and Prof. Anita Palepu for their support and leadership.I am grateful to the Department of Electrical and Computer Engineering andthe Department of Medicine at the University of British Columbia (UBC) for pro-viding the platform for the enclosed research and my growth. In addition, I amgrateful to UBC Graduate Student and Postdoctoral Studies (G+PS) for the Four-Year Doctoral Fellowship (4YF), four years of Graduate Support Initiative (GSI)awards, and the British Columbia Graduate Student (BCGS) award. Finally, I amalso thankful to the Natural Sciences and Engineering Research Council of Canadaxxxvi(NSERC) and the Canadian Institutes of Health Research (CIHR) for funding thisresearch.On a more personal note, I am eternally thankful to my advisors, family andfriends, labmates and coworkers all those that saw me through graduate school:Thank you, Purang, for your support and guidance for nearly a decade now! Thankyou, Teresa, for trusting me, empowering me, and being my role model! Thankyou, Dr. Scarfe, for seeing me through my 20s and entire grad school! Thankyou, Christina, for all your guidance and meticulousness. Thank you, Rob, foryour guidance and leadership. Thank you, Ross, for all the times you listened toand supported me and the countless hours you spent with our graduate students!Thank you, Steve, for your leadership and support - it has meant so much to me.Thank you, Tom, for believing in me and helping me grow as an engineer! Thankyou, Nathan, for all the times you made my Linux and data problems disappear!Thank you, Zhibin, for the technical guidance and the guppies and tetras! Thankyou, Samira and Sharareh, for being my friends, your leadership, and the 3 o\u2019clockcoffees! Thank you, Mohammad, for the MICCAI trips, and echo discussions!Thank you, Elena, Farah, Michael, Mehran, Qi, Hooman, Megha, Ghazal, Pardiss,Mobina, Tom, Diane, Kelly, Sanj, Donya, Pooneh, Sharmi, Tami, Shima for beingmy friends, peers, and supporting me! Thank you, Radmehr, for being my brotherand friend for life! Thank you, Mama Kattie, for all your selflessness and believingin me, even when I don\u2019t! Thank you, Baba Mehrdad, for all you have done that Icontinue to realize! And thank you, Mauricio Antonio Pepo, for being my partner,loving and grounding me.\u201cIl faut vivre et cre\u00b4er. Vivre a` pleurer.\u201dI had the time of my life! \u2661xxxviiChapter 1Introduction1.1 Clinical Background1.1.1 Cardiac Anatomy and FunctionThe heart is the most vital organ in charge of periodically pumping oxygenatedblood to all organs in the human body. The heart\u2019s structure mainly consists offour chambers and valves, which enable the single-directional flow of blood in orout of the chambers (Fig. 1.1). The four blood chambers are the left ventricle (LV),left atrium (LA), right ventricle (RV) and right atrium (RA). The ventricles arethe larger chambers that pump blood outwards, while the atria allow the inflow ofblood.The function of the blood involves supplying and recycling oxygen-filled bloodthroughout each cycle (heartbeat). Each cardiac cycle consists of a systolic anddiastolic phase (Fig. 1.2). In systole, LA receives oxygen-rich blood through thepulmonary veins from the lungs, passing it through the mitral atrioventricular valveto the LV. The LV pushes the blood to the rest of the organs via the arteries throughthe aortic valve. In diastole, the RV receives deoxygenated blood via the inferiorand superior vena cava. Once filled, RA transfers this blood to the RV, passing itthrough the tricuspid valve. Next, the carbon dioxide-rich blood is sent out throughthe pulmonary valve, into the pulmonary artery and back to the lungs to filter.1Figure 1.1: The cardiac anatomy includes the four heart chambers, i.e. LV,LA, RV, RA, and four valves, i.e. the mitral (bicuspid), aortic, tricuspid,and pulmonary valves. Systole involves the flow of oxygenated blood(path: lungs, pulmonary veins, LA, mitral valve, LV, aortic valve, aorta,other organs). Diastole involves the flow of deoxygenated blood (path:other organs, superior and inferior vena cava, RA, tricuspid valve, RV,pulmonary valve, pulmonary artery, lungs). The heart schematic wasadapted from Wikimedia Commons [229].1.1.2 Heart DiseaseHeart disease refers to a family of conditions or abnormalities that impact the struc-tural and functional performance of the heart. Heart disease is the leading globalcause of death [233], associated with 31% of annual global death [51], and 8.5million in 2015 alone. Types of heart disease include heart failure, coronary arterydisease (CAD), arrhythmias, congenital heart disease, and heart infections.Heart FailureHeart failure (HF) is a complex clinical syndrome that refers to the diminished car-diac performance for pumping blood in and out of the left ventricle ([199]), whichleads to the heart failing to supply enough blood to fulfill the body\u2019s needs. HF2Figure 1.2: The cardiac cycle consists of systole (LV contraction) and dias-tolic (LV relaxation and expansion). Arrows show the direction of bloodflow, and the Wiggers diagram shows the ventricular volume changes(red) and voltage changes (blue) as a function of time. (Heart imagesand Wiggers diagram adapted from Wikimedia Commons [230, 231].)is the most prevalent and critical type of heart disease as it alone affects over 26million people worldwide [199] and has been dubbed a silent worldwide pandemicdue to morbidity, mortality, and economic implications [222, 228]. Over 600,000Canadians live with HF [185], and the prevalence increases every year as the pop-ulation ages [89]. The prevalence of HF is projected to increase by 46% between2012 and 2030 [31]. As a result, annual HF-related healthcare costs exceed $3billion [89]. HF prognosis remains poor, as patients hospitalized due to HF are re-ported to have a mortality rate of 10% at one month, 22% at one year, 50% at fiveyears, and 90% at ten years [31, 192]. Cardiac evaluation of heart disease patientsor suspected patients is ubiquitous and critical.HF can occur if the heart is too weak to pump outwards or fails to fill up with3blood sufficiently, i.e. suboptimal systolic or diastolic function. HF with reducedejection fraction (HFrEF) occurs when the heart\u2019s systolic function is inadequate,i.e., insufficient blood is being ejected from the LV to the rest of the body everycycle. HFrEF is associated with the majority of HF-related mortality ([35]). Giventhe reciprocal relationship between left ventricular ejection fraction (EF) and sixmonth-mortality in patients with HFrEF, timely diagnosis may have a critical im-pact on outcomes. On the other hand, HF with preserved ejection fraction (HFpEF)may occur due to left ventricular diastolic dysfunction (LVDD), i.e. complicationsin refilling the LV with blood during diastole.Coronary Artery DiseaseCoronary artery disease (CAD) refers to the narrowing of the arteries due to thebuild-up of plaque, which inhibits blood supply to the heart. CAD is a leadingcause of death and disability and impacts more than 2.4 million Canadians [40].CAD is a significant source of health concerns in both genders, although it is nearlytwo-fold higher in males and increases the likelihood of premature death by threeto six times [185], making it a significant public health concern. Currently, the ref-erence standard for CAD diagnosis is catheterization and angiography [127, 170],an invasive and costly procedure. Non-invasively, echo exams can be performed atthe bedside to indirectly diagnose obstructive CAD or myocardial infarction (a.k.a.heart attack) by detecting regional wall motion abnormalities (RWMAs) [218].RWMAs are areas of abnormal myocardial thickening and motion that mayrepresent territories of compromised blood flow (myocardial ischemia). RWMAoccurs in patients who present with chest pain or heart failure. The early detectionof RWMAs can impact patient care by alerting care providers to the presence ofCAD. The early diagnosis of acute coronary syndromes is crucial for prompt med-ical and procedural therapy referral. \u201dTime is muscle\u201d in the CAD diagnosis, asuntreated, may lead to myocardial infarction. Although the presence of RWMAsis relatively specific for the detection of obstructive CAD, it has low sensitivity at67%, preventing it from becoming a mainstream method of CAD diagnosis despitethe increasing accessibility of ultrasound [48, 69, 181].4Heart Valve DiseaseValve disease may refer to a family of conditions impacting any of the heart valves:aortic, mitral, tricuspid, and pulmonary. Healthy valves regulate blood flow in oneintended direction for the intended cardiac phase (Fig. 1.1). On the other hand, anunhealthy valve may be too narrow (stenotic), leaky (regurgitating), or prolapsed.The insufficient valve opening is stenosis, which causes inadequate blood flowthrough the narrow gateway.Improper closing of the valves, known as regurgitation,can lead the blood to leak or flow backwards. In some cases, improper closingmay be due to valve prolapse, i.e. leaflets not closing correctly or slipping out ofplace. Other common types of valve disease include mitral annular calcification(MAC), which is caused by the gradual deposition of calcium on the mitral valve.Calcification increases the stiffness of the valve and may deform the valve in moresevere cases, leading to valvular dysfunction.ArrhythmiasArrhythmias are conditions where the cardiac rate or rhythm is affected for sometime. The heart rhythm may increase, decrease, or become irregular during anarrhythmia. An arrhythmia may indicate a current heart attack but is also linkedto several other cardiac conditions such as HF, previous heart attacks, obstructiveCAD, as well as non-cardiac causes such as infections, diabetes, or high bloodpressure.1.2 Heart Disease Management and Cardiac ImagingToday, many proven effective therapies that can dramatically improve patients\u2019survival and quality of life are available. Nonetheless, therapy and surgical inter-ventions depend on early and timely diagnosis of heart disease, types, extents, andrisk factors based on clinical history, demographics, etc. Advancements in imag-ing technologies allow clinicians to assess cardiac mechanics and cardiovascularhemodynamics non-invasively.5(a) CMR (b) CCT(c) SPECT (d) EchoFigure 1.3: Common cardiac imaging modalities for assessing the heartstructure and function. The left ventricle is highlighted in each image.1.2.1 Heart Imaging ModalitiesImaging modalities used for cardiovascular evaluation include cardiac nuclear imag-ing, computed tomography, resonance imaging, fluoroscopy, and echocardiogra-phy [117] (Fig. 1.3). These modalities allow clinicians to observe the cardiacstructure in two or three spatial dimensions throughout several heart cycles, andevaluate the cardiac function based on inferred motion and hemodynamics.Nuclear ImagingNuclear imaging involves an injection of radioactive substances and imaging equip-ment to trace the blood flow [159]. Types of nuclear imaging include single-photonemission computed tomography (SPECT, shown in Fig. 1.3c) and positron emis-sion tomography (PET), which rely on the emission of nuclear particles (photonsor positrons) for image formation. Nuclear imaging enables the detection of blood6blockages and hence can be used for CAD and coronary heart disease diagno-sis [159]. However, though small in dose, tracer injection and radiation in nuclearimaging introduce health risks.Cardiac Computed Tomography (CCT) ScansCCT (Fig. 1.3b) may be used to detect conditions such as heart failure, heart at-tack, inflammation, CAD, atrial fibrillation, and congenital heart defects [159]. Inaddition, CCT calcium scans are used to identify coronary calcification, i.e. find-ing locations in arteries with calcified plaque build-up non-invasively, which canblock the supply of oxygenated blood to the heart muscle [189]. Nonetheless, CCTinvolves ionizing x-rays, which are harmful. Furthermore, image acquisition canbecome costly (up to over $1,000 per scan [87]), requiring expensive specializedequipment.Cardiac Magnetic Resonance (CMR) ImagingMagnetic imaging (Fig. 1.3a) provides detailed insight into the tissues with highhydrogen contents, making CMR an excellent modality for examining the heartmuscles and connecting blood vessels. CMR can be used to obtain accurate mea-surements of chamber sizes and their function [159]. CMR is used for diagnosingseveral heart complications such as heart failure, valve disease, CAD, arrhythmias,etc. Further, CMR can be reliably used to detect abnormalities in the cardiac mus-cle tissue, such as muscle scarring in a heart attack, inflammation due to a heartinfection, or congenital diseases [159]. CMR is also non-ionizing and has moreminor associated risks than CCT. Nonetheless, CMR requires expensive machineryand equipment (up to $4 million) and is not feasible on large scales. For example,a CMR exam is estimated to cost up to $6,000 depending on factors such as thelocation and complexity of heart conditions [88].EchocardiographyCardiac ultrasound or echocardiography (echo) is the most common modality forheart imaging (Fig. 1.3d). Echo is non-ionizing and harmless, involves portableand more affordable imaging equipment, and can provide diagnostically useful7real-time images of the heart function and anatomy [36]. Various types of echocar-diography include trans-thoracic echo (TTE), transesophageal echo (TEE), three-dimensional (3D) echo, and stress echo. TTE involves imaging the heart throughthe chest wall, while in TEE, images are acquired through the esophagus [159].Echo is routinely used for diagnosing heart failure, valve disease, myocardial in-farction (heart attack), CAD, and some congenital defects [159]. Thanks to its af-fordability (less than $100,000 for full-capability cart-based machines by vendorssuch as Philips and GE), the echo is accessible at many point-of-care or emergencycentres.With more than 200,000 echo exams performed annually in Canada alone,TTE is the most frequent cardiac imaging exam [70, 159, 184]. TTE is one ofthe most accepted modalities used for diagnosis, follow-up and management ofpatients or suspected patients of heart disease [127, 129].1.2.2 Cardiac Assessment in EchoTTE is routinely used in clinical settings to assess cardiac health quickly. An echoexam often entails the acquisition of many (up to 300) video clips (cine loops)of 2D cross-sections of the heart. In addition to B-mode, Doppler imaging mayalso be used for hemodynamic analysis. The overall clinical workflow for echoacquisition and interpretation is depicted in Fig. 1.4.8Figure 1.4: The clinical workflow involved in echo acquisition and interpre-tation. Images of various views and ultrasound modes are captured bya sonographer and pushed to the hospital cardiology PACS (CPACS)after each study. Cardiologists later retrieve these studies and reviewand annotate them to assess cardiac health. Finally, the annotations anddiagnoses are pushed back to the CPACS database.9Standard Echo ViewsEchocardiographers acquire cine loops in standard complementary views. Alongwith simple annotations, this data gets recorded to the hospital\u2019s cardiology PictureArchiving and Communications System (CPACS). The various views are later ex-amined in depth by a cardiologist to detect structural or functional abnormalities,perform routine measurements, and evaluate the heart\u2019s function. Standard echo(a) (b)(c) (d)Figure 1.5: Standard cross sections for 2D echo images and the correspond-ing views of the heart: a) apical two-chamber (A2C), b) apical four-chamber (A4C), c) parasternal long-axis (PLAX), and d) parasternalshort-axis at the papillary muscle level (PSAX-PM). (View imagesadapted from Wikimedia Common).10views include:\u2022 Apical views:\u2013 apical two-chamber (A2C);\u2013 apical three-chamber (A3C);\u2013 apical four-chamber (A4C);\u2013 apical five-chamber (A5C);\u2022 parasternal long-axis (PLAX) view.\u2022 parasternal short-axis (PSAX) views:\u2013 PSAX at the apex level (PSAX-A);\u2013 PSAX at the mitral level (PSAX-M);\u2013 PSAX at the papillary muscle level (PSAX-PM);\u2022 subcoastal four-chamber (S4C) view;\u2022 suprasternal (SUP);\u2022 RV-focused views:\u2013 RV inflow tract (RVIT); and\u2013 RV outflow tract (RVOT).Figure 1.5 shows the cross-sectional planes corresponding to the most commonlyused of these views, i.e. A2C, A4C, PLAX, and PSAX.Clinical Guidelines for DiagnosticsDiagnosis of heart disease in echo involves often following extensive clinical pro-tocols and requires measuring tens of standardized parameters from echo data ofstandard views. Various aspects of echo-based cardiac indices are summarizedin Fig. 1.6. These measurements may describe size (e.g., length, area, volume),anatomical abnormalities (e.g., tissue abnormality, such as valvular calcification),11Figure 1.6: Categorization of cardiac indices and parameters derived fromecho. Colours represent aspects of determining these parameters: stan-dard echo views, mode of ultrasound imaging, temporal data dimension,and measurement method.systolic or diastolic function (e.g., cardiac output, ejection fraction, diastolic dys-function), motion abnormalities (e.g., wall motion abnormality, valvular motionabnormalities), etc. Amongst these parameters, evaluating the left ventricle (LV)function plays an essential role in patient management and prognosis [117]. LVsystolic function is linked with many cardiovascular conditions and is highly cor-12related with mortality rates [111]. The main parameters measured for LV functionassessment are EF, WMA, and global longitudinal strain (GLS).Left Ventricular Ejection Fraction (LVEF)Figure 1.7: Biplane Simpson\u2019s method of disks for measuring EF. LV is seg-mented in ED and ES frames of both A2C and A4C cine loops (a); LVvolume is calculated by summing over the estimated disk volumes (b).(Subfigure (a) from DocPlayer.net; and (b) from CardioServ.net.)Left ventricular EF (LVEF or EF in short) is the most-commonly measured in-dex for quantitative evaluation of cardiac function and is routinely measured in 2Decho. EF measures the percentage of blood that leaves the heart with each con-traction. For example, the LV function is the principal indication at the Vancouver13General Hospital (VGH) and the University of British Columbia (UBC) Echo Lab,a reported 80% of all echo requests. EF plays a significant role in many diagnosisprocedures, including heart failure. HF requires assessing the heart\u2019s contractileand relaxation properties with echo. EF, which is an estimation of the heart\u2019s con-tractile (systolic) function, plays a critical role in the categorization of HF. The ba-sic classification of HF with preserved, reduced, or mid-range EF is fundamentaland drives the treatment strategies. The current standard of care for EF quantifi-cation in 2D echo is Simpson\u2019s biplane method of disks [127]. In this method,the 3D structure of LV is modelled as a stack of circular disks visible in A2C andA4C views. To measure the end-diastolic volume (EDV) and end-systolic volume(ESV), LV volume is then computed as the sum of the volumes of the disks in theED and ES frames (Fig. 1.7), which are manually extracted by scrolling throughsimultaneously-acquired electrocardiogram (ECG) signals burned onto the echoimages. However, studies suggest manual segmentation-based measurement of EFsuffers from intra- and inter-user variability, especially among novice cardiolo-gists [33, 47]. However, LV segmentation is time-consuming and challenging dueto the presence of noise and unclear endocardial boundaries.Another more commonly-used EF estimation approach is through visual as-sessment of echo cine loops. Expert cardiologists often rely on direct visual esti-mation of EF in clinics [116]. Studies suggest that direct visual estimation of EFis closely correlated to quantitative segmentation-based techniques [82]. Extensivepractice and experience allow cardiologists to study cardiac cine loops (often inthe A2C and A4C views) and eyeball the approximate EF value. Visual cues suchas myocardial motion and atrioventricular plane displacements can reveal informa-tion on how much blood is being pumped out every cycle, which is EF by defi-nition. In addition, the visual estimation of EF is faster and less labour-intensivethan the biplane Simpson\u2019s method of discs. However, this method is a highlyreader-dependent technique, leading inexperienced novice operators to hesitate touse it [36, 116]. Moreover, eyeballing EF is not a reliable option for other clinicianswith limited echo training, such as point-of-care clinicians or first responders.14Figure 1.8: Cardiac wall motion and thickening. The LV wall consists ofthe inner and outer boundaries endocardium, pericardium (serous mem-brane lining), and the myocardium (muscular tissue). Blue arrows indi-cate systolic myocardial motion.Wall Motion AnalysisEchocardiographers can comprehensively evaluate LV function by studying theLV myocardial wall (Fig. 1.8) motion [129]. Wall motion analysis has provento be a significant factor in clinical decision-making for emergency patients withchest pain and those with congestive heart failure [138]. Detection of wall mo-tion abnormalities is essential for cardiovascular assessment and is highly cor-related with complications such as myocardial ischemia, CAD and myocardialinfarction [117, 127, 138]. According to the American Society of Echocardiog-raphy (ASE) recommendations, wall motion analysis is done visually in a semi-quantitative fashion, where thickening of the myocardial wall is studied in 16 in-dividual segments throughout a cardiac cycle [127]. A regional wall motion scoreindex (WMSI) is assigned to these 16 segments. Nonetheless, RWMA is a skillthat requires experience and expertise and is not usually undertaken by first-linecare providers.Strain Rate Imaging and Doppler Tissue ImagingStrain rate imaging (SRI) techniques evaluate the wall kinetics by quantifying itsphysical strain (Fig. 1.9). Doppler tissue imaging (DTI) is an SRI approach for15MyocardiumHeart ChamberStrain DirectionLV MyocardiumCircumferentialRadial LongitudinalFigure 1.9: The main directions of deformation and strain are imposed on theLV myocardium.quantitative evaluation of LV wall motion by studying the myocardium\u2019s peak tis-sue velocities [152]. A more recent approach for wall motion analysis is by speckletracking echo. Speckle tracking uses the gray-scale B-mode echo data to detectand track blocks of ultrasound speckles, i.e. constructive or destructive ultrasoundback-scatter from the myocardium [8, 127], and effectively the wall itself.With advancements in SRI, Global Longitudinal Strain (GLS) is becoming agrowingly popular cardiac parameter. GLS is measured as the overall longitudi-nal strain of LV, as shown in Fig. 1.9 It is currently dubbed the best quantitativedescriptor of systolic LV function. Furthermore, several studies suggest its supe-riority over EF for ventricular evaluation [111, 205, 209], especially in terms ofintra- and inter-observer reproducibility [134]. Nonetheless, GLS estimation re-quires 1) detection of the ES ejection phase; 2) speckle fiducial point selection onthe myocardium; 3) selection of regions of interest (ROI) for motion assessment;and 4) automatic frame-by-frame tracking of speckles in A2C, A4C, and PLAXviews [111]: While manual intervention and accurate wall tracking contribute tothe robustness of GLS, they limit the feasibility of GLS assessment in point-of-care, where minimum manual involvement is desired.Visual Assessment of Wall MotionIn clinics, wall motion analysis is done visually in a semi-quantitative fashion [118,127]. Specifically, the observed thickening of the myocardial wall is studied insmall individual segments throughout a cardiac cycle. A wall motion score index16Figure 1.10: Normal vs. dysfunctional regional wall motion and differentseverities.(WMSI) is then assigned to individual interest segments based on the severity ofdysfunction (Fig 1.10). As the nature of this method itself is subjective, WMAdetection and grading via visual assessment is highly observer-dependent and re-quires substantial experience and skills [138, 170]. An agreement between lessexperienced and expert clinicians is often below 50% [170].1.2.3 High Echo Demand and Importance of Timely DiagnosisAt the Vancouver General Hospital (VGH) and the University of British Columbia(UBC) Echo Lab, the largest echo laboratory in BC, and the second-largest lab-oratory in Canada, an average of 20,000 echo studies are performed each year.While echo provides detailed information on cardiac structure and function, full-length studies that we do in conventional echo laboratories are resource and labour-intensive. The workflow of an echo exam is approximately 30-60 minutes: 15-30minutes for the initial image analysis by the sonographer and 10-20 minutes for thereview, integration and report preparation by the examining cardiologists [214].Currently, the demand for echo exams significantly exceeds the available clini-cal attention and resources. In urban centers, the current reported wait time for echoin BC is 5-12 months. The demand for echo studies is expected to continuouslyrise with the aging population and increased risk of heart disease. Furthermore,therapeutic interventions are not equally accessible to heart disease patients, lead-ing to inequity in healthcare delivery with respect to sex, socioeconomic status andplace of diagnosis [128]. For example, many BC rural communities have no access17to echo machines.1.3 Towards Automated Echo Interpretation1.3.1 Machine Learning and EchoWith recent advancements in machine learning (ML), many research groups havefocused on developing automatic and semi-automatic techniques to assist cardiacassessment. Though data privacy issues remain cumbersome, large datasets aremade available to develop ML-based algorithms to move towards automated car-diac diagnostics. A rich body of literature focuses on eliminating labour-intensiveand error-prone steps in interpreting cardiac images, including echo.Anatomy Localization in B-ModeFrom a computer vision standpoint, most research for automatic cardiac measure-ments involves some degree of localization of heart anatomy (ventricles, atria, etc.)in B-mode images. In addition, automatic segmentation of heart chambers, espe-cially LV, has been attempted by many researchers [52, 56, 63, 101, 130, 147, 148,162, 165, 206, 245]. Compared to classical ML techniques, deep learning (DL) ap-proaches are particularly favoured for echo-based cardiac evaluation. DL methodsattempt to mimic human decision-making, are not limited to working memory, andhave proven helpful for unsupervised image recognition tasks, especially in noisymodalities such as echo [120]. In addition, several variations of semantic segmen-tation encoder-decoder networks (such as U-net) have been utilized to trace theanatomy of interest. In addition to echo, a rich body of literature revolves aroundDL-based cardiac segmentation [23, 32, 108, 141, 158, 210, 227, 238, 248]. Theserecent works include neural networks such as fully convolutional multi-scale resid-ual DenseNets[115] and multi-level convolutional LSTM (ConvLSTM) [243, 244].Speckle Tracking MethodsClassical ML methods used to classify wall motion and detection of WMA includeradial basis functions [49], random forest [67], unsupervised multiple kernel learn-ing [196], dictionary learning [173], support vector machines (SVM)-based wall18motion classification (in CMR) images [144]. In addition, Omar et al. proposed aconvolutional neural network (CNN) to distinguish between normal and abnormalwall motion in 3D stress echo [170]. In order to robustify tracking, many workshave focused on developing statistical spatio-temporal cardiac atlases, which relyon cardiac motion and shape priors [6, 186, 187, 211, 212, 226]. For example,Peressutti et al. combined a motion atlas with non-motion information to extractclinically relevant features. Spatiotemporal atlas [179]. Oktay et al. proposedan anatomically constrained neural network [168, 169]. To evaluate the wall mo-tion in CMR, Puyol et al. segments the myocardium, followed by feature trackingand strain estimation [188]. Nevertheless, STE is subject to limitations such assuboptimal tracking of endocardial borders, sensitivity to acoustic shadowing andreverberations [127].Doppler-based MethodsSome researchers have designed methods for automating this objective quantitativeapproach to assessing local myocardial dynamics [86, 127, 152, 182, 183]. Poreeet al. [182] combined DTI with B-mode optical flow, while Porras et al. [183]integrated tri-plane B-mode images (e.g. apical views) and DTI to track the LVwall. Nevertheless, DTI techniques suffer from inherent Doppler limitations: 1)tissue velocities can only be measured in the ultrasound direction, causing orthog-onal velocity components to be missed in measurements; and more importantly, 2)measurements made are highly susceptible to noise caused by global heart motionand blood flow [152].1.3.2 Challenges and Considerations for ML-based Echo Analysisand DiagnosticsDespite the advantages of echo imaging, interpretation of echo data is nontriv-ial, leading to high intra- and inter-observer variability in echo-based diagnos-tics [33, 47]. This is due to intrinsic ultrasound limitations (noise and speckles,frequency vs. depth trade-off, etc.), as well as the dependence of image qualityon the correctness of the acquired imaging planes. Additionally, LV function mea-surements (EF, WMA, and GLS) are particularly complex to measure as they rely19on videos acquired from several imaging windows. Nonetheless, the high spatio-temporal resolution of echo allows automatic machine-directed anatomy and func-tion evaluation [214]. Below are some of the main challenges and considerationsin designing ML-based solutions for cardiac evaluation.Expert Annotation Availability and Feasibility Trade-offsMost echo-based cardiac measurements, including LV function, require some de-gree of localization of a specific portion of the anatomy (LV blood pool for EF andmyocardial segments for wall motion analysis). Automatic solutions often utilizesegmentation and tracking to assist with this localization. On the other hand, suchtechniques are not preferred in point-of-care, as they: 1) add an intermediate stepto the workflow; and 2) are dependent on localization accuracy.Noisy Data and High Uncertainty in Clinical LabelsCardiac assessment in echo is an inherently difficult problem as echo is a noisymodality. In addition, echo-based measurements suffer from quality dependenceand high intra- and inter-observer variability. This leads to uncertain and noisylabels, which further adds to the complexity of automated solutions for cardiac as-sessment. Furthermore, most databases used for training diagnostic ML models arecreated based on selective extraction of patients with specific primary pathologiesand conditions [107]. This is while the presence and degree of various pathologiescan significantly alter the heart\u2019s appearance.Integrating Information from Multiple ViewsCardiac measurements often rely on cine loop analysis of various views. However,handling multiple videos for deriving measurements in a noisy modality is non-trivial, especially in the lack of obvious anatomical and temporal correspondenceacross multiple cine loops.Algorithmic Complexity of Deriving Cardiac IndicesTroubleshooting ML models require more domain knowledge in some cases com-pared to others. For example, tasks like segmentation of easily visible anatomy20(e.g., a blood chamber or a valve) are simpler to replicate artificially. In contrast,tasks that involve function, information from multiple views, patient history, andinter-sectionality of medical conditions are more sophisticated and require cardiol-ogy domain expertise.1.4 Thesis OverviewFigure 1.11: Overview of a supervised learning framework for automatic car-diac measurement extraction and heart disease prediction. For the i-thpatient that visits the echo clinic at time t, i.e. Pti , echo images Xtiare acquired. Measurements Y ti are derived from echo manually byclinicians or automatically by trained ML models, which can then beused to determine if the patient has a heart disease diagnosis duringthe visit (Dti). Chapters 2 and 3 focus on Xti \u00d0\u2192 Y ti for EF. Chapter 4investigates the generalizability of a model trained on an average co-hort on a strictly diseased wall motion abnormality cohort to evaluatethe performance of X ti \u00d0\u2192Y ti in the presence of severe cardiac condi-tions. Chapter 5 investigates the feasibility of using ML to determinediastolic dysfunction based on clinical measurementsi.e. Y ti \u00d0\u2192Dti .1.4.1 ObjectivesThe overarching goal of this thesis is to investigate the feasibility of supervised MLmodels for evaluating cardiac function and grading degrees of dysfunction. Wespecifically focus on the diagnostically-relevant methodology that acknowledgesand addresses:21\u2022 lack or limits of extensive clinical annotations and tracings to enable scalableand extendible ML-based solutions for various diseases;\u2022 noise and variability in clinical data and observations to promote robustnessin computer-generated predictions; and\u2022 inherent subjectivity of clinical guidelines for deriving measurements anddiagnoses, to model and integrate clinical knowledge and cardiovascular re-search legacy,To this end, the objectives addressed in the upcoming chapters are:\u2022 to visually assess EF for early HF diagnosis in echo without any segmenta-tion labels; (pink in Fig. 1.12);\u2022 to improve the robustness of EF estimation in echo by leveraging multipleavailable labels and their variabilities (pink in Fig. 1.12);\u2022 to visually detect RWMA in echo without myocardial annotations for earlyCAD diagnosis (orange in Fig. 1.12);\u2022 to assess diastolic function based on echo measurements and quantify leftventricular diastolic dysfunction (DD or LVDD) severity (lavender in Fig. 1.12).1.4.2 HypothesesIn line with the objectives above, the following hypotheses are investigated:\u2022 Hypothesis EF-1: ML can be reliably used in echo to detect patients at ahigh risk of heart failure with reduced EF (HFrEF) via visual assessment(Chapter 2).\u2022 Hypothesis EF-2: ML can be used for direct, precise estimation of EF for awide range of EF without segmentation (Chapter 3).\u2022 Hypothesis EF-3: Observer variability modelling in a multi-task frameworkis feasible to improve the robustness of a visual EF assessment (Chapter 3).22\u2022 Hypothesis RWMA-1: Direct visual analysis with ML is feasible for a com-plete systolic analysis, including detecting RWMAs (Chapter 4).\u2022 Hypothesis RWMA-2: Spatio-temporal LV segmentation can withstandvariations and abnormalities in the presence of RWMAs (Chapter 4).\u2022 Hypothesis DD-1: Updated clinical guidelines majorly impact LVDD di-agnosis based on echo measurements and demographic information (Chap-ter 5).\u2022 Hypothesis DD-2: Supervised ML can be reliably used to learn and predictthe latest clinical guidelines for LVDD diagnosis based on measurementsextracted from echo (Chapter 5).23Figure 1.12: Overview of thesis focusing on using ML for systolic (EF andRWMA) and diastolic function assessment in echo. Each row repre-sents a hypothesis and the corresponding carried out studies. Chap-ters 2 and 3 focus on EF estimation based on echo images. Chapter 4investigates visual assessment and endocardial segmentation for wallmotion analysis in echo videos. Chapter 5 explores the feasibility ofML to automate LVDD classification based on clinical measurements.Although this thesis focuses not on the outcomes themselves, the po-tential adverse outcomes linked to the diagnoses are highlighted to putthe work in the clinical context.241.4.3 Thesis Context and ScopeOur group has made strides in investigating the feasibility of ML and computer vi-sion (CV) tools for assistance with echo acquisition and automating echo-derivedmeasurements and observations. Figure 1.13 maps out the active areas of this re-search (referred to as INformation FUSion for Echocardiography or INFUSE inthis manuscript) and highlights this thesis\u2019s chapters in the bigger context. The redbox indicates the scope of this thesis. Purple indicates research by Jafari and Sahe-bzamani, which focused on CV methodology for indirect measurement, i.e. via LVlandmark detection and endocardial segmentation. This research can be broadlycategorized into three themes based on ultimate objectives:\u2022 ML for assisting echo acquisition: e.g. automated quality grading, echoview classification and quality enhancement (green in 1.13), as well as ultra-sound probe navigation;\u2022 ML for automating echo measurements diagnostics: including for EF,RWMA, LVDD (purple and red in 1.13), arrhythmia (blue), valve disease,etc.; and\u2022 ML for auxiliary tasks in echo: that indirectly assist with acquisition, e.g.view classification (green [136, 225, 234]), and diagnostics (blue), e.g. byenabling temporal data processing (phase detection [58, 59]) and temporalintegration of multiple cines (echo synchronization [61]).1.4.4 Summary of ContributionsThe thesis chapters are summarized below, highlighting the novelty and signifi-cance.Chapter 2: Automatic Risk-based Classification of Ejection Fraction in EchoChapter 2 proposes a novel framework for segmentation-free direct assessment ofEF in echo using a dual-stream spatio-temporal network to identify patients withlow EF who are at risk of heart failure with reduced EF (HFrEF). This frameworkconsists of 2D convolutional neural networks (CNNs) for creating frame-level fea-ture vectors, which then feed to recurrent neural networks (RNNs) for temporal25aggregation. We have reported experiments investigating several different neuralnetwork architectures, variations of CNNs and RNNs, and impacts of feature ex-traction based on a single view (A2C or A4C) and dual view (synchronized A2Cand A4C). By directly classifying EF as an indicator of global systolic function,we have shown for the first time (in [25, 28]) that ML can be used to eliminate theneed for ground-truth segmentation for echo measurements. Accurate clinical seg-mentations are expensive to acquire and suffer from observer variability, affectingthe quality of automated predictions.Chapter 3: Dual-view Joint Estimation of EF with Uncertainty Modelling inEchoChapter 3 presents a novel dual-view pseudo-siamese framework that utilizes athree-dimensional (3D) architecture for spatio-temporal embedding and simulta-neously maps the extracted features to four EF outputs available in our clinicaldataset, suggesting high variability. The four EF labels are 1) biplane visual EF(categorical), 2) biplane Simpson\u2019s EF (continuous), and 3,4) single-plane EF fromA2C and A4C. The network hence concurrently performs one classification andthree regression tasks. Each output is connected to the appropriate input via theA2C, A4C or biplane stream. We define a multi-term hybrid objective function forthe model parameter optimization.We additionally propose a novel observer variability modelling on the net-work output, where the uncertainties of the clinical labels are assumed to followa Gaussian distribution characterized by a mean and variance, which the modelcan implicitly predict. A novel objective function is defined and implemented asa Gaussian probability distribution function (PDF), and a cumulative distributionfunction (CDF) is used for the regression and classification tasks. We present ourresults suggesting that this variability modelling improves the prediction accuracy.Moreover, reporting a distribution or range for the prediction, rather than a point-estimate, sheds light on the model confidence and, subsequently, the non-expertuser\u2019s trust in the computer-generated prediction.26Chapter 4: Machine Learning for Left Ventricular Wall Motion Analysis inEchoChapter 4 we report on experiments exploring the feasibility and generalizabilityof a segmentation-free tri-plane wall motion analysis using spatio-temporal neuralnetworks. Chapter 4 investigates the use of direct and indirect ML-based frame-works for capturing regional abnormalities in LV systolic function from relevantecho cines).We study the ML-predicted echo quality of the cines in the wall motion ab-normality cohort to identify the optimal imaging planes recommended for a noviceimager, e.g. at point-of-care. This novel application of echo view and qualityclassification [136] enables researchers to determine preferred disease-specific di-agnostic echo views by reviewing large pathologic datasets automatically.We briefly present experiments with the apical tri-plane prediction of RWMA,EF, and global wall motion abnormality (GWMA) based on A2C, A4C, and PLAXcines of patients with at least one abnormal LV segment. Nonetheless, the availabledata could not achieve such a direct visual assessment.Chapter 4 then presents our novel automatic framework to quantify the regionalsegmentation errors for 12 LV segments visible in A2C and A4C views without ad-ditional fine-grained expert annotations. We have investigated the performance ofLV segmentation models previously trained on average distributions on a strictlydysfunctional cohort. We have proposed a novel regional endocardial wall distancemetric to obtain local errors in wall detection in the presence of pathology. Finally,we have performed a comprehensive analysis of the ML-predicted results with re-spect to systolic function parameters, i.e. EF, GWMA, and EF, as well as fairnessaspects such as patient\u2019s age, sex and body-mass index (BMI).Chapter 5: Automatic Diastolic Dysfunction Diagnosis from Echo-derivedParametersChapter 5 shifts gears to diastolic function and explores the inherent reproducibil-ity of LVDD diagnosis according to the latest global guidelines and the utility ofsupervised ML for learning these clinical guidelines for LVDD diagnosis.Chapter 5 first outlines a comprehensive comparative study of diastolic func-tion assessment based on the American Society of Echo (ASE) 2009 Guidelines27and the updated ASE\/EACVI (European Society of Cardiology) 2016 Guidelines.We implement the two methods and compare the LVDD outputs based on pa-rameters and measurements related to diastolic function available in the clinicaldatabase. We report our statistical findings, which suggest significant differencesin the interpretation of diastolic function using the two methods.Chapter 5 then proposes a novel DL-based approach for LVDD predictionbased on echo-derived measurements available in our local hospitals\u2019 database.We employ a feed-forward network to learn to classify four determinate categori-cal LVDD labels (normal, mild, moderate or severe dysfunction) derived using theaforementioned ASE\/EACVI 2016 algorithm. We show that the neural networkcan successfully implicitly learn the diagnostic algorithm and predict the presenceof LVDD based on relevant parameters. We further propose a novel continuousLVDD scoring framework by extending the network to perform a regression task.This is the first use of ML for diastology and the first method for assigning contin-uous diastolic function scores.28Figure 1.13: Overview of the INFUSE projects that utilize ML towards au-tomating echo examination. The main goals are to assist with echoacquisition and interpretation for diagnostics, and auxiliary modulesindirectly enable these goals. The bigger red box represents the scopeof this thesis, and enclosed smaller outlined boxes represent chaptersof the thesis, with collaborations with Asgharzadeh and Kazemi Es-fehani et al. highlighted in orange and gray. Other colours representadjacent research by peers (blue for Taheri Dezaki, purple for Jafariand Sahebzamani). Items outside coloured boxes are active projects inpreliminary stages at the time of submission of this thesis.29Chapter 2Automatic Risk-basedClassification of EjectionFraction in Echo2.1 Introduction2.1.1 Clinical BackgroundEjection Fraction (EF) in EchoLeft ventricular ejection fraction (LVEF or EF) is the single most commonly mea-sured cardiac parameter in echo [36]. EF is defined as the ratio of the volume ofblood pumped out of the end of every systole (ESV) and the maximum amount ofblood in LV at the end of diastole (EDV), describes the performance of the heart asa pump, according to Equation 2.1 below:EF = EDV \u2212ESVEDV\u00d7100% (2.1)30EF is the critical predictor of prognosis in most cardiac conditions, including valvedisease, coronary artery disease, and heart failure [36]. Low EF is associated withhigh mortality rates, with a reported 5-year survival rate of 64.8%, and a 10-yearsurvival rate was 44.7% [207] after bypass surgery in coronary artery disease pa-tients. The standard-of-care for EF assessment in echo is the Biplane Simpson\u2019smethod of disks (as shown earlier in Fig. 1.7). In the Simpson\u2019s method, the area ofthe LV is calculated in the maximum (ED) and minimum (ED) points in both apicaltwo-chamber (A2C) and apical four-chamber (A4C) views. Volumetric assessmentis then performed by modelling the LV as a stack of disks with infinitesimal heightsand diameters derived from the traced LV blood pool.2.1.2 Related WorksMachine Learning for Segmentation-based EF AssessmentMany works have focused on automating LV segmentation for measuring EF inecho images. Several machine learning (ML)-based solutions have been recentlyproposed for automatic LV segmentation ([38, 56, 63, 101, 130, 162, 245]). EFestimation is challenging due to its dependence on\u2022 accurate LV segmentation;\u2022 accurate detection of the main LV axis; and\u2022 extraction of ED and ES frames.Even amongst cardiologists with echo training, studies reveal high intra- and inter-user variability, particularly those with less experience ([33, 47]). In order to im-prove the segmentation, some works propose incorporating domain-specific fea-tures [99] and sequential learning via the use of recurrent neural networks (RNNs).This chapter was adapted from i) D. Behnami, C. Luong, H. Vaseli, A. Abdi, H. Girgis, D. Haw-ley, R. Rohling, K. Gin, P. Abolmaesumi, and T. Tsang. Automatic detection of patients with a highrisk of systolic cardiac failure in echocardiography. In Deep Learning in Medical Image Anal-ysis and Multimodal Learning for Clinical Decision Support, pages 65\u201373. Springer, 2018; andii) D. Behnami, C. Luong, H. Vaseli, H. Girgis, A. Abdi, D. Hawley, K. Gin, R. Rohling, P. Abol-maesumi, and T. Tsang. Automatic cine-based detection of patients at high risk of heart failure withreduced ejection fraction in echocardiograms. Computer Methods in Biomechanics and BiomedicalEngineering: Imaging & Visualization, 8(5):502\u2013508, 2020.31However, though promising for LV volume estimation in a given frame, these meth-ods can lack robustness for EF prediction. This is due to the dependence of EF onaccurate LV tracing in ED and ES and the selection of ED and ES frames them-selves. To help extract these key frames, automatic cine-based phase detectiontechniques have been previously proposed [58, 74, 119, 252]. However, the mostcommon approach for cardiac phase detection is through LV segmentation itself,assuming that ED and ES frames correspond with the largest (EDV) and smallest(ESV) volumes.Visual Assessment and Eliminating Reliance on SegmentationAn alternative and more workflow-efficient technique for EF evaluation is a directvisual assessment of echo cine loops. This rapid method of EF estimation, morecommonly employed by experienced cardiologists ([116]), has been shown to cor-relate well with the volumetric segmentation-based approach ([82, 232]). Never-theless, direct EF estimation relies heavily on the interpreter experience and maynot be appropriate for novice clinicians who have not undergone sufficient train-ing ([36, 116]).Several studies have investigated the direct estimation of LV pa-rameters has been investigated in CMR by several studies [65, 81, 110, 235, 236,246, 247]. These measurements include LV volume and EF [65, 81, 110, 236],outflow tract classification [167], etc. Xue et al. has proposed LV quantification bydeep multitask learning [237]. These works have shown the feasibility of machinelearning techniques, especially deep neural networks, for EF analysis. Kabani andEl-Sakka used convolutional neural networks (CNNs) to pre-localize and LV seg-ment for volume estimation and subsequently EF calculation. Xue et al. similarlyrelied on a deep CNN to extract features and RNNs to aggregate them for the dy-namic embedding of the LV motion, etc. Compared to CMR, standard echo imag-ing planes introduce more significant variance in the appearance of the LV anatomyin 2D echo images. Moreover, the PSAX views used for LVEF estimation in MRcapture a much simpler cardiac motion (2.1a) and field-of-view compared to theviews used in echo (Fig. 2.1b and 2.1c).32SystoleDiastole(a)Systole Diastole(b)Systole Diastole(c)MyocardiumCavityLVFigure 2.1: Comparison of LV motion in ES and ED phases of PSAX (a),A2C (b) and A4C (c). Deformations, and movements of chambers andvalves are more complex in A2C and A4C (used in echo) compared toPSAX (used in CMR), causing echo-based LV assessment to be moredifficult.2.1.3 Challenges of Visual EF Assessment in EchoDirect cine-based EF estimation in echo is a challenging problem:\u2022 Ultrasound images are inherently noisy and yield blurry chamber bound-aries, making LV size changes difficult to quantify.\u2022 The quality of the 2D cross-sectional echo images is variable due to de-pendence on the experience level of the echo operators. Furthermore, theultrasound quality may worsen with age, disease, and obesity, which coinci-dentally are correlated with heart disease risk.\u2022 Cardiac anatomy and function are complex and variable, especially in theapical views (Fig. 2.1). The complexity is compounded by variations in the33appearance of the imaged heart in the presence of pathology.\u2022 Echo images only capture 2D cross-sectional views of the heart, which maybe foreshortened. LV foreshortening may inevitably lead to incorrect mea-surements, even if the image analysis task is accurate.\u2022 To perform a more accurate volumetric assessment, two A2C and A4C viewscan be used. In addition to doubling the acquisition and analysis, the fusionof the two measurements derived from the two views is not trivial in thebiplane EF assessment.2.1.4 Chapter OverviewLV boundary at ESLV boundary at ED\ud835\udc38\ud835\udc39 =\ud835\udc38\ud835\udc37\ud835\udc49 \u2212 \ud835\udc38\ud835\udc46\ud835\udc49\ud835\udc38\ud835\udc37\ud835\udc49Sufficient LV volume changeSystolic LV deformationLow-risk\ud835\udc38\ud835\udc39 > 40%High-risk\ud835\udc38\ud835\udc39 < 40%Insufficient LV volume changeFigure 2.2: EF can be evaluated by studying the dynamics of the heartthroughout the cardiac cycle. Low-risk (left) and high-risk (right) sys-tolic performance are associated with sufficient and insufficient changesin the LV volume, respectively. The solid contours depict the ES volume(ESV), and the dashed contours show the ED volume (EDV).This chapter investigates the feasibility of automatic segmentation-free binaryclassification of EF using several different architectures for spatio-temporal featurelearning of echo cine loops. We present a deep neural network that mimics theclinicians\u2019 eye-balling technique in echo to help classify exams as high-risk. (EF \u226440%) or low-risk (40% < EF \u2264 75%), as shown in Fig. 2.2.We show that the proposed dual-channel can be effective in direct EF estima-tion. This two-step approach involves 1) extraction of frame-level spatial features34and 2) aggregation of these feature maps using recurrent networks, which enablesequence learning. In order to incorporate both A2C and A4C cines in decisionmaking, we synchronize the two cines based on the cardiac phase. Furthermore,the cine-based framework allows for direct estimation of EF via analysis of cineloops, therefore eliminating the dependence on accurate LV segmentation and ex-traction of ED and ES frames. The following contributions are made:\u2022 Our approach directly estimates EF from echo cine loops, eliminating theneed for LV segmentation and detection of key cardiac frames. LV segmen-tation can be challenging due to the high variability in echo image qualityand image settings, as well as variability in the operator\u2019s experience in ob-taining the correct echo standard views;\u2022 We propose a dual-stream framework for A2C and A4C views, consistingof view-specific spatial feature extraction blocks as well as shared recurrentneural network (RNN) layers;\u2022 We report the performance of several state-of-the-art networks and empiri-cally show that all the dual-view frameworks perform equally or better thana single apical view in classifying low-risk vs. high-risk EF.2.2 Material and Methods2.2.1 Echo Clinical DatabaseEthics approval was obtained from Vancouver Coastal Health (VCH) to access thisdata for research. As depicted earlier in Fig. 1.4, our echo data consists of an echoimage database (denoted DEcho) and the corresponding measurements and pathol-ogy reports (DFileMaker). DEcho contains echo cine loops, measurement screenshots,Doppler cines and images, etc., from a decade of Vancouver echo data. DFileMakeron the other hand, includes patient (e.g., name, age) and exam information (e.g.,date, examiner), diagnostic information for standard measurements, comments,etc. Exams in DFileMaker the clinical report recorded DEcho were linked togetherusing the hospital-assigned patient identification information and acquired examdate. All exams were acquired using Philips iE33 ultrasound machines and had35a frame resolution of 800\u00d7600 pixels. Frame rate (FR) and heart rate (HR) varyacross patients; FR of 25\u221266 Hz and HR of 60\u2212100 beats per minute. 80% of thedata was used for training and the remainder for testing.Clinical EF Labels in DFileMaker and Patient CohortEF labels used in this study were recorded by expert echocardiographers using thebiplane Simpson\u2019s method. The exams were reviewed and A2C and A4C cineloops were extracted using the echo view classifier proposed by ([234]). In orderto gather the data needed for automatic EF estimation, the DFileMaker database wassearched for echo samples with EF measurements. Fields dedicated to EF include:\u2022 EFA2CSimpson\u2019s and EFA4CSimpson\u2019s; calculated by the examining cardiologist usingSimpson\u2019s method on A2C and A4C images, respectively.\u2022 EFBiplaneSimpson\u2019s; calculated as12 \u00d7(EFA2CSimpson\u2019s+EFA4CSimpson\u2019s); when both EFA2CSimpson\u2019sand EFA4CSimpson\u2019s exist.\u2022 EFEyeballed; categorical label (as shown in Table 2.1) estimated by the exam-ining cardiologist via visual inspection of LV in the cine loops.Class Index EFEyeballed Assessment of Systolic Function1 < 20% Severe dysfunction2 20\u221235% Moderate-severe dysfunction3 35\u00b110% Moderate dysfunction4 35\u221250% Mild-moderate dysfunction5 50\u00b110% Mild dysfunction6 50\u221265% Lower limits7 65\u00b110% Normal8 > 75% HyperdynamicTable 2.1: Default values of EFEyeballed in DFileMaker database.Each exam in DFileMaker can contain any combination (or none) of the abovefour EF labels. An analysis of the samples with both EFBiplaneSimpson\u2019s and EFEyeballedfields available revealed non-negligible discrepancies between these EF labels (morein the next chapter, Fig. 3.2).36A subset of DEcho was downloaded for EF experiments according to the follow-ing criteria: 1) EFBiplaneSimpson\u2019s and EFEyeballed EF labels were recorded in DEcho, andin agreement; 2) correspondences could be found between DEcho and DFileMakerbased on the study identification information. All studies were anonymized, as re-quired by our ethics application. The data has been kept on encrypted drives on theuniversity servers.2.2.2 Data PreparationExtraction of Relevant Echo ViewsDuring an echo exam, clinicians acquire many studies on various cardiac views.Initially, the downloaded database was manually inspected in a smaller feasibilitystudy to extract A2C and A4C cine loops. However, each echo exam (acquiredon a given day from a given patient) in DEcho can contain up to 300 cine loopsand screenshots. Hence, manual extraction of views relevant to studies is not fea-sible for a higher number of samples. An automatic view detection approach wasdesigned, implemented, and validated in our group to alleviate this issue. Thismethod utilizes a deep CNN-based network to classify ten standard cardiac viewsfrom echo data. A2C and A4C were extracted using this view classifier, and examswith both these available views were kept.Biplane Cine Loops SynchronizationDirect EF estimation relies on dynamic information available in both A2C and A4Ccine loops. From a technical standpoint, synchronization reduces the dimensional-ity of the data by concatenating in-phase view-specific spatial features of differentviews, hence, achieving more efficient dual-channel sequential learning. In orderto synchronize the two cine loops, one cardiac cycle is extracted by detecting thefirst two consecutive R-peaks in the electrocardiogram (ECG) signal burnt onto theecho images. In each view, this is done by extracting the green-hued ECG signal isextracted from the final frame. The R-peaks and ED frames are detected by findingthe local maxima in the ECG signal.Cines are then temporally sampled from one full visible cycle in each cine loop37AxC, where AxC \u2208 {A2C,A4C}. To extract one cycle from each AxC cine, we findthe of R peaks from the PQRST sequence (Fig. 2.4a) in its available ECG and trimthe cine to frames RAxC1 to RAxC2 . An equal number of F = 25 frames are uniformlysampled from each sequence (Fig. 2.4b). This number of frames was selected basedon the shortest full-cycle sample present in the dataset (25 frames), and is in agree-ment with a previous work that demonstrated the feasibility of similar networks forphase detection in echo [58] (30 frames). Each cine loop AxC, AxC \u2208 {A2C,A4C},is then trimmed between two R-peaks RAxC1 to RAxC2 . This step effectively synchro-nizes the two trimmed cine loops based on their phase relative to the cardiac cycle,even if FR and heart rate (HR) vary. The images are cropped down to the ultra-sound beam. Finally, a fixed beam-shaped binary mask is applied to the frames toremove image annotations. The frames are then scaled down to 128\u00d7128 pixels.If Ii denotes the i-th processed frame of the 2D grayscale echo cine clip, any cinein AxC view can hence be described as a 3D tensor XAxC = [IAxC1 \u2236 IAxCCLAxC]; whereCLAxC is the cycle length, and CLAxC = RAxC2 \u2212RAxC1 +1. 2D frames of 800\u00d7600pixels are cleaned using a binary beam-shaped mask, cropped around the beamarea, and downsized to 128\u00d7128 pixels.Binarized Risk-based EF LabelsWe formulate the detection of patients at a high risk of heart failure with reducedEF (HFrEF) as a binary classification problem for evaluating YHigh\u2212risk, where:YHigh\u2212risk = 1\u2212 [EF40 ]. (2.2)EF is the ground-truth EF label for each exam calculated using Simpson\u2019s method,as recorded in the corresponding clinical report, and YHigh\u2212risk \u2208 {{0},{1}}. ([.]operator takes the integer part of its input). By definition, 0% \u2264 EF \u2264 100%. How-ever, only samples with EF \u2264 75% were included in this study due to the lack ofhyperdynamic (EF > 75%) cases in the database. As the EF labels are very noisywith discrepancies (more in Chapter 3), we focus on distinguishing between thelow-risk and high-risk EF classes for this task.Let EFBinary denote the risk-based binary labels. We define EFBinary such thatEFBinary = 1 for EFSimpson\u2019s \u2264 40%, and EFBinary = 0 for 40% <EFSimpson\u2019s \u2264 75%.38Figure 2.3 visualizes the clinical labels in the database (EFSimpson\u2019s and EFEyeballed)and the derived risk-based binary labels used in the present classification network(EFBinary). Cases with EFSimpson\u2019s > 75% are excluded from this study due to thevery limited number of samples. A total of 1,186 samples with the above criteriawere gathered; 541 high-risk and 645 low-risk cases. The dataset was divided intoa 4 \u2236 1 ratio for training, validation and test.39(a)(b)(c)Figure 2.3: EF labels in DFileMaker (a), i.e. EFSimpson\u2019s and EFEyeballed, andassigned risk-based labels EFBinary used in this section (b). EFSimpson\u2019sis a manually entered value based on Simpson\u2019s method of disks, whileEFEyeballed is an estimated value in eight fixed values. (See Table 2.1 fordetails on these classes.)40Ventricular diastole Ventricular systole Ventricular diastoleOne cardiac cyclePQRST(ED)(ES)(a). . . . . . ED ES EDR1 R2A4C. . . . . .\ud835\udc87\ud835\udfcf \ud835\udc87\ud835\udfd0 . . . . . . . . . . . \ud835\udc87\ud835\udc6d\u2212\ud835\udfcf \ud835\udc87\ud835\udc6dR1 R2 R3A2CA2C(b)Figure 2.4: The cardiac phase is extracted from ECG. Subfigure (a) showsthe PQRST complex visible in ECG and (b) shows an example of syn-chronized A2C and A4C echo cines. Cines are temporally resampledbetween consecutive ED frames, i.e. RAxC1 to RAxC2 , and effectively syn-chronized.412.2.3 Neural Network ArchitectureThe architecture of the proposed network for binary classification of EF is shown inFig. 2.5. This network consists of spatial feature extraction blocks and RNN-basedlayers for temporal learning.Dual-view Spatial Feature LearningWe experimented with various architectures, including CapsuleNet [194] and DenseNets[94] for frame-wise spatial feature extraction. In order to obtain a compact repre-sentation of the content of the frame images IAxCi , we use a DenseNet-like ([94])architecture depicted in Fig. 2.5b. The i-th frame IAxCi is fed through a 2D convo-lutional layer, followed by a cascade of three interconnected dense blocks and twotransition blocks. If ZAxCi denotes the output feature vector of the DSFE block forframe IAxCi , the input cine XAxC can be compactly described as [ZAxC1 \u2236 ZAxCCLAxC].The dense blocks are designated with eight convolutional channels in the first blockand a growth rate of 12. In order to contain the network size and parameters, withthe network complexity increasing with extra connections, a transition block isdesignated after the first two dense blocks. The transition block consists of a con-volutional layer, followed by a pooling layer. As suggested by Huang et al., a com-pression factor of 0.5 is used in the transition layers[94]. The dense block consistsof a batch normalization layer, a rectified linear unit (ReLU), and a 2D convolu-tional layer. Batch normalization reduces the input variance, therefore improvingtraining stability. ReLU activation, described as f (x) = max(0,x), introduces andpromotes non-linearity in the network. The main characteristic of the DenseNetarchitecture is that each layer is interconnected with all those that precede it. Thisprovides multiple alternate paths for the gradients to flow back through the networkduring training, leading to more effective learning, especially when dealing withsmaller datasets ([94]).Initially, sampled synchronous A2C and A4C frames are fed into feature ex-traction blocks. The flattened output of an feature extraction for a frame t is afeature vector XAxCm,t of length M\u00d71; m = 1 \u2236M. In the dual-view framework, XA2Cm,tand X4XCm,t are then concatenated to form a dual-view feature vector XA2C+A4Cm,t oflength 2M\u00d71. For an exam with two streams and sequence length of F frames, a42feature matrix XA2C+A4Cm,t of size 2M\u00d7F is constructed, where t = 1 \u2236 F . XA2C+A4Cm,tis a dense representation of the cardiac cycle based on two views.RNNs for Temporal Encoding of Cine LoopsThe other vital components in the network are the RNN blocks, which enable se-quential and temporal learning. In the standard RNN, at time step i, the i-th step ofthe sequence and the outputs from all previous steps are fed through a dense layer.This effectively allows the system to remember information from past time stepsand encode the sequence. A significant drawback of RNNs is that as the sequencelength increases, the gradient from earlier time steps starts to vanish, leading to theloss of information from the earlier phase of the sequence.One way to alleviate this issue is by using Gated Recurrent Units (GRU). Inaddition to the main architecture of an RNN unit, a GRU also contains an updateand a reset gate [46], which help selectively choose whether features should be keptor discarded. This mechanism enables GRU to remember relevant informationfrom past time steps. The choice of the bidirectional GRU (bi-GRU) is rootedin the fact that in bidirectional RNNs, the output of each time step is calculatedbased on both past and future steps. This gives the network more context aboutthe sequence at hand and is applicable for EF analysis as our sequential data (echocine) is periodic. We investigated various RNNs, including cascades of uni- and bi-directional Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU).The extracted frame-level features [ZAxC1 \u2236ZAxCCLAxC] are concatenated synchronouslyand fed through a bi-GRU with 128 hidden cells.Auxiliary and Feed-forward LayersDropout and batch normalization layers are used after feature extraction blocks toprevent overfitting. The spatiotemporal feature maps, i.e., the output of bi-GRU,are then passed through a dense layer with 1000 neurons with ReLU nonlinearity.The choice of the 1000-neuron layer was inspired by the original ImageNet chal-lenge [54], which contained 1000 classes. Finally, a dense layer with two neuronsand softmax activation, i.e., f (x) = 11+e\u2212x , is used to map these dense features toone of the two possible classes, high risk and low risk.432.2.4 Implementation and Model TrainingThe proposed framework is implemented in Keras with TensorFlow backend ([45]).The entire network was trained from scratch with random weight initialization. Thenetwork was trained end-to-end on an Nvidia GeForce GTX 980 Ti, with stochas-tic gradient descent (SGD) parameter optimization. Stratified batch sampling wasused to compensate for the class imbalance. The TimeDistributed wrapper wasused to apply the DSFE operations to all frames. The weights of the DFSE blockare updated separately for A2C and A4C views. However, the parameters of theDFSE block are shared between F frames of each view as the image content doesnot change significantly throughout a cine loop. The start point of the synchro-nized input cines was shuffled in training within the range of RAxC1 to RAxC2 . Aug-mented data are created on the fly via randomly generated transforms, includingrotation, scaling (\u00b120%), cropping (20% in each direction) and gamma transforma-tion (0.4 1.4) on the intensities. These transforms are generated randomly withinthe aforementioned ranges and applied to the whole cine at hand for each cine.The total number of augmented samples corresponds with the number of epochsas each iteration, a new of transforms are generated and applied on the input cineloop.44DFSE\ud835\udc70\ud835\udc39\ud835\udc342\ud835\udc36\ud835\udc3c1\ud835\udc342\ud835\udc36\ud835\udc7f\ud835\udc342\ud835\udc36Concatenate\ud835\udc81\ud835\udc39\ud835\udc342\ud835\udc36\ud835\udc811\ud835\udc342\ud835\udc36 Bi-GRU (128)A2C ChannelDFSE\ud835\udc70\ud835\udc39\ud835\udc344\ud835\udc36\ud835\udc3c1\ud835\udc344\ud835\udc36\ud835\udc7f\ud835\udc344\ud835\udc36\ud835\udc81\ud835\udc39\ud835\udc344\ud835\udc36\ud835\udc811\ud835\udc344\ud835\udc36A4C Channel\ud835\udc81\ud835\udc39\ud835\udc342\ud835\udc36+\ud835\udc344\ud835\udc36\ud835\udc811\ud835\udc342\ud835\udc36+\ud835\udc344\ud835\udc36ReLU(1000)\ud835\udc4cHigh-riskDual-channelSoftmax(2)(a)Conv2D (3x3)Batch NormalizationReLUConv2D (3x3)Conv2D (3x3)Max Pooling (2x2)ReLUDense BlockTransition Block\u00d7 3\ud835\udc81\ud835\udc56\ud835\udc34\ud835\udc65\ud835\udc36\ud835\udc70\ud835\udc56\ud835\udc34\ud835\udc65\ud835\udc36\u00d7 2(b)Figure 2.5: The architecture of the proposed dual-view network for risk-based classification of EF. Frame-level image features are extracted foreach channel, then synchronously concatenated and fed through a bi-GRU for temporal embedding (a). The architecture used in the DSFEblock consists of 2D convolutional layers, three interconnected denseblocks, two transition blocks and ReLU nonlinearity (b).452.3 Experiments and ResultsClassification Accuracy (%)505560657075808590DNet+u-LSTMDNet+b-LSTMDNet+u-GRUDNet+b-GRUCNet+u-LSTMCNet+b-LSTMCNet+u-GRUCNet+b-GRUDNet+u-LSTMDNet+b-LSTMDNet+u-GRUDNet+b-GRUCNet+u-LSTMCNet+b-LSTMCNet+u-GRUCNet+b-GRUDNet+u-LSTMDNet+b-LSTMDNet+u-GRUDNet+b-GRUCNet+u-LSTMCNet+b-LSTMCNet+u-GRUCNet+b-GRUA2C A4C A2C+A4CView(s):Architecture:Figure 2.6: EF classification accuracy using DenseNet (DNet) and Capsu-leNet (CNet) as the spatial feature extraction and various RNN versionson A2C, A4C and synchronous A2C+A4C views.Using the proposed dual-channel method, an overall accuracy of 83.15%, the pre-cision of 82.6% and recall of 81.1% were achieved. The test accuracy for binaryclassification based on A2C, A4C, and synchronized A2C+A4C is provided in Ta-ble 2.2. Quantitative results obtained in this study are demonstrated in Fig. 2.6. Thehighest performance is achieved using the dual-view approach with DenseNets andbidirectional GRUs. Using the same experimental set-up, we observed that A4Cis a more reliable view for direct evaluation of EF compared to A2C. This ob-servation is most likely because 1) A2C is a generally more challenging view tocapture, which often leads to suboptimal endocardial border definition as comparedto A4C; 2) the LV is more likely to be foreshortened in A2C images compared toA4C ([166]), and 3) A4C might benefit from more context-relevant informationfrom four chambers. We observed that better performance was obtained when thenetwork saw both A2C and A4C views. Table 2.3 shows the results broken downinto four ranges, each described by (EFmin,EFmax] intervals. The number of totaland test samples are provided as well. The higher number of samples in the low-risk range compared to high-risk may have contributed to the difference in successrates. Our quantitative results suggest misclassification occurs more frequently atthe hard threshold of EF = 40%.Visualized ResultsFigure 2.7 shows several pairs of A2C and A4C samples. The ES and ED frames(the extreme frames in terms of the appearance of LV) are shown as overlaid image46EF range (0,40] (40,75]YHigh\u2212risk 1 0A2C 68.5% 73.8%A4C 77.5% 81.7%A2C+A4C 81.1% 84.9%Table 2.2: Classification accuracy was obtained based on A2C, A4C andA2C+A4C cine loops using the proposed DenseNet-based DSFE and bi-GRU.EF range (0,25] (25,40] (40,55] (55,75]A2C+A4C 84.9% 78.4% 80.5% 86.5%Total samples 194 330 408 254Test samples 40 71 75 51Table 2.3: A breakdown of the EF classification accuracy based on EF rangeand number of samples.pairs. Green (ED) and purple (ES) areas suggest motion and absolute changes inthe position of the imaged anatomy. In the LV region, large purple regions cor-respond to significant LV volume changes in the systolic phase (low-risk EF). Asdepicted by these images, we observed that such systolic volume changes of theLV were more directly apparent in the A4C view compared to A2C. This observa-tion is consistent with the quantitative results. We also found that the quality of thecaptured plane, especially A2C, and the accuracy of the ECG-based synchroniza-tion are determining factors for the performance of the proposed model. A2C andA4C views are shown on the left and right of each pair, respectively. The imageoverlays indicate the cycle\u2019s ED (green) and ES (purple) frames for better visual-ization. Y PredictedHigh\u2212risk shows the prediction of the network for the given test sample.The large purple regions correspond with a significant volume difference betweenES and ED frames (ESV and EDV) and low-risk EF (Fig.2.7a). Volume changesmay appear more subtle in the A2C views (Fig. 2.7b). High-risk EF correspondswith small LV volume changes, hence smaller purple regions in 2.7c. The qualityof the acquired planes and the closeness to the sharp boundary of 40% impact theperformance of the proposed solution (Fig. 2.7d).472.4 Discussion and ConclusionThis chapter introduced a new framework based on DenseNet, CapsuleNet andRNN layers and showed that EF estimation from echo cines in A2C and A4Cecho is feasible. The presented framework can be used for direct cine-based,segmentation-free prediction of patients at a high risk of HFrEF. The proposedapproach enables the use of information lying in the dynamics of the heart in twoorthogonal and complementary views. It eliminates the reliance on accurate LVsegmentation and ED and ES frames detection by learning spatio-temporal fea-tures associated with EF. Binary classification of left ventricular systolic functionis a crucial step toward direct and automatic estimation of EF. A framework forthe classification of EF can assist front-line clinicians in detecting patients at ahigh risk of HFrEF and allow for expedited referral to the echocardiography labfor comprehensive imaging.2.4.1 Optimal Model PerformanceSpatio-temporal EncodingWe observed that DenseNet achieved higher accuracy compared to CapsuleNet.Given the performance of CapsuleNet on public data sets [194], this was inconsis-tent with our initial expectations. However, we suspect that this is due to the smallsize of our training set for learning such a complex yet subtle problem. DenseNetshave been proven effective for learning spatial features in relatively small trainingsets [94]. LSTM and GRU often performed equivalently, although the highest ac-curacy was obtained using GRU blocks. The results also consistently suggest thatbidirectional recurrent layers are equivalent to or better than unidirectional ones.For example, the optimal deep model, consisting of DenseNet + bidirectional GRU,achieved a success rate of 83.1% on the test set for detecting high-risk EF.Single-plane vs. Bi-plane ResultsOur results suggest that the A2C view alone is a less reliable view for EF estima-tion. A4C alone, on the other hand, appears to be a much more robust option forthe direct estimation of EF with the current framework. However, the most accu-48rate results are achieved by combining both apical views. This observation is alsoaligned with anecdotal clinical evidence, where A2C views are more difficult to ob-tain over A4C and are more likely to be foreshortened ;[166], hence EF estimationfrom A2C can be less reliable.2.4.2 Effect of Image Quality on PerformanceA key pattern recognized in visual inspection is the link between model perfor-mance and the quality of apical images. Misclassified images generally have un-clear LV boundaries, which cause a great deal of variance in the appearance of theheart and its motion. Also, despite automatic and manual view classification donefor this study, confusion between the four apical views (A2C, three-chamber, A4Cand five-chamber) appears to remain a challenge and a potential source of error(e.g., Fig. 2.7d). Thus, a bottom-up approach for improving EF accuracy can im-prove the quality of the input data. Abdi et al. proposed a deep learning solution forautomatic estimation of echo quality [2], which can be used to provide feedback toultrasound operators for improving the quality of data acquisition.2.4.3 Phase Detection and Cine SynchronizationImpacts of Synchronizing Input Videos on Model OptimizationOur experiments suggest cine loop synchronization can help achieve more effi-cient and effective spatio-temporal feature learning, which in turn, can help makethe proposed multi-channel framework more robust, flexible, and extendable. Thisis most likely because the inputs of the RNN blocks, i.e. the frame feature vectors,are denser and richer when constructed from two complementary views, allow-ing for more effective temporal learning. Nonetheless, phase dependence can beeliminated altogether by having two separate RNN streams, one per A2C and A4Cviews. This decouples the two views from one another, enabling the use of poten-tially informative cines in full. However, this architecture causes a sudden increasein the network size and is still less successful for EF estimation based on our ex-periments thus far.49ECG-based vs. Cine-based SynchronizationA resolvable limitation of the proposed solution is the dependence on ECG, whichis unavailable in point-of-care for phase detection and synchronization. Cine-basedsynchronization is preferred in the absence of ECG in point-of-care and emergencyrooms. Moreover, visual inspection of the results revealed a correlation betweenmisclassification and apparent improper synchronization (see e.g., Fig. 2.7b, whichshows asynchronous A2C and A4C views based on the valve state). We believeimproving the phase detection can contribute to achieving more accurate results.Supervised Learning for Phase DetectionIn order to detect the cardiac phase independent of ECG, cine-based cardiac phasedetection can be implemented into the network. Some works have previously ex-plored cine-based cardiac phase detection using CNN+RNN-based networks ex-plored [58, 252]. While promising for detecting ED and ES frames, such an ap-proach is not ideal for the direct estimation of cardiac measurements. This is be-cause any errors made in estimating the phase of each view are left uncorrectedand propagated through the next measurement steps. A possible solution has beenproposed by Dezaki et al. for A4C images, which can be similarly extended toA2C. This method can automatically identify ES and ED, which could be used toachieve potentially richer temporal sampling of systolic and diastolic phases.Self-supervised Learning for Direct SynchronizationDezaki et al. have proposed self-supervised learning for automatic synchronizationmethod may be used as a generic solution for multi-view or multi-modal videosynchronization with minor or non-trivial similarities. In this approach, the goalis to minimize the phase shift (temporal dissimilarities) between pairs of frames.This is possible since the heart motion appears periodic regardless of the 2D car-diac view. This periodicity is evident in the position and movement of the valves,rhythmic blood pool size changes, and overall cardiodynamics. A similar frame-work can be used to detect conditions that affect the heart\u2019s rhythm, such as atrialfibrillation [60, 161].502.4.4 Extending Binary to Multi-class EF ClassificationWhile a binary risk-based EF classification tool could assist with immediate deci-sion making in point-of-care, it suffers from a flaw: it imposes a sharp boundaryon the actual regression labels (EFSimpson\u2019s). This issue can be amended by addinga medium-risk class or more classes of EFEyeballed. In addition, we plan to includeexams from other ultrasound machines to obtain enough data for this multi-classclassification.While the presented binary accuracy may seem low, it is worth noting thatthe ground truth EF labels used in this study may suffer from some uncertaintiesand inaccuracies. In addition, as mentioned above, studies suggest EF estimationis highly user-dependant. This was evident in our database as well. The clinicalreport database analysis revealed significant discrepancies between the qualitativeand quantitative EF reported. We only an approximate 70.1% of the (EFSimpson\u2019s)and (EFEyeballed) labels agree. While these cases were excluded from the presentedstudy, we suspect that the accuracy of the clinical ground truth labels may be sim-ilarly compromised to some extent. With a correlation coefficient of only 0.71between EF labels derived from cine loops (directly estimated values) and thoseacquired via volumetric analysis (Simpson\u2019s method). Chapter 3 focuses on lever-aging these variabilities in EF clinical labels to obtain a more robust segmentation-free assessment of systolic function.We believe the proposed solution could achieve better performance shouldmore data be used. Given the limited number of samples with consistent EF labels,we plan to utilize a semi-supervised learning approach to first train an auto-encoderon a more extensive unlabelled set of A2C and A4C cine loops and subsequentlytrain a supervised network on the labelled EF data. Furthermore, incorporatingmore data in training may allow us to move toward a regression-based formula-tion, eliminating the issue of having a sharp threshold at EF = 40%.2.4.5 Investigating the Impacts of LV Localization Accuracy onFunction AssessmentDirect estimation of EF is a complex problem due to the noisiness of echo, complexanatomy and motion, especially in the presence of pathologies, and dependence51on image quality. Given that LV localization appears to be the critical step insome EF estimation approaches proposed for cardiac MR [110], another questionworth exploring is whether LV localization helps with EF accuracy in echo. Whilethe motion of the atria and right ventricle can contain subtle information aboutEF, excluding them decreases variance from the neighbouring chambers. Existingencoder-decoder segmentation networks can be modified and used to localize, trackand crop LV throughout the cine.52A2C A4CLarge purple regions:Large systolic volume changesLow-risk is detected correctly.High EFLarge volume changes(a)A2C A4CSystolic volume changes appear more visible in A4C.Systolic volume changes appear more subtle in A2C.High EFLarge volume change(b)A2C A4CSmall purple regionsSmall systolic volume changesHigh-risk is detected correctly.Small EFSmall volume changesLV is not fully captured.(c)A2C A4CSmall systolic volume changesMid-range is harder to classifyLow-risk is detected incorrectly.EF close to 40%Inaccurate A4C image plane(d)Figure 2.7: Example results of dual-channel segmentation-free classificationof high-risk vs. low-risk EF using DenseNets and bi-GRU.53Chapter 3Dual-view Joint Estimation of EFwith Uncertainty Modelling inEcho3.1 Introduction3.1.1 EF and Observation VariabilityThe most critical clinical measurement of an echo exam is EF, which evaluates thesystolic performance of the heart and the strength of the contractile function. Thestandard of care in 2D echo for calculating EF is the biplane Simpson\u2019s method ofdisks, which involves measuring the minimum, i.e., end-systolic (ESV), and max-imum, i.e., end-diastolic (EDV), volumes of the LV by estimating the LV surfaceThis chapter was adapted from i) D. Behnami, H. Y. A. Girgis, C. Luong, D. Hawley, R. Rohling,K. Gin, P. Abolmaesumi, and T. Tsang. Artificial intelligence for visual assessment of left ventricularsystolic function in patients with a wide range of ejection fraction. Journal of the American Societyof Echocardiography, 32:121\u2013122, 06 2019; and ii) D. Behnami, Z. Liao, H. Girgis, C. Luong,R. Rohling, K. Gin, T. Tsang, and P. Abolmaesumi. Dual-view joint estimation of left ventricularejection fraction with uncertainty modelling in echocardiograms. In International Conference onMedical Image Computing and Computer-Assisted Intervention, pages 696\u2013704. Springer, 2019.Uncertainty modelling presented in this chapter contributed to the patent P. Abolmaesumi,Z. Liao, T. Tang, and D. Behnami. Neural network image analysis. https:\/\/patents.google.com\/patent\/US20210365786A1, August 2021.54Key frame DetectionLV SegmentationVolume EstimationKey frame DetectionLV SegmentationVolume EstimationVisual AssessmentBiplane EF Calculation\ud835\udc6c\ud835\udc6d\ud835\udc7d\ud835\udc8a\ud835\udc94\ud835\udc96\ud835\udc82\ud835\udc8d\ud835\udc69\ud835\udc8a\ud835\udc91\ud835\udc8d\ud835\udc82\ud835\udc8f\ud835\udc86A4C ViewA2C ViewFigure 3.1: The clinical workflow for EF assessment. The workflow\u2019s darkand light purple paths illustrate the Simpson\u2019s and visual assessmentmethods, respectively. (Heart schematics: courtesy of 123 Sonography.)area in A2C an A4C. The accuracy of Simpson\u2019s method is highly dependent onaccurate\u2022 selection of end-diastolic (ED) and end-systolic (ES) frames; and\u2022 segmentation of the LV endocardium in both apical windows.The alternative technique for volumetric estimation of EF in clinics is the visualassessment of echo cine series. This approach is commonly used by experiencedechocardiographers, who can subjectively estimate EF accurately after years ofpractice. Visual assessment is also robust to segmentation and frame selectionerrors. Figure 3.1 demonstrates the clinical workflow for measuring EF using thesemethods. Nevertheless, both methods suffer from high inter- and intra-observervariability, making EF estimation challenging [70]. Factors contributing to suchvariability in EF labels include:\u2022 low inherent image quality in echo;\u2022 inaccurate segmentation or keyframe detection;\u2022 errors due to volume estimation from 2D images; and\u2022 and the fuzziness in human reasoning [242].Several works have previously attempted LV segmentation for EF assessment [101,130, 206, 245, 251]. Other related works include three-way classification of EF55\ud835\udc04\ud835\udc05\ud835\udc12\ud835\udc22\ud835\udc26\ud835\udc29\ud835\udc2c\ud835\udc28\ud835\udc27\u2032\ud835\udc2c\ud835\udc01\ud835\udc22\ud835\udc29\ud835\udc25\ud835\udc1a\ud835\udc27\ud835\udc1e\ud835\udc04\ud835\udc05 \ud835\udc04\ud835\udc32\ud835\udc1e\ud835\udc1b\ud835\udc1a\ud835\udc25\ud835\udc25\ud835\udc1e\ud835\udc1dFigure 3.2: Correlation of EF labels (EFSimpson\u2019sBiplane and EFEyeballed) in thecurrent echo database. Labels are very noisy, with a correlation coeffi-cient of only 0.71. Red data points indicate the samples for which thetwo EF labels do not agree, while the black crosses show samples whoselabels are in agreement.using residual networks [204] and segmentation-free estimation of EF in cardiacmagnetic resonance (CMR) images [81, 110, 215, 236]. In Chapter 2, we pro-posed a direct estimation of EF in echo cine series using 2D convolutional networks(CNNs) for frame-level feature extraction and recurrent neural networks (RNNs)for the temporal embedding of the videos. Upon further investigation of the labels,we found significant observer variability and disagreement in EF measurements(Fig. 3.2). The previous chapter addressed the issue of noisy labels by only includ-ing samples whose EF labels were in agreement.Figure 3.3: Gaussian PDF characterized by (\u00b5,\u03c3) to model the observervariability in the clinical EF labels.563.1.2 Chapter SummaryThis chapter builds on our previous work and proposes a highly accurate EF estima-tion method based on an extensive and diverse dataset to leverage the uncertaintiesand variabilities in the EF labels instead of eliminating them. The key contributionsare the following:\u2022 our proposed approach can estimate four EF labels simultaneously, includingsingle plane and biplane Simpson\u2019s measurements based on regression andvisual assessment based on classification;\u2022 we model the observer variability and uncertainties in clinical measurementsas a Gaussian distribution, whose parameters can be separately learned asindependent variables in neural networks. We empirically show that un-certainty modelling in clinical measurements can improve the robustness ofthe EF estimation framework. Moreover, providing such critical computer-generated clinical measurements in terms of distribution rather than a singlevalue may help improve clinicians\u2019 interpretability and adoption of machinelearning techniques.3.2 Materials and Method3.2.1 Uncertainty ModellingThe inherent low quality of clinical images and variability in the ground truth la-bels leads to inherent uncertainty, referred to as aleatoric uncertainty [114]. Inlearning-based approaches, this type of uncertainty cannot be resolved simply byacquiring more data and increasing the number of training samples. To model suchuncertainty in a supervised learning framework, we define the clinical measure-ment as a random sample drawn from a distribution of expert-annotated labels.Formally, let (xi,yi) denote the i-th datum, where xi represents the input data usedfor deriving the measurement yi. We can define yi as an observation made from thepossible distribution di; yi\u2190\u00d0 di.In this chapter, we characterize di as a normal distribution with the mean \u00b5(xi)and standard deviation function \u03c3(xi). We choose the Gaussian distribution be-57cause biplane EF labels are estimated as the average of A2C and A4C-based EFlabels; hence the observation distribution can be assumed to be symmetrical, withthe majority of observations clustered around the mean value \u00b5 . The parame-ters of the distribution di can be learned using a neural network with weightsdenoted by W\u00b5 and W\u03c3 . The observation distribution can then be expressed asdi \u223cN (\u00b5(xi;W\u00b5),\u03c32(xi;W\u03c3)). For simplicity, let W = W\u00b5 \u222aW\u03c3 . By modellingthe observation as a Gaussian distribution, we can now use the Gaussian probabil-ity density function (PDF) to compute the likelihood of the expert distribution di.The Gaussian PDF (Fig. 3.3 can hence be computed as:p(di\u2223xi,W) = 1\u221a2pi\u03c32(xi) exp(\u2212\u2223\u2223yi\u2212\u00b5(xi)\u2223\u222322\u03c32(xi) ); (3.1)which assumes a lower probability of occurring with deviation from the mean. Wecan estimate W by optimizing the negative log-likelihood (NLL):\u2212 ln(p(di\u2223xi,W)) = 12( \u2223\u2223yi\u2212\u00b5(xi)\u2223\u22232\u03c32(xi) + ln\u03c32(xi)). (3.2)For learning the discrete classification labels used in the categorical grading of clin-ical measurements, we can extend the uncertainty modelling framework by com-puting the class likelihood from the Gaussian cumulative density function (CDF).The Gaussian CDF for z becomes:\u03a6(z) = 12(1+erf( z\u2212\u00b5(xi)\u03c3(xi)\u221a2)), (3.3)where the error function is erf(z) = 2\u221api \u222b z0 exp(\u2212t2)dt. The likelihood that a pre-diction y\u02c6i is made from an expert-annotated class c j can thus be computed basedon the class interval defined by (lc j and uc j] over the regression space:p(c j\u2223xi,W) = p(y\u02c6i \u2208 (lc j ,uc j]\u2223xi,W)) =\u03a6(uc j)\u2212\u03a6(lc j). (3.4)Model parameters W can be learned by minimizing a classification loss, e.g., thecategorical cross-entropy (CCE) loss `CCE .583.2.2 EF DatasetThe dataset used for this study was obtained through Vancouver Coastal Health. Itconsists of echo cine series in various cardiac views (input data) and the diagnosticreports created by examining echocardiographers, which list several echo quan-tities, including four EF-related measurements. Following the notation from theprevious section, we define i-th datum (xi,yi), where xi represents the input echocine series required for EF assessment (A2C and A4C) for the i-th exam. Similarly,yi denotes the corresponding ground truth expert-annotated EF measurement, andyi \u2208 [0,1] since EF is expressed as a percentage:(xi,yi) = ({VA2Ci ,VA4Ci },{EFA2Ci,Simpson\u2019s,EFA4Ci,Simpson\u2019s,EFBiplanei,Simpson\u2019s,EFBiplanei,Visual)}.The superscript and subscript represent the measurement method and the corre-sponding required views for measurement, respectively. Hence, Vvi denotes thecine series captured in the view v = {A2C,A4C}, and consists of a stack of Fgray-scale echo frames Ivi, f of height and width H\u00d7 W for f = 1 \u2236 F . Biplane mea-surements correspond with both A2C and A4C cine series.The Simpson\u2019s labels were acquired by expert echocardiographers via segmen-tation of LV in the ED and ES frames. These are continuous (regression) labelswithin the range of 0%\u2212100%. EFBiplaneVisual labels are categorical and were visuallyestimated directly from the cine series of A2C and A4C views, without any LV de-lineation, by expert clinicians. Figure 3.4a shows the coefficients of determination(R2 scores) for the combinations of the four EF labels and highlights the clinicalaleatoric uncertainty in measuring EF in echo. A breakdown of the categoricalEF labels is provided in Fig 3.4b for the classification of EFBiplaneVisual . The interval(lcj ,ucj] is defined over the regression space for class c j=1\u22364. Throughout this chap-ter, the colours blue and red are used for inputs and outputs associated with A2Cand A4C, respectively.3.2.3 Neural Network ArchitectureA multi-task neural network was devised to simultaneously learn the four differ-ent clinical EF labels while incorporating the observer variability in the data. Anoverview of the proposed model is depicted in Fig. 3.5.59\ud835\udc452 \ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc342\ud835\udc36 \ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc344\ud835\udc36 \ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52\ud835\udc6c\ud835\udc6d\ud835\udc49\ud835\udc56\ud835\udc60\ud835\udc62\ud835\udc4e\ud835\udc59\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc342\ud835\udc36 1.00 0.58 0.88 0.79\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc344\ud835\udc36 0.58 1.00 0.88 0.80\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc520.88 0.88 1.00 0.90\ud835\udc6c\ud835\udc6d\ud835\udc49\ud835\udc56\ud835\udc60\ud835\udc62\ud835\udc4e\ud835\udc59\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52 0.79 0.80 0.90 1.00(a)\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52Class \ud835\udc84\ud835\udc8b\ud835\udc8d\ud835\udc84\ud835\udc8b \ud835\udc96\ud835\udc84\ud835\udc8b LV Function Diagnosis1 0.0 0.20 Severe dysfunction2 0.20 0.40 Moderate dysfunction3 0.40 0.55 Mild Dysfunction4 0.55 0.80 Normal Function(b)Figure 3.4: A closer look at the EF dataset and labels. The variability and in-herent aleatoric uncertainty in the four EF measurements is highlightedin (a) in terms of agreement between the four labels in terms of R2 scores(as low as 0.58 for EFA2CSimpson\u2019s and EFA4CSimpson\u2019s). A break-down of the cat-egorical EFBiplaneVisual labels is provided in (b).Spatio-temporal Feature Embedding (STFE)In order to encode the cine loops, we first extract spatio-temporal features by apply-ing 3D convolution (C3D) in the STFE block. C3D-based structures have provenpromising for video analysis tasks [221], and despite being computationally ex-pensive, are feasible for analyzing relatively short echo cine series, which capturea few heartbeats. In this approach, the input video is represented as a stack of2D video frames, creating a 3D tensor consisting of two spatial and one tempo-ral dimension; H\u00d7W\u00d7F. The STFE block contains five (3,3,3) C3D and (2,2,2)max-pooling layers.60ConcatenateFC (Sigmoid)FC (Sigmoid)FC SigmoidFC (Sigmoid)FC (Sigmoid)FC (Sigmoid)FC (Sigmoid)FC (Sigmoid)\ud835\udc6aCDF LikelihoodFC(ReLU)FC(ReLU) FC(ReLU)C3DMax-poolingFC(ReLU)FC(ReLU) FC(ReLU)\u00d75C3DMax-pooling\u00d75\ud835\udf48\ud835\udf41\ud835\udf48\ud835\udf41\ud835\udf48\ud835\udf41\ud835\udf48\ud835\udf41A2C CineA4C CinePrediction with Uncertainty ModellingDual-view Pseudo-Siamese Spatio-temporal Embedding InputsFigure 3.5: The proposed architecture for dual-view cine-based joint estima-tion of four EF labels with uncertainty modelling, which characterizeseach prediction as N (\u00b5,\u03c3).Pseudo-siamese Multi-tasking FrameworkThe network consists of two streams designated for the A2C and A4C cine series.A pseudo-Siamese structure is utilized in that the streams have a similar architec-ture, but the parameters are not coupled. The obtained spatio-temporal feature vec-tors are merged through a concatenation layer after the STFE block. The outputsare the four aforementioned EF labels. EFA2CSimpson\u2019s and EFA4CSimpson\u2019s are linked to theinput A2C and A4C cine series, respectively. The other two outputs EFBiplaneVisual andEFBiplaneSimpson\u2019s are linked to both A2C and A4C views as they involve biplane mea-surements. The model is trained by jointly minimizing the loss for the four EFlabels:`total = `regEFA2Ci,Simpson\u2019s+`regEFA4Ci,Simpson\u2019s+`regEFBiplanei,Simpson\u2019s+`CCEEFBiplanei,Visual. (3.5)3.2.4 Data PreparationWe obtained a retrospective dataset of size N = 2,181 patients consisting of clinicalecho cine series xi and the four corresponding ground truth expert-generated EFlabels (yi for i = 1 \u2236 N). The main selection criteria involved the existence of allfour relevant EF labels. The dataset is diverse and includes patients with a wide61range of EF (see Fig. 3.8) and echo series acquired using machines manufacturedby different vendors (mainly GE and Philips). No demographic restrictions wereused for selecting the dataset. A2C and A4C cine loops were extracted using anecho view classifier [225]. Figure 3.4a shows the correlation of the four EF labelsin terms of R2 scores to highlight the variability in these clinical measurements.3.2.5 TrainingFigure 3.6: Overall regression results on the test set (N=430) in terms of coef-ficient of determination (R2), MAE, and standard deviation \u03c3 on the re-gression labelsEFA2CSimpson\u2019s, EFA4CSimpson\u2019s and EFBiplaneSimpson\u2019s without and withobserver variability modelling.The network was trained end-to-end from scratch on an Nvidia Tesla GPU.Adaptive moment (Adam) optimization was used, with the learning rate of \u03b1 =1e\u22124, which was found experimentally. To account for the imbalanced distribu-tion of samples, for each sample, we assigned weights inversely proportional to thefrequency of the EFBiplaneSimpson\u2019s class to which they belonged. Optimization conver-gence was achieved with the uncertainty modelling turned off in our experiments.A similar set of hyperparameters were used with the uncertainty switched on there-after. All of the parameters were randomly initialized. In order to prevent modelover-fitting, heavy data augmentation was performed by applying random gammaintensity transformations, rotation, zoom and cropping on the fly during training.Similarly, the starting point of the cine series was selected randomly during trainingto ensure the invariance of the visual assessment model with respect to the cardiacphase. Weight decay regularization was used in training.623.3 Experiments and ResultsA randomly drawn 20%-portion of the dataset was set aside as test data to evaluatethe proposed model. Quantitative results obtained are listed in Fig. 3.7 in termsof R2 score and mean absolute error (MAE) for regression of the Simpson\u2019s labelswith and without uncertainty modelling. For the four-class classification, an ac-curacy of 91.4% was obtained. The confusion matrix obtained on the test set isshown in Fig. 3.8. Class confusion occurs mainly for adjacent classes. Figure 3.9demonstrates a sample set of A2C and A4C videos and the corresponding labelsand prediction.63(a) (b)(c) (d)(e) (f)Figure 3.7: Regression plots demonstrating the correlation of the joint EFmodel results (vertical) and the expert annotation EF labels (horizon-tal) on EFA2CSimpson\u2019s (blue), EFA4CSimpson\u2019s (red), and EFBiplaneSimpson\u2019s (purple). Ineach case, the point-estimate results are shown on the left (subfiguresa, c, e), while the results of the uncertainty modelling is shown on theright (subfigures b, d, f). Modelling the observer variability using theproposed method improves the model performance.64(a)Normal 0 1 13 119Mild 1 2 184 10Moderate 0 75 2 3Severe 16 2 2 0\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52Severe Moderate Mild Normal(b)Figure 3.8: Confusion matrices showing test results (N=430) for four-wayclassification of EFBiplaneVisual without (left) and with (right) observer vari-ability.LabelGround TruthModel Prediction\u00b5 \ud835\udf0e\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc342\ud835\udc3628% 31% 4%\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc344\ud835\udc3634% 37% 4%\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc5231% 30% 3%\ud835\udc6c\ud835\udc6d\ud835\udc49\ud835\udc56\ud835\udc60\ud835\udc62\ud835\udc4e\ud835\udc59\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52 Moderate ModerateLabelGround TruthModel Prediction\u00b5 \ud835\udf0e\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc342\ud835\udc3660% 50% 5%\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc5b\u2032\ud835\udc60\ud835\udc344\ud835\udc3659% 61% 4%\ud835\udc6c\ud835\udc6d\ud835\udc46\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc60\ud835\udc5c\ud835\udc8f\u2032\ud835\udc94\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc5260% 54% 5%\ud835\udc6c\ud835\udc6d\ud835\udc49\ud835\udc56\ud835\udc60\ud835\udc62\ud835\udc4e\ud835\udc59\ud835\udc35\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc5b\ud835\udc52 Normal NormalFigure 3.9: Results on two test samples: the network inputs cine loops(videos on the left), expert-annotated ground labels, and the network\u2019spredictions, expressed in terms of (\u00b5,\u03c3) in the tables. (To play thevideos, open the PDF in Adobe Acrobat, Internet Explorer or other PDFviewers, enable Flash Player, and click on the snapshots.)653.4 Discussion and ConclusionThis chapter introduced a dual-stream, multi-output network for joint estimation ofregression and classification of clinical EF labels in echo video data. We incorpo-rated the uncertainties in the individual measurements by modelling the EF mea-surements as a random variable drawn from a normal distribution, whose mean andstandard deviation describe the predicted clinical measurement, and the aleatoricuncertainties.3.4.1 Impacts of Uncertainty ModellingThe advantages of the proposed uncertainty modelling are two-fold. First, from atechnical standpoint, modelling the observation as a distribution N (\u00b5,\u03c3) by de-coupling \u00b5 and \u03c3 allows the model better to capture the inherent and inevitablevariability in the observation, leading to better convergence in training, and sub-sequently, better model performance. From a clinical perspective, providing theclinician with computer-generated EF measurements, expressed in terms of EF\u00b1\u03c3 ,is beneficial because it gives the user more context regarding the reliability of themodel\u2019s predictions and improves the interpretability of the machine learning so-lution, which can, in turn, increase the clinicians\u2019 confidence in integrating thetechnology in their workflow.3.4.2 Model Design and Performance in the Thesis and LiteratureContextsThis chapter demonstrated that robust visual segmentation-free volumetric assess-ment of EF is feasible with deep learning. Nevertheless, the quality of the echo im-ages still plays a crucial role in the reliability of the proposed solution. Future workinvolves incorporating quality scores [225] as inputs to the EF measurement frame-work. Our proposed solution achieves accuracy superior to automatic cine-basedliterature [25] (i.e. Chapter 2) and comparable to semi-automatic segmentation-based works [110, 204], with the key added advantage of estimating four EF param-eters simultaneously while explicitly modelling the label uncertainty. We observedthat the network performed better EF prediction in biplane compared to single-plane analysis. This is consistent with our previous findings in chapter 2 [25]66and the clinical literature [70]. We focused on C3D neural networks for spatio-temporal feature learning as they have proven more successful for video analysisthan architectures that relied on RNNs [25, 204, 221] for aggregating frame-levelfeatures and temporal embedding. We also found that these architectures were no-tably easier to train compared to RNN-based ones. Furthermore, features acquiredby purely convolutional layers can be visualized and interpreted more intuitivelythan networks containing recurrent layers. Therefore, the interpretability of unsu-pervised features computed by neural networks plays a crucial role in integratingautomated technologies. Future work may also include investigating other sym-metrical probability distributions, such as the student\u2019s t-distribution.3.4.3 Beyond Aleatoric Modelling for EFProbabilistic formulation of EF enables more robust EF estimation, which is es-sential for integrating automated EF assessment in clinical workflows. Since thepublication of the aleatoric uncertainty modelling in this Chapter [27], Ouyanget al. has utilized spatio-temporal convolutions for jointly learning continuous EFlabels and heart failure. Our group has since proposed a Bayesian deep neuralnetwork for video-based estimation of EF in echo for epistemic uncertainty mod-elling [113]. In this framework, model weights are characterized as random vari-ables belonging to Gaussian distributions with trainable parameters. This methodwas validated on the EchoNet Dynamic [172] public dataset, and superior perfor-mance was observed with respect to deterministic modelling. In the context ofEF estimation, probabilistic modelling has also been proposed via Bayesian LVsegmentation [105], and LV landmark detection [103], which aim to leverage theuncertainties in LV tracing and landmark prediction.67Chapter 4Machine Learning for LeftVentricular Wall Motion Analysisin Echo4.1 Introduction4.1.1 Clinical BackgroundCoronary Artery Disease (CAD)CAD is a leading cause of death and disability [40] and impacts over 2.4 millionCanadians. Undiagnosed, CAD can further damage myocardial muscles, increasethe myocardial infarction (heart attack) risks, and reduce full recovery odds; \u201dtimeis muscle\u201d [12]. Hence, early diagnosis of left ventricular CAD is essential toensure appropriate treatment, including revascularization, medication prescription,etc. Diagnosing acute coronary syndromes allows prompt referral for medical andThis chapter was adapted from i) D. Behnami, C. Luong, M. Jafari, N. Van Woudenberg,D. Hawley, R. Rohling, P. Abolmaesumi, and T. Tsang. Can AI-driven LV delineation withstandregional wall motion abnormalities in TTE echo. Journal of the American Society of Echocardiogra-phy (in press), June 2022; and ii) D. Behnami, C. Luong, M. Jafari, N. Van Woudenberg, R. Rohling,T. Tsang, and P. Abolmaesumi. Generalizability of disease-agnostic ML-based LV segmentation ona cohort with regional wall motion abnormality. Rebuttal Stage at MICCAI, May 2022.68procedural therapy. Currently, the standard of care for CAD diagnosis is catheteri-zation and angiography [170], a costly and invasive procedure.Regional Wall Motion Analysis in EchoEchocardiography (echo) is routinely used to non-invasively assess the heart\u2019s wallmotion and regional systolic function. Experienced echocardiographers can iden-tify myocardial wall motion abnormalities in patients with chest pain or heart fail-ure. The goal is to identify regional ischemia in the myocardium, caused by apotential blockage in the arteries, which fail to supply enough oxygenated blood toLV to pump out in systole. Regional wall motion abnormality (RWMA) is highlycorrelated with complications such as CAD, myocardial ischemia, and myocardialinfarction [117, 127, 138]. RWMA detection plays a significant role in clinicaldecision-making for emergency patients with chest pain and congestive heart fail-ure [138]. In addition, RWMA can help identify myocardial infarction or obstruc-tive CAD [125]. The areas of abnormal myocardial thickening and motion mayrepresent territories of compromised blood flow, indicating disease in:\u2022 the Right Coronary Artery (RCA) (highlighted in blue in Fig. 4.1);\u2022 the Left Anterior Descending Artery (LAD) (green in Fig. 4.1); and\u2022 the Left Circumflex Artery (LCx) (pink in Fig. 4.1).Despite being a cornerstone of echo interpretation, LV wall motion analysis re-mains one of the most challenging skills to acquire. RWMA detection requireshigh levels of imaging proficiency (at least Level II and often Level III echo train-ing) [170].1 is highly subjective and observer-dependent. Figure 4.2 depicts theframe sequence for tri-plane apical windows used for RWMA analysis. Even ex-perienced cardiologists are prone to missed or misdiagnoses of RWMA, while theagreement between less experienced cardiologists and expert sonographers is of-ten less than 50% [170]. The subjectivity of wall motion analysis is due to generalecho quality and noise-to-ratio challenges, compounded with the complexities andnuances of the myocardial motion for small segments of the LV wall.1According to the Canadian Society of Echocardiography standards, Level II includes 450 inter-pretations and Level III denotes over 1000 interpretations [34].69Figure 4.1: Tri-plane LV regional wall motion analysis and the 16-segmentLV model. Orthogonal cross-sectional apical views are shown in yellow(A2C), orange (A4C) and red (A3C). CAD regions RCA, LAD, and CXare highlighted in blue, green, and pink.Sixteen-segment Model for Wall Motion QuantificationLV wall motion analysis involves probing the myocardial wall for abnormal sys-tolic thickening in echo[129]. To detect RWMA, echo cines RWMA is often vi-sually inspected in 16 regions or segments of the LV [129] (16-segment a.k.a.bull\u2019s eye model (Fig. 4.1). According to the American Society of Echocardio-graphy (ASE) recommendations, wall motion analysis is done visually in a semi-quantitative fashion, where thickening of the myocardial wall is studied in 16 in-dividual segments throughout a cardiac cycle [127]. Each segment receives a re-70gional wall motion index (WMSI) that describes the myocardial thickening andmotion semi-quantitatively, as listed in Table 4.1. In 2D transthoracic echo (TTE),WMSI Diagnosis Description1 Normal \u226540% systolic increase in myocardial thickening2 Hypokinesis 10-40% systolic myocardial thickening3 Severe hypokinesisor akinesis<10% myocardial systolic myocardial thickening4 Dyskinesis No systolic myocardial thickening; paradoxicalsystolic motion; segment moves outwards, awayfrom the center of the LV5 Aneurysmal Diastolic deformation; segment pouches out inboth systole and diastoleTable 4.1: Regional Wall Motion Index (WMSI) and measurement crite-ria [127].wall motion analysis is done based on a set of three complementary views:\u2022 three orthogonal apical windows:\u2013 apical two-chamber (A2C) (highlighted yellow in Fig. 4.1));\u2013 apical three-chamber (A3C) (red) or parasternal long-axis (PLAX); and\u2013 apical four-chamber (A4C) (orange); or\u2022 three parallel parasternal short-axis (PSAX) views:\u2013 PSAX at the LV apex level (PSAX-A),\u2013 PSAX at the mitral valve level (PSAX-M),\u2013 PSAX at the papillary muscle level (PSAX-PM).Apical windows are more commonly used for wall motion analysis, as PSAX viewsare harder to capture and yield lower-quality images.4.1.2 Related WorksSupervised machine learning (ML) has shown great promise in assisting with car-diac health assessment in echocardiographic (echo) videos. Many groups have71contributed to automating echo analysis and interpretation tasks thanks to the ubiq-uity of neural networks. However, these models often involve large data-hungryconvolutional or recurrent architectures models are trained with clinical data tocapture and map spatio-temporal embeddings in echo videos to predict output la-bels [25, 27, 28, 123, 172].ML for Evaluating Systolic FunctionOur group previously showed the feasibility of using ML-based frameworks toestimate left ventricular ejection fraction (EF) from echo cine loops in A2C andA4C videos [25, 27, 28, 101\u2013103, 105, 113]. Most of the proposed approachesfor EF estimation involve localization, LV segmentation, or detecting landmarks inthe end-diastolic (ED) and through end-systolic (ES) frames [103, 139]. In con-trast, others attempt to eliminate the segmentation steps to directly assess the car-diac kinetics [25, 27, 28, 113, 124, 172]. While these methods demonstrated theefficacy of neural networks for echo video analysis and cardiac anatomy delin-eation, they rely on the acquisition of large numbers of ground-truth segmentationmasks in a noisy modality. Much of the emerging literature in this area involvesthe assessment of the systolic cardiac function, focusing on innovative techniquesfor more accurate segmentation of the LV walls and landmarks [103, 139], com-pensation for the limited and noisy training data [102, 137], observer variabilitymodelling [27, 105, 113], imposing temporal consistencies, etc.ML for CAD and Wall Motion AbnormalitySeveral works have emerged in the past few years that attempt WMA or CADdetection in echo. Omar et al. proposed a 16-segment WMSI quantification us-ing principal strain analysis for myocardial wall motion assessment in stress echo.Leclerc et al. proposed an encoder-decoder network to perform multi-chamber seg-mentation of the endocardium and pericardium. Kusunose et al. used two parallelnetworks on the PSAX views and three frames (end-diastolic (ED), mid-systole,and end-systolic (ES)), where one network classifies control samples from asyn-ergic (abnormal) LVs, while the other indicates existence of CAD in LAD, LCX,or RCA. Huang et al. attempted RWMA detection using three-dimensional 3D72U-net with segment-wise wall annotations for 2,736 still images. They empha-sized the importance of sufficient image quality and tracing for reliable RWMAdetection [95].Hamila et al. used 2D convolutional neural networks (CNNs) tosegment LV in A4C and a 3D CNN for binary classification of myocardial infarc-tion. Degerli et al. proposed neural networks to segment individual wall segmentsand myocardial infarction labels. Using a VGG-16 and LSTM-based architecture,Muraki et al. attempted direct spatio-temporal assessment with the function andoutcome labels (normal myocardium or acute myocardial infarction).Beyond echo, researchers have focused on multi-chamber segmentation andatro-ventricular function assessment statistical shape model on MR [211], multi-modal imaging of ischemic heart disease [145], volumetric rendered LV based on4D CT [44]. Raghavendra et al. used a wavelet-based method to decompose theimage into frequency sub-bands, from which entropy features were extracted asRWMA indicators.4.1.3 Challenges of Regional Wall Motion AnalysisIn addition to challenges inherent to ultrasound imaging, wall motion analysis isparticularly difficult to automate:\u2022 Multi-view Analysis: Wall motion analysis requires three input views, allof which are prone to view quality concerns, as discussed earlier for EF(Chapters 2 and 3).\u2022 Fine-grained and Noisy Labels: As LV is divided into 16 segments forwall motion analysis, motion analysis is done on very small sections of echoimages (up to six segments in a given view). This subsequently leads to highobserver variability, especially as the segment boundaries are subjectivelyeyeballed.\u2022 Weak Regional Labels: The report of wall motion analysis for a given sam-ple contains an array of WMSI codes without the spatial mapping that de-scribes the region of the cine loop from which the score was obtained.734.1.4 Chapter OverviewThis chapter focuses on investigating the utility of the developed ML methodologyon a wall motion abnormality (WMA) patient cohort.ML-driven Identification of Optimal Echo Views for Determining SystolicDysfunctionWe retrieve clinical echo data from a patient cohort with wall motion abnormalitiesfrom Vancouver Coastal Health (VCH), and upon de-identification, we investigatethe cardiac view qualities to identify the optimal views for wall motion analy-sis. View classification and quality grading are done using a previously-developedspatio-temporal model by our group [136]. We report our findings and demonstratethat the apical trio is better in quality than PSAX views, although A2C yields lowquality. This is a novel application of ML-driven quality assessment to automat-ically analyze large datasets and determine the diagnostically-reliable views for agiven condition or disease.Experiments with Direct Tri-plane Wall Motion AnalysisWe investigate the feasibility of global and regional systolic function assessmentwithout intermediate segmentation using a supervised multi-task learning frame-work. We follow the multi-stream and multi-task spatio-temporal architecture ofChapters 2 and 3, with A2C, A4C, and A3C views. The aim is to train the model topredict the presence of RWMA, with additional tasks of global wall motion abnor-mality (GWMA, where all the segments WMSI> 1) and EF estimation. We reportthe results of the said unseen test and empirically show that, unlike the global func-tion, RWMA cannot be reliably detected directly with our available tools and datasamples.LV Segmentation Regional Performance QuantificationWe investigate the robustness and generalizability of spatio-temporal LV segmen-tation models trained on an average population (< 20% wall motion abnormalities)when applied to a cohort of patients diagnosed with RWMA. RWMA affects themotion signatures captured by neural networks, commonly used for LV analysis74and quantification in the recent literature. We focus on the LV segmentation modelsfor A2C and A4C views developed by our group [102] trained in a semi-supervisedframework. We analyze the model performance both locally and globally:\u2022 from a purely vision-based standpoint;\u2022 diagnostic function-related; and\u2022 demographic and fairness aspects.Investigating fairness aspects of the model performance is essential as the preva-lence of CAD is nearly two-fold higher in males and increases the likelihood ofpremature death by three to six times [185]. We break down the model\u2019s per-formance based on the region of LV and cross-reference the results with clinicallabels describing the global and regional LV function. The following contributionsare made:\u2022 We propose a framework for quantifying the regional segmentation errorswith weak labels for 12 LV segments visible in A2C and A4C views, withoutthe need for additional fine-grained expert annotations;\u2022 We define a wall distance metric to measure regional wall localization errorsin the presence of pathology;\u2022 We explore the function and disease-related aspects of the ML segmentationmodels concerning EF, GWMA, and RWMA; and\u2022 We explore the fairness aspects of the ML segmentation models with respectto available demographic information, i.e. age, sex, and body mass index(BMI).75Figure 4.2: Synchronized echo frame sequences the apical planes (left toright: A2C, A4C, and A3C) for one cycle for a patient with RWMAdiagnosis in segments 4, 10, and 14 (refer to Fig. 4.1), emphasizinghigh data noise, which makes both clinical and automated wall motionanalysis extremely difficult.764.2 Material and Methods4.2.1 RWMA Patient CohortClinical DatabaseThe data used in this study originate from the VCH clinical echo database, reportedby experienced echocardiographers with level III training (six interpreters). All im-ages were digitally recorded and interpreted on the syngo Dynamics VA20 platform(Siemens Healthcare Limited, Oakville, Ontario). These standard segmental labelswere documented in the echocardiography database as part of the clinical report.Each of the 16 LV wall segments was coded from 1-to 5 based on clinical crite-ria [127](Table 4.1). Experienced cardiologists interpreted the echo studies withlevel III echo training (6 separate interpreters). The reference standard is the la-belling for RWMAs from this expert group of echocardiologists. Each of the 16left ventricular wall segments (Fig. 4.1) was categorically coded with WMSI as 1-5based on the ASE\/EACVI ([127]).Studies were acquired by certified sonographersand interpreted following the American Society of Echocardiography standards(ASE) [127]. All analyzed professional sonographers performed echo studies onfull-functionality echo platforms (Philips iE33 and GE Vivid I) with commerciallyavailable phased-array probes.Data Selection CriteriaThe cohort used in the analysis was a random sample of echo exams with reportedabnormal LV systolic function completed at two academic referral centres fromSeptember 25, 2017, to January 15, 2019. All samples include EF and RWMA la-bels with at least one dysfunctional segment, i.e. with at least one LV wall segmentcode WMSI>1.The dataset additionally contains patient information at the study-level age, sex,weight and height, which can shed light on the fairness aspects of the model per-formance. We combine the weight and height information to get the BMI score:BMI = weight(kg)[height(m)]2 . (4.1)77Repeat studies from the same individual were not excluded, i.e. studies from pa-tients with multiple visits were included, although grouped strictly in either trainingor test sets. The WMA cohort above was downloaded and anonymized from the(a) EF (b) GWMAFigure 4.3: Distribution of global systolic function labels in the WMA cohort(n=2,910).regional CPACS database. The dataset represents a range of patient demograph-ics, image quality, and clinical characteristics, excluding patients without complexcongenital heart disease (as defined by the American Heart Association guide-lines [146]), as this condition affects cardiac motion from a patient\u2019s birth andhence falls outside the scope of CAD.4.2.2 View Classification and Quality QuantificationTo analyze the impacts of RWMA and potential CAD on ML-generated predictionaccuracy, we start by using the view and quality classification model developed byour group [136, 225].Network ArchitectureThe model architecture is shown in Fig. 4.4. This joint model is based on the pre-viously validated view classifier and quality regression model [136]. The two tasksare unified inside a single TensorFlow 2 model trained on 2D TTE cine loops. Thismodel extracts spatio-temporal features and simultaneously detects the view andquantifies the image quality associated with the view for a sequence of frames ina cine loop. The spatial embedding is done with a Densely-connect Neural Net-78Figure 4.4: Network architecture for simultaneous view classification andquality estimation in echo as proposed by [136]. A denseNet block [94]extracts frame-wise spatial features, which are then aggregated throughan LSTM for temporal embedding. Finally, the spatio-temporal featurevectors are mapped to one standard echo view and a continuous qualityscore.work (denseNet) module with batch normalization [94, 98], which extracts imagefeatures from individual frames in the input video. These features are temporallyaggregated using a Long Short Term Memory (LSTM) module [92] module. TheLSTM module receives the spatial frame image features produced by the DenseNetlayers and generates a corresponding set of temporally-dependent features. The re-sulting spatio-temporal embeddings are passed through two fully-connected layersto map the inputs to predicted outputs:\u2022 a classification branch and module to predict the echocardiographic view;\u2022 a regression branch and module to predict the corresponding view qualityfor every video loop frame.In both cases, the consensus is obtained based on the frame-wise quantities by find-ing the model for the predicted view and the mean value for the predicted quality.79Quality Class In-dexAssigned QualityScoreAmount of Clearly Visible Ex-pected Blood-tissue Interface1 100% 75-100%2 75% 51\u201375%3 50% 26\u201350%4 25% 0-25%Model TrainingThe echo data used to train and validate the model were independent of those ex-amined in this study. The majority of the studies were performed by certified sono-graphers, with a small proportion generated by cardiology or sonography trainees.The model was trained on 16,772 randomly selected 2D echo videos (3,157 patientstudies) downloaded and anonymized from the regional CPACS. Joint optimizationof the view and the quality tasks was done with a loss function of a combination ofa cross-entropy (for the view classification task) and a mean absolute difference er-ror (for the quality estimation task) with the stochastic gradient descent algorithm.The criteria for ground-truth labelling were similar to other published methods forimage quality scoring [122, 149, 154, 154, 219]. Echo study quality was manu-ally graded for each clip by a level III echocardiographer using the following scalebased on view-specific expected structures [2, 136, 225, 234]: Based on the previ-ous evaluations, the training accuracy for view classification was 92.35%, with atest accuracy of 86.21% [136, 234]. The absolute error for the image quality esti-mation for each view was 0.12 \u00b1 0.09, using a validation set of 3,078 cine loops.Model Deployment for Processing the WMA CohortThe anonymized raw cine loops were fed through the ML-based view classifiers,and the three critical views were identified and retrieved. All studies containingat least one of the three views were included. Studies with any of the 16 seg-ments scored as 0, i.e. unable to grade due to quality or technical factors, wereexcluded. A2C and A4C views were identified and retrieved using an in-houseML-based view classifier model [136]. The TensorFlow ML model was deployedon a GeForce GTX 16 Series Graphics Card by NVIDIA. The cine loops weretemporally sampled to include 30 frames, and the images were cropped around the80ultrasound beam. View classes and quality scores were generated for a total of84,741 videos originating from 1,145 echo exams, with reported LV dysfunction(EF < 52% or the presence of RWMA). The estimated total analysis time was 47hours, i.e. 2 seconds on average for each clip). The model identified 22,768 videoloops of the five views of interest.4.2.3 Modelling Myocardial Wall Motion(a)(b)Figure 4.5: Systolic myocardial thickening can be modelled as a set of tuples(ximyo,yimyo,\u03b8 i), given the endocardial Wendo and pericardial wall coordi-nates Wperi (see Equations 4.2 and 4.3). W \u2032peri is the projection of Wendoon the Wperi curve with one-to-one mapping.81Consider the LV myocardium, which consists of the endocardial wall Wendo (pix-els on the inner LV boundary) and the pericardial wall Wperi (outer boundary).Each point on the Wendo and Wperi walls can be represented with the correspond-ing x and y coordinates. The myocardium muscle hence be described as a set of(ximyo,yimyo,\u03b8 i) tuples where:ximyo = 12(xiendo+x\u2032iperi); yimyo = 12(yiendo+y\u2032iperi); (4.2)where (x\u2032iperi,y\u2032iperi) are the coordinates of the pericardial boundaries (W \u2032peri), withone-to-one mapping to Wendo. As the pericardium curve encapsulates the endo-cardium, NWperi > NWendo . The thickness of the myocardial wall thickness at the i-thpoint \u03b8 i can be calculated as:\u03b8 imyo =\u221a(x\u2032iperi\u2212xiendo)2+(y\u2032iperi\u2212yiendo)2 (4.3)If the function \u0398(.) describes the thickness of the myocardial wall W i,\u03c6myo at i-thlocation, systolic thickening can be defined as:\u03b8 i\u2236 = \u0398(W i,EDmyo )\u2212\u0398(W i,ESmyo )\u0398(W i,EDmyo ) (4.4)The myocardial thickening for the k-th wall segment for the values of i on the k-thsegment becomes:\u03b8 i\u2236 \u2aa7k = \u0398i(W EDmyo,\u2aa7k)\u2212\u0398i(W ESmyo,\u2aa7k)\u0398i(W EDmyo,\u2aa7k) (4.5)Averaging over all the points on a given segment, a mean systolic wall thickeningratio can be obtained for segment k:\u03b8\u2236\u2aa7k = 1Nk \u2211i\u2208\u2aa7k\u03b8 i\u2236 \u2aa7k (4.6)Based on Equation 4.6, an RWMA output label yRWM\u2aa7k can then be defined for seg-ment k; as per criteria in Table 4.1, i.e.:1. normokinetic yRWM\u2aa7k = 1 if \u03b8\u2236\u2aa7k > 40%;822. hypokinetic yRWM\u2aa7k = 2 if 10% < \u03b8\u2236\u2aa7k < 40%;3. dyskinetic yRWM\u2aa7k = 2 if \u03b8\u2236\u2aa7k < 10%;We exclude the aneurysmal class (class 4 in Fig. 4.12), as the systolic wall thicken-ing ratio cannot directly describe the wall motion, and the condition is overall lessprevalent. Aneurysmal wall motion refers to the abnormal bulging of the segmentin both systole and diastole.4.2.4 RWMA Detection via Multi-task Visual AssessmentUsing a pseudo-siamese multi-stream and multi-task network similar to Chapter 3,we experiment with a framework to learn the regional and global systolic functionconcurrently. Given input cines Xv (v \u2208 {A2C,A3C,A4C}, in visual assessment, theaim is to train the model to learn the mapping directly.(XA2C,XA3C,XA4C)\u00d0\u2192 (yEF ,yGWM,yRWMk=1\u223616) (4.7)We additionally derive global wall motion abnormality (GWMA) as a binary value,where all segments are coded as at least 2:yGWM = [\u221116k=1 yRWMk2\u00d716 ] (4.8)The clinical labels investigated are EF (possible range 0%\u2212 100%, actual range10%\u221285%),Tri-plane Psuedo-siamese NetworkAs shown in Fig. 4.6, synchronized input videos in the A2C, A3C, and A4C arefed into spatio-temporal feature extraction blocks in three streams. Neural networkvariations include:\u2022 2D feature extraction (e.g. a series of denseNets [94] or resNets [216],or more compact networks such MobileNetV2 [198]) combined with re-current layers for temporal feature aggregation (Long short-term memory(LSTM [92]) or Gated Recurrent Units (GRUs [57]).83Figure 4.6: Network architecture in experiments with apical tri-plane directsystolic wall motion analysis. Inputs are synchronized cines of A2C,A3C, and A4C views, and outputs are regional (RWM) and global (EF,GWM) labels.\u2022 3D convolutional blocks (C3D) [220] and (2+1)D [221] for spatio-temporalembedding.View-specific features are then frame-wise aggregated across the views to create afourth joint stream. The joint stream is connected to an EF (global metric) outputbranch and a GWMA classification branch. An output branch is considered foreach visible segment. The RWMA outputs are connected to the corresponding84echo views. The tri-plane embeddings are then passed through fully-connectedlayers to the outputs yEF ,yGWM,yRWM.Objective Function for Systolic AssessmentFor continuous labels, a mean-square error was used as a regression loss `reg. Bi-nary labels describing the global wall motion yGWM are learned via a binary cross-entropy (BCE) loss (`GWMBCE ). Finally, for each view, the regional labels yRWM is avector with a length of 6 (number of visible LV segments as shown in Fig. 4).`total = `EFreg +`GWMBCE +`RWMCCE ; (4.9)where`RWMCCE = 16\u2211k=1yRWMk . (4.10)The backbone architecture used for these experiments is a variation of exper-iments explored by Asgharzadeh, with the inclusion of an EF output branch. EFestimation is an auxiliary task whose feasibility was proven in previous chapters toregularize network optimization further. With respect to [16], in addition to archi-tecture differences, we simultaneously utilized a multi-term loss to learn the EF,GWM, and RWMA labels.4.2.5 Wall Motion Analysis with Weak LabelsWe evaluate the robustness and reliability of segmentation-based approaches tobetter understand the data and tasks at hand. In a segmentation-based wall mo-tion analysis, with both W vendo and Wvperi walls available in each view, Wmyo can becomputed as in Fig. 4.2.3. Hence the aim is to predict the mappings:(XA2C,XA3C,XA4C)\u2192 (W A2Cmyo ,W A3Cmyo ,W A4Cmyo )\u2192 \u03b8\u2236\u2aa7k;k = 1 \u2236 16 (4.11)Nonetheless, our echo database does not contain the location of each segment k onthe overall LV nor W vmyo or Wvperi clinical tracings. Hence ground-truth annotationsfor W vmyo across different segments is not available. To take advantage of available85annotations, we instead focus on A2C and A4C tracings available for ED and ESframes, calculated as part of the Biplane Simpson\u2019s workflow for EF estimation.We thus reduce the problem to the following:(XA2C,XA4C)\u2192 (W A2Cendo,W A4Cendo\u2223yRWMk ; 16\u2211k=1yRWMk > 16) (4.12)and attempt to evaluate the prediction accuracy of Wendo for A2C and A4C views,where LV endocardium segmentation has been developed by Jafari et al.. Fig-ure 4.7 illustrates the study workflow with respect to [137].Figure 4.7: Study overview for investigating the generalizability of disease-agnostic endocardial segmentation models for A2C and A4C (green andred) on a clinical WMA cohort (blue box). The models were previouslytrained [102] on a dataset with an estimated 10% WMA.86Neural Network for LV Endocardial SegmentationWe use in-house LV segmentation models presented in [102, 137] to obtain theLV endocardial borders for A2C and A4C cine loops. The segmentation backboneis a lightweight 2D U-net-based [193] architecture with 123k parameters. Thesampling blocks in the encoding branch contain 3\u00d73 2D convolutional layers, 3\u00d73max-pooling, batch normalization and ReLU activation. Input images are gray-scale 128\u00d7170 frames, with preserved aspect ratios cropped around the ultrasoundbeam. The final layer has sigmoid activation to generate pixel-level binary LVlikelihood maps for the input frames.The segmentation models were trained in an adversarial setting to imitate clini-cal annotations further. In addition to the encoder-decoder model described above,a simple convolutional network discriminates between ground-truth echo annota-tions. As a result, model-generated predictions produce segmentations that resem-ble expert annotations, provided for the ED, ES, and one additional random frame.Both A2C and A4C models are trained on a control dataset extracted from our lo-cal hospital\u2019s database with no criteria (approximately 80% with no wall motionabnormality).Localizing Regional SegmentsIn the absence of segment-wise clinical annotations, we mathematically obtain theapproximate coordinates of the visible segments to evaluate the LV model\u2019s perfor-mance on samples with RWMA. Consider the binary image I containing a bullet-shaped LV mask Mendo, endocardial wall Wendo (or M and W for simplicity), withcoordinates \u2126W \u2286 \u2126M. The top two eigenvectors of the elongated object can becalculated to obtain the base-apex major axis\u2190\u00d0\u2192lma jor and the minor axes intersectingleft and right walls. We divide the apex-base axis into three segments parallel tothe minor axis. C 13and C 23are calculated from the deviation \u2206x =C 13.HL.Sin\u03b2\u0302 and\u2206y=C 13.HL.Cos\u03b2\u0302 . We can hence compute the segments\u2019 mask and wall coordinates\u2126M,\u03c6\u2aa7k and \u2126W,\u03c6\u2aa7k for the k-th segment A\u2212F on frame \u03c6 . The correspondence be-tween segments A\u2212F and the clinical LV segment codes in the 16-segment model(shown in Fig. 4.1 and 4.8) are listed in Table 4.2 below for the two relevant apicalviews.87Figure 4.8: Schematics of dividing the endocardial wall in A2C and A4Cviews into six segments for regional assessment of segmentation per-formance. The LV mask M is shown in white, with the major and minoraxes red and blue, respectively. Segments A\u2212F are determined for theground truth and predicted LV masks. CO is the centroid, and \u03b2\u0302 denotesthe orientation of the bullet-shaped LV.Segment Let-ter(Fig. 4.8)16-segment Model Segment Code (Fig. 4.1)View A2C A4CA 4 (Basal inferior) 3 (Basal inferoseptal)B 10 (Mid inferior) 9 (Mid inferoseptal)C 15 (Apical inferior) 14 (Apical septal)D 13 (Apical anterior) 16 (Apical lateral)E 7 (Mid anterior) 12 (Mid anterolateral)F 1 (Basal anterior) 6 (Basal anterolateral)Table 4.2: Correspondences of alphabetical segment codes (used in this chap-ter for error quantification in A2C and A4C) and the conventional 16-segment numerical coding used by echocardiographers.For segment wall coordinates, we find the intersection of the selected segmentcoordinates with the set of LV boundary pixels \u2126W . For the k-th segment k \u2208{A,B,C,D,E,F}, the LV segment wall coordinates are the set \u2126W\u2aa7k where:\u2126W\u2aa7k = \u2126W \u2229 \u2126M\u2aa7k. (4.13)88Criteria for Evaluating the Model Robustness and ReliabilityRegional Performance MetricsThe location of LV wall segments is estimated as the intersection of the segments\u2019set of mask coordinates and the boundaries (edge) of the original LV masks ofsize N\u03c6k and N\u0302\u03c6k for ground truth and model predictions, respectively. We define aregional wall distance error for the k-th segment is defined asdW,\u03c6\u2aa7k = 1N\u03c6k \u22c5N\u03c6k\u2211i=1 \u2225\u2126W,\u03c6\u2aa7k,i , \u2126\u0302\u2032W,\u03c6\u2aa7k,i \u2225; (4.14)where \u2126\u0302\u2032W,\u03c6\u2aa7k,i of length N\u03c6k for frame \u03c6 . We establish correspondences betweenthe ground truth and the model-predicted LV wall point-sets \u2126\u0302W,\u03c6 using the near-est neighbour search at test time. For I with dimensions w\u00d7h pixels, mean walldistance error can have a range up to\u221aw2+h2. Additionally, S\u00f8rensen\u2013Dice iscalculated as:D\u03c6\u2aa7k = \u2126M,\u03c6\u2aa7k \u2229 \u2126\u0302M,\u03c6\u2aa7k\u2126M,\u03c6\u2aa7k \u222a \u2126\u0302M,\u03c6\u2aa7k , (4.15)for ED and ES frames, where ground-truth tracings are available. Knowing thebreakdown of the coordinates along both axes, we can compute the coordinates ofthe segments A,B,C,D,E,F1 labelled from the bottom-left segment going clock-wise as in Fig. 4.8. As clinical annotation is extremely expensive to acquire forthe myocardial wall, we focus on the endocardial wall, for which the clinical LVtracings are available.Global Segmentation PerformanceWe additionally measure the location of the mask centroid CO, orientation \u03b2\u0302 , themajor axis length WapexWbase (a.k.a LV internal dimension LVId), the minor axissize (LV width), and LV surface area was calculated. For each parameter, we reportthe percent error for the parameters y calculated from the ground-truth mask \u2126M,\u03c6\u2aa7k1We use alphabetical (instead of numerical) references to segments to prevent confusion with thenumerical coding in the LV 16-segment model.89y and those calculated from the predicted mask \u2126\u0302M,\u03c6\u2aa7k denoted as y\u02c6 as:y% = y\u2212 y\u02c6y \u00d7100%. (4.16)Reliability for Assessment of the Systolic FunctionBeyond the purely-vision-related metrics above, we calculate the stroke volume(SV) and ejection fraction (EF) to evaluate the effectiveness of the models for esti-mating systolic function metrics. Using the end-diastolic and end-systolic volumes(V ED and V ES):SV =V ED\u2212V ES; and (4.17)EF = V ED\u2212V ESV ED\u00d7100. (4.18)We use the area-length method for volume estimation. For \u03c6 \u2208 {ED,ES}:V \u03c6 = 0.85\u00d7 (A\u03c6)2L\u03c6; (4.19)where the length L\u03c6 describes the length of WapexWbase in phase \u03c6 . LV lengthL\u03c6 , LV area A\u03c6 , and LV volume are calculated to report segment-wise errors. Wecalculate and report errors in the pixel units as the ultrasound machine\u2019s depthparameter is unavailable. We also report the error in EF as the difference betweenthe ground truth and EF derived from LV segmentation:\u2206EF = EF \u2212 E\u0302F (4.20)4.3 Experiments and Results4.3.1 Optimal Quality Views for Regional Wall Motion AnalysisRepresentative images and respective ML model image quality scores are shownin Fig. 4.9.90Figure 4.9: Representative frames cine loops in five views of interest graded by the view and quality ML model. ML-predicted quality scores correspond well with the visibility and clarity of the cardiac anatomy.91(a) (b)Figure 4.10: ML model-predicted mean (a) and maximum (b) image qualityscore percentage by the view. Boxplots show the distribution predictedmean and maximum image quality score for each of the five views ofinterest. The A4C view had significantly higher scores as comparedto all other views (p < 0.001). The A2C view had lower scores thanall other views, except the PSAX-M\/PM view (p < 0.001, A2C viewvs. PSAX-M\/PM view p=1). The A3C and PSAX-M\/PM views hadsimilar maximum image quality scores (p=1).Views of InterestA total of five views were considered for each study where available. For a givenstudy, quality scores were obtained for samples of view of interest (Qvjv), wherev \u2208 {A2C,A3C,A4C,PLAX ,PSAX \u2212M\/PM}and Nvj denotes the number of available cine loops for the v view; j \u2265 0, as wefocus on the views clinically used for RWMA detection. We combined PSAX-Mand PSAX-PM to form one PSAX-M\/PM class for our analysis. This decisionwas rooted in the heavy overlap and similarities in appearance for slightly off-axisimages often present in clinical studies.92Quality MetricsWith Nvj \u2265 1, the mean and maximum quality quality scores are calculated for eachview:Q\u0302v = 1NvjNvj\u2211j=1Q\u0302vi (4.21)Q\u0302vmax =max(Q\u0302vj) (4.22)Comparative AnalysisView (v) Q\u0302v (%) Q\u0302vmax (%)A2C 53.1\u00b115.8 60.4\u00b115.4A3C 57.9\u00b115.9 64.3\u00b115.7A4C 61.3\u00b114.9 70.6\u00b113.9PLAX 56.1\u00b113.2 65.3\u00b113.7PSAX-M\/PM 50.7\u00b115.0 60.9\u00b116.8Table 4.3: ML-predicted mean and maximum quality scores were analyzedfor the RWMA-relevant views for n=1,145 multi-view echo exams.The mean and maximum quality scores for the remaining views are summarized inTable 4.3 and Fig. 4.10. Exams with missing views were excluded from maximumor mean calculation. Wilcoxon signed-rank test was performed for pair-wise com-parisons of ML-derived image quality scores of the five views A2C, A3C, A4C,PLAX, and PSAX-M\/PM. All analyses were performed overall and separated intomales and females.The A4C view had significantly higher scores than all other views (p < 0.001),whereas the A2C view had lower scores compared to all other views (p < 0.001).Overall, the results are similar, with the best mean score with the A4C view demon-strating the highest scores at 61.3%\u00b114.9% (p<0.001 compared to all other views).The poorest mean quality score was the PSAX-M\/PM view at 50.7%\u00b115% (p <0.0001 compared to all other views). When accounting for sex, the highest andpoorest quality views remained the A4C and PSAX-M\/PM, respectively. However,the PSAX-M\/PM view in females was numerically worse but statistically similarto that of the A2C view.93The highest maximum ML model-derived image quality score among the fiveviews evaluated was the A4C view. The top quality score for A4C images acrosssamples was 70.6%\u00b113.9% (p < 0.001 compared to all other views). Conversely,the images with the poorest maximum quality score were the A2C chamber viewat 60.4%\u00b115.4% (p < 0.001 compared to the PLAX, A3C, A4C views) and thePSAX-M\/PM view at 62.2%\u00b116.6% (p < 0.001 compared to the PLAX, A3C, andA4C view). These findings were consistent between the sexes.Data DistributionsFigure 4.11 shows the demographic distributions (sex, age, weight, height andBMI). Figures 4.3 and 4.12 depict the data distributions for global and regionalsystolic dysfunction.94(a) Age (b) Sex(c) Weight (d) Height(e) Calculated BMIFigure 4.11: Demographic data distributions in the WMA cohort (n=2,910).95(a) Segment 1 (b) Segment 2 (c) Segment 3 (d) Segment 4(e) Segment 5 (f) Segment 6 (g) Segment 7 (h) Segment 8(i) Segment 9 (j) Segment 10 (k) Segment 11 (l) Segment 12(m) Segment 13 (n) Segment 14 (o) Segment 15 (p) Segment 16Figure 4.12: Breakdown of WMSI labels for segments 1-16 (shown inFig. 4.1), indicating the distribution of regional systolic function inthe WMA cohort. Description of WMSI scoring is given in Table 4.1.964.3.2 Experiments with Direct Segmentation-free Wall MotionAnalysisIn addition to standard hyperparameter search, experiments included:\u2022 Input Views: all seven combinations of the three views (23\u22121) possibilities;\u2022 Input Sampling: some variations of echo frame spatial (64\u00d764-256\u00d7256)and echo cine temporal sampling (8-25 frames per cycle);\u2022 Data augmentation: various types and degrees of data augmentation per-formed on the fly;\u2022 Feature extraction: variation of spatio-temporal neural networks (as de-scribed in Fig. 4.2.4);\u2022 Multi-view embedding fusion: variations of merging the extracted featuresto form the multi-view stream;\u2022 Outputs: various combinations of classification and regression outputs forthe yRWM, yEF and yGWM; excluding yEF and yGWM outputs altogether, etc.;\u2022 Optimization: variations of a weighted sum in the objective function, aswell as optimizers and regularization strategies;\u2022 Class imbalance compensation: different methods for compensating foroutput distribution imbalance (e.g. stratified batch sampling, class and sam-ple weighting).With yRWM considered as continuous labels, the coefficient of determination wasconsistently R2 < 0.2 across the views and segments. As the experiments were in-conclusive, the results presented here are not extensive. Example results for directEF, RWMA, and GWMA are provided in Fig. 4.13 below.Despite extensive experimentation, segmentation-free RWMA detection wasnot reliably achieved on the unseen test set, given our limited data. This is whilecompared to [16], the data cohort used in our experiments was six-fold larger(n=2,910) and entailed more dysfunctional wall segments across the relevant views.Asgharzadeh reported binary classification accuracies of up to 69.2%, on a dataset97(a)(b) (c)Figure 4.13: Typical results for direct systolic visual assessment for RWM,GWM and EF on an unseen pathologic test set (n=246) after conver-gence on the training set. With the available data, reliable direct re-gional systolic assessment could not be achieved.of (n=489) studies. A crucial factor to consider when interpreting these results isthat the prevalence of WMA in our cohort is much higher in our cohort comparedto [16], with 20% WMA.The visual assessment aims for a direct mapping of echocine loops to the highly noisy yRWM. Direct visual assessment is more challengingto troubleshoot as to its black-box nature. Hence, to further understand the RWMAproblem difficulty and noise level in data, we next focus on quantifying modelreliability via segmentation, which is more interpretable.4.3.3 Endocardium Segmentation PerformanceQualitative results of the segmentation performance analysis are shown in Fig. 4.14on several RWMA samples. For the subset of the data where the LV endocardial98tracings could be retrieved from the CPACS, we successfully obtained segmenta-tion results using the aforementioned (Section 4.2.5) A2C and A4C models. Weinvestigated the model performance based on segmentation (Dice D), wall distanceerror d, and \u2206EF :\u2022 different degrees of overall and global dysfunction;\u2022 regional model performance, i.e. comparison of performance on dysfunc-tional vs. normal LV segments;\u2022 comparison of A2C and A4C views;\u2022 robustness to cardiac phase, i.e. comparison of ED and ES results; and\u2022 fairness aspects, i.e. impacts of age, sex, and BMI of study subjects.In each case, we performed two-sample t\u2212tests to determine if the differences inthe distributions were statistically significant.Overall Segmentation Accuracy and EF Estimation99Figure 4.14: Qualitative comparison of ground-truth LV segmentation andmodel prediction on failure samples. We noted a tendency to missparts of the wall, especially lateral and septal walls (all) appearing insegments D and E, underestimate predicted area and subsequently vol-ume, fail drastically where shadows lead to blurry walls (b and e), over-estimate the major to minor axis length, yielding more slim LV maskprediction.100Quantity y vs. y\u02c6 Range Units View \u03c6 n (\u00b5,\u03c3) DiscussionSegmentationy, y\u02c6(D\u03c6 )[0.00,1.00]- A2C ED 690 (0.820,0.093)p = 0.02 difference between the EDand ES frames; 0.925 for controlcohort n = 85\u201d \u201d \u201d \u201d \u201d ES \u201d (0.814,0.100)p = 0.02 difference between the EDand ES frames; 0.903 for controlcohort n = 85\u201d \u201d \u201d \u201d A4C ED 646 (0.858,0.009)p = 0.01 difference between the EDand ES frames 0.936 in control co-hort n = 85\u201d \u201d \u201d \u201d \u201d ES \u201d (0.864,0.102)p = 0.01 difference between the EDand ES frames; 0.903 in control co-hort n = 85Volume y > 0 px3 A2C ED 690 (7.65,3.43)e4Tendency to underestimate ED vol-ume with ML; Range is for 600\u00d7800px images downsized to 128\u00d7170px\u201d y\u02c6 \u201d \u201d \u201d \u201d \u201d (5.08,2.54)e4\u201d\u201d y \u201d \u201d A4C ES 646 (4.44,2.39)e4Tendency to underestimate ES vol-ume with ML\u201d y\u02c6 \u201d \u201d \u201d \u201d \u201d (3.39,2.04)e4\u201dEF y [0.00,1.00]- A2C,A4CED,ES1,336 (0.43,0.10)Tendency to underestimate EF withML; EF calculated as in Equa-tion 2.1.Table 4.4: Summary of overall model performance for LV segmentation, volume estimation, and EF calculation forA2C and A4C views on ED and ES frames. Compared to the control cohort, consistently more significant errorswere obtained, suggesting lower performance with the existence of regional dysfunction, most likely not repre-sented in the original training set. We observed a tendency to underestimate the LV area and calculated volumein both ED and ES frames in both views. Higher errors were observed in predicting the ED area and volumecompared to ES.101Fig. 4.15 and 4.16 show the distributions of the predictions and clinical annota-tions for LV localization and function assessment are shown in Fig. 4.15 and 4.16.Our quantitative analysis (Table 4.4) revealed that performance was consis-tently poorer for severely dysfunctional samples for segmentation and functionmetrics. Samples with a few dysfunctional segments seem to illustrate poorerperformance in the \u2206EF compared to others. This is likely because our originalclinical training set consists of approximately 80% patients with no RWMA dys-functional segments. On the other hand, samples with > 2 dysfunctional severely-affected walls exert a global hypokinesis, which may appear more familiar to theML models that tend to learn the mean spatial-temporal behaviour. Looking at abreakdown of GWMA results, we noticed larger \u00b5 and \u03c3 for \u2206EF in globally dys-functional samples compared to those with only some regional dysfunction. Wealso quantified the errors for four categories of EF i.e.:\u2022 severe dysfunction (< 20%);\u2022 moderate dysfunction ([20%,35%));\u2022 mild dysfunction ((35%,50%]);\u2022 normal (> 50%).However, we did not observe statistically significant differences across EF clusters.102(a) CxO (b) CyO(c) WapexWbase (d) W LOWRO(e) Area (f) VolumeFigure 4.15: Comparison of ML prediction and clinical LV localization met-rics in terms of LV centroid (a and b), the length and width (major andminor axes dimensions in c and d), and the estimated area and volume(e and f). The data distributions are modelled as normal N (\u00b5,\u03c3).103(a) Stroke Volume(b) EFFigure 4.16: Comparison of ML prediction and clinical volumetric metricsstroke volume (a) and EF (b). The data distributions are modelled asnormal N (\u00b5,\u03c3).104Contrasting the segmentation results for ED and ES frames, we found ES metricsconsistently superior to ED (p < 0.05) in A4C and vice versa in A2C, as shownin Fig. 4.17a. A4C yielded better localization and EF estimation, as shown inFig. 4.17b.Regional Performance across LV Wall Segments105(a)(b)Figure 4.17: Comparison of segmentation accuracy D\u03c6 for key cardiacphases (\u03c6 ) ED and ES, where clinical tracings are available. Perfor-mance segmentation is significantly better in ED for A2C and ES forA4C. In both views, the tendency to underestimate EF is higher thanto overestimate (\u2206EF > 0). Higher errors were observed in A4C EFestimation compared to A2C. The risk of overestimating EF is lowerin A4C.106Quantity A2C A4CClinical La-belRWMA NoRWMARWMA NoRWMADistribution \u00b5 \u03c3 n \u00b5 \u03c3 n \u00b5 \u03c3 n \u00b5 \u03c3 nd\u03c6\u2aa7k (px) 22.9 19.3 5838 20.4 17.8 2442 20 29 5666 20.8 31.5 2086D\u03c6\u2aa7k 0.65 0.21 0.65 0.18 0.69 0.22 0.69 0.21D\u03c6 0.817 0.097 690 - 0.861 0.1 646 -ED ES ED ESD\u03c6 0.82 0.093 690 0.814 0.1 690 0.858 0.98 646 0.684 0.102 646\u2206EF (over-all)0.126 0.445 690 - 0.3 0.48 646 -\u2206EF inLAD0.131 0.488 457 0.115 0.345 233 0.318 0.486 494 0.238 0.464 152\u2206EF inRCA0.124 0.446 598 0.137 0.442 92 0.299 0.483 505 0.303 0.476 141\u2206EF in LCX - - 0.314 0.503 469 0.263 0.418 177Table 4.5: Summary of regional prediction accuracy across different subsets in terms of regional wall distance d\u03c6\u2aa7k,regional Dice D\u03c6\u2aa7k, overall Dice D\u03c6 and EF calculation error \u2206EF with and without the presence of RWMA.Results are broken down based on A2C and A4C views and their corresponding visible CAD regions. Notations:y expert-annotated quantity, y\u02c6 model-predicted quantity, \u03c6 phase (ED or ES), mask Dice D\u03c6 , number of samples nwith mean and standard deviation (\u00b5,\u03c3).107Figure 4.18: Overall comparison of wall distance errors dW,\u03c6\u2aa7k for diseased orhealthy (normal) regional wall motion across all visible segments k.The possible range of the wall distance error is the diagonal length ofthe 128\u00d7170 images (< 214 px).Compared to A2C, we saw significantly better results in Dice in A4C, althoughA2C yielded better EF estimation metrics. Wall distance errors in A2C were signif-icantly lower (p< 0.01) in normal compared to dysfunctional. However, the normaland abnormal distributions of A4C regional wall distance errors were similar.A closer look at the individual segments revealed inferior performance in A2Cof the diseased distributions for mid-anterior (E) in ED and ES and basal-anterior(F) in ES (p < 0.05). However, the pattern persisted with more minor differencesfor the other segments and phases. The distributions were not as different for A4Cwall distance errors. However, the lowest errors were obtained for segment mid-inferoseptal (B) (ES and ES), all < 16 px. These segments incidentally had thelowest differences between healthy and unhealthy groups, suggesting that evenin the presence of RWMA, the LV models can be reliably used to track the LVwall in this location. A4C also accounts for the highest errors in the apical-lateralsegment (F). Overall, on average, A2C segment-wise wall Dice errors were higher108for normal vs. dysfunctional (WMSI> 1), with an equal or more minor standarddeviation without significant significance. Differences were less pronounced inA4C except for the apical lateral segment (segment D).Dividing the EF estimation accuracy by the visible segments\u2019 CAD regions(shown in Figure 4.1), we did not see significant differences between the healthyand dysfunctional clusters of segments. In A4C, however, RCA and LCX hadlower errors for healthy compared to unhealthy cases. This suggests there is noperformance bias in normokinetic vs. dyskinetic segments in the RCA region inboth views and the LAD region in A4C. We did not obtain statistically significantdifferences across the CAD regions for Dice scores.Fairness Aspects of ML-generated PredictionsThe majority of the dataset is male patients (505 and 472 studies for A2C and A4C,respectively) compared to females (185 and 174, respectively). Although not statis-tically significant, the results suggested higher performance in the male cohort thanin females in terms of mean and distribution for Dice and \u2206EF . The bias observedin the model performance across the sexes likely stems from females being under-represented in the training datasets. Echo datasets are commonly predominantlymale due to the higher prevalence of cardiovascular disease in men [143].The WMA cohort contains a predominantly senior population with 78% 60+years old. In subjects below 40, we noticed a smaller skewed distribution of metrics(significant differences between mean and median), suggesting better performance.We did not otherwise observe particular patterns of bias in model performanceconcerning subjects\u2019 age.Performance Visualization with Weak Regional LabelsFigures 4.21 and 4.22 visualize the error quantification in A2C and A4C. The re-gional segmentation performance was assessed with dW,\u03c6\u2aa7k and D\u03c6\u2aa7k for static EDand ES frames (Figure 4.5). Overall performance was evaluated based on D\u03c6 forthe LV mask M\u03c6 and \u2206EF on each cine loop.109(a) A2C(b) A4CFigure 4.19: Wall distance errors dW,\u03c6\u2aa7k across individual segments of A2Cand A4C for segments A-F. Regional errors are consistently higher forsegments marked as diseased, i.e. yRWMk > 1, compared to healthy.110(a) Dice(b) \u2206EFFigure 4.20: Comparison of segmentation accuracy D\u03c6 (a) and segmentation-based EF estimation errors \u2206EF across views and number of dysfunc-tional segments (yRWM). Overall segmentation and EF accuracy are notsignificantly affected by the presence of RWMA.111Figure 4.21: Example RWMA sample and visualization of the regional wall distance errors in an A4C view. The firstand second columns show the image overlay of target masks and predictions, respectively, for ED and ES frames,illustrating that the ML model performs poorly around the apex in ED. The third column shows the target andpredictions overlay, and the third row depicts the ED-ES overlay. The regional clinical labels are shown in thebottom-left colormap for the six LV segments. Estimated LV wall motion from the target masks and predictionsare colour-coded on the corresponding columns. The fourth column shows colormaps highlighting the walldistance errors for ED and ES, as well as mean and maximum wall distance errors. The relative distributionof the regional wall detection errors corresponds with the RWMA labels, suggesting the model\u2019s difficulty intracking the LV wall when RWMA exists. Other clinical labels are listed on the far left. Localization errors forED and ES frames are listed on the far right.112Figure 4.22: Example RWMA sample and visualization of the regional wall distance errors in an A2C view. The firstand second columns show the image overlay of target masks and predictions, respectively, for ED and ES frames.The ML model demonstrates an overall tendency to underestimate the LV area and, subsequently, volume. Thethird column shows the target and predictions overlay, and the third row depicts the ED-ES overlay. The regionalclinical labels are shown in the bottom-left colormap for the six LV segments. Estimated LV wall motion fromthe target masks and predictions are colour-coded on the corresponding columns. The fourth column showscolormaps highlighting the wall distance errors for ED and ES and mean and maximum wall distance errors.The relative distribution of the regional wall detection errors corresponds with the RWMA labels, suggestingthe model\u2019s difficulty in tracking the LV wall when RWMA exists. Errors in ED seem more prominent than inES. Other clinical labels are listed on the far left. Localization errors for ED and ES frames are listed on the farright.1134.4 Discussion and Conclusion4.4.1 Echo Quality for Systolic AssessmentThis chapter demonstrated a novel application of ML-based view and quality grad-ing to efficiently classify echocardiogram clips by view and image quality. We an-alyzed clinical data of over 80,000 videos without manual annotation. The overallfindings of this study demonstrate that there are specific echo views that may be ofhigher quality and hence diagnostic yield compared to others.We demonstrated thata spatio-temporal ML model can rapidly predict RWMA-relevant echo image qual-ity and that the standard view with the highest maximum and mean image quality isthe A4C view. Conversely, the views of the lowest quality were the A2C and PSAXviews. These findings align well with the clinical observations previously made ona cohort with reported abnormal LV systolic function [143], which suggested theapical views were least likely to exclude abnormal segments when LV dysfunctionwas present. The echo view most likely to encompass an abnormal segment whenone was present was the A2C view at a prevalence of 93.4%, with close secondsbeing the A4C, A3C, and PSAX-PM views at 90.4% [143]. This also agrees withprevious ML-driven EF analysis observations that A4C video clips yielded higheraccuracy than A2C clips [25, 28], as presented in Chapters 2 and 3.Sources of Poorer Quality ScoresThe difference in view qualities is unknown but may reflect anatomic and er-gonomic challenges inherent to cardiac ultrasound. Several factors may impactimage quality, including patient characteristics and medical conditions. The non-diagnostic images account for 21% of obese patients (BMI>30), as opposed to7.8% of non-obese patients. This study of a consecutive cohort of 1,108 echo ex-ams also found inpatient status to be a predictor of a non-diagnostic echo, with anodds ratio of 1.75 [68]. Our group demonstrated similar findings concerning imagequality, hospitalization status, and diagnostic yield [142]. Poorer image quality wasobtained in the inpatient cohorts (mechanical ventilation or spontaneously breath-ing) compared to outpatients, especially for mechanically ventilated patients.114Diagnostic Implications of Image QualitySeveral studies have highlighted the importance of image quality in cardiac ul-trasound for effective clinical use [68, 142]. Studies with overall inferior imagequality had fewer standard parameters reported, suggesting the lower diagnosticutility of an echo exam [142]. The current study demonstrated that the A4C viewhad the best overall image quality and that the A2C view yielded relatively poorerimage quality as quantified by our ML model. When contextualized, these sug-gest that the view that best accounts for the probability of optimal image qualityand detection of LV systolic dysfunction would be the A4C view. Though the roleof image quality in echo interpretation seems intuitive, several studies have con-firmed the correlation of poor image quality with lower diagnostic yield, worseinter-observer agreement, and lower reproducibility in reporting [90, 217]. For ex-ample, a study comparing 3D echo and gated single-photon emission computedtomography (SPECT) imaging demonstrated that left ventricular volume measure-ments were less reliable with poorer echo image quality as graded by the percent-age of endocardial border definition [219]. Furthermore, in a cohort of heart failurepatients, image quality was also associated with an increased likelihood of a clin-ically unjustified repeat echo with an odds ratio > 2, indicating a potential addedcost beyond possible clinical implications [132]. This finding can also contributeto the literature that may enable the development of evidenced-based point-of-carecardiac ultrasound (POCUS) protocols, as POCUS becomes a standard extensionof the physical exam.Study LimitationsThere are several limitations involving the presented quality study:\u2022 Though our ML model has been shown to perform well for quality estima-tion, the image quality scores estimated by the ML model were developedusing a view-agnostic framework based on the percentage of blood-tissueinterface observed for the expected structures of a view. Hence, the ground-truth annotations for training were derived from relatively simple criteriawith limited accounting for valvular structures and image depth, potentiallyrestricting generalizability.115\u2022 The presented study excludes analysis of enhanced ultrasound images orthose that included colour Doppler, as the ML model [136] does not sup-port these modalities.\u2022 The exams selected for this study were a sample of studies with abnormal LVsystolic function. Therefore, the cohort likely represents an older populationwith a higher burden of cardiovascular and other comorbidities2, which mayadversely affect image quality and yield lower quality scores compared to arandom sample of all available data.4.4.2 Difficulties with Segmentation-free Wall Motion AssessmentWe found direct RWMA to be a challenging problem, and hence, contrary to directEF prediction, the methodology was not successful with the available data and re-sources. We hypothesize that this is due to the noisy modality, variability in clinicallabels compounded by the above smaller size of affected regions in RWMA:\u2022 Accurate RWMA assessment involves myocardium thickening, which re-quires accurate segmentation of endocardial (inner) and pericardial (outer)walls, whereas EF requires only endocardial tracing. This makes myocardialassessment doubly prone to segmentation errors by default.\u2022 Visual wall motion analysis is at least several times more difficult comparedto EF by definition, as it affects a much smaller portion of the field of view inthe echo image (up to six regions of LV are visible in each of the three viewsmentioned above). On the other hand, EF is a global measure of systolicfunction and impacts the appearance of the LV blood pool, which appearsdark in images.\u2022 Fundamental limits of ultrasound physics may impose a considerable andunequal challenge on wall motion analysis. Wall segments that are furtheraway from the transducer suffer more from poor visibility as they are imagedwith attenuated ultrasonic waves compared to those closer to the probe. Thismeans e.g. segments A and F in A2C and A4C views are fundamentally2Presence of medical conditions coexisting with a primary disorder116harder to view compared to segments closer to the LV apex. Additionally,wall segments not parallel to the off-centre vertical walls that form an anglewith the direction of ultrasound waves may also appear blurrier compared tothose aligned with the beam, such as segments B and C, in an image wherethe infero-septal wall is centred.4.4.3 Generalizability of Disease-agnostic ML-based Segmentationon WMA CohortsThis chapter presents a comprehensive investigation of the performance of twodisease-agnostic spatio-temporal LV function models (A2C and A4C) on an unseenWMA cohort. We presented a framework for regional and global assessment ofautomated LV segmentation, endocardial wall detection, and EF estimation. This isthe most extensive independent validation performed on clinical echo data focusingon a particular disease to the best of our knowledge.Our experiments showed that in the presence of RWMA, ML models trainedon average clinical distributions might fail to capture the wall reliably. We also ex-plored the results through a fairness lens and observed some biases regarding sex.We obtained better performance on ES than ED and a more reliable segmentationin A4C, despite more accurate EF estimation in A2C. We conclude that beyondthe development of large innovative neural network architectures and training reg-imens, valuable insights can be unveiled about the limitations of state-of-the-artmethodology to guide future research better. Careful integration of complex clin-ical knowledge about sophisticated anatomy and physiology (such as those of theheart), performance and impact assessment can unlock the next steps in reliableML-driven computer-aided diagnosis.Understanding the Propagation of Errors in Calculating Clinical MetricsTo get a tangible estimate of the LV perimeter, assume LV covers 1\/3 of the ul-trasound image height, and the base and apex account for 1\/8 of the image width.For an 800\u00d7600 image, LV perimeter accounts for approximately 500 pixels alto-gether. Let L\u03c6 , A\u03c6 , and V \u03c6 denote the respective LV long-axis, area, volume of theLV at phase \u03c6 . For the typical pixel spacing of 0.0332mm\/px, each pixel within117the image accounts for 0.0011mm2.Propagation of these errors from EF calculation from volume estimates (Equa-tion 4.18) yields:E\u0302FEF= (V\u0302 ED\u2212V\u0302 ES)V\u0302 ED(V ED\u2212V ES)V ED= (V\u0302 ED\u2212V\u0302 ES)V ED(V ED\u2212V ES)V\u0302 ED (4.23)As per Equations 4.19 and 4.23, the ratio of V\u0302 \u03c6 to V \u03c6 errors can be described as:V\u0302 \u03c6V \u03c6= 0.85 (A\u0302\u03c6 )2L\u0302\u03c60.85 (A\u03c6 )2L\u03c6 =(A\u0302\u03c6)2(A\u03c6) = (A\u03c6 +\u2206A\u03c6A\u03c6 )2 (4.24)As the errors in LV diameter are negligible for small NW , let us assume L\u02c6\u03c6L\u03c6 = 1.Hence, as the average segmentation error or overall size of LV increases, volumecalculation errors increase quadratically.To estimate the error in area calculations \u2206A = A\u2212 A\u02c6, assume a scenario wherethe predicted LV wall is off by a uniform Ner pixels across the overall enclosedcurve. If the number of pixels on the wall is NW , for a pixel spacing of (px, py), thechanges in the area calculation\u2206A = (Ner.px.py).NW (4.25)E\u0302FEF= V EDV\u0302 ED= (AED)2(AED+\u2206AED)2 (4.26)If segmentation accuracy is not equivalent in ED and ES, EF errors also become afunction of stroke volume.Significance of Investigating Disease-agnostic Models and Pathologic DatasetsThe initial proof of concept with ML and computer vision tools has been demon-strated by researchers worldwide in the past few years. Current publications of-ten report results based on small data sets, and although eye-catching, they areprone to overfitting and overestimation of accuracies [133, 200]. However, it iscrucial to err on the side of caution in developing ML-based image analysis toolsas we move towards the generation of clinical-grade diagnostic tools [133]. Heart118disease affects heart structure and function, causing diseased cohorts to exhibitdifferent behaviour compared to control cohorts. In addition, pathologies affectcardiac function, impacting spatio-temporal patterns captured by ultrasound sig-nals and artificially interpreted by ML models. While a few public datasets exist(EchoNet Dynamic [172], HMC-QU [96], etc.), they only contain a small set oflabels. Notably, the performance of state-of-the-art models has not been exploredmuch in diseased cohorts. These ML-based models are often trained and validatedon high-quality cherry-picked private datasets to show initial feasibility. Hence,to further enable translational research in ML-driven diagnostics, incorporatinglow-level medical domain knowledge and a fine-grained understanding of the ro-bustness of such models for diagnostic tasks is essential.Beyond academic settings, ML-based state-of-the-art echo analysis methodsare not yet sufficiently validated in clinical cohorts with various complex condi-tions (cardiovascular and otherwise). The significance of this issue was highlightedby the recent recall of two commercialized automatic EF tools. In 2019, the Amer-ican Food and Drug Administration (FDA) issued a recall for the GE HealthcareVscan Extend ultrasound device due to the overestimation bias of ML-based EFestimation (Z-1839-2019 [71]). Shortly after, in 2020, the Butterfly iQ device fea-turing a similar Auto-EF tool was recalled (Z-1601-2020 [72]) due to the lack of510(k) approval, which requires testing and documentation to ensure that safe dis-tribution and usage for the said Class 2 device. The recent FDA recalls of commer-cialized EF products suggest these algorithms are not yet ready for safe large-scaleclinical deployment. Hence, further comprehensive analysis of ML model per-formance in the presence of pathology is essential to unlocking the next steps inautomated ML-driven diagnostics.4.4.4 Future Directions for Wall Motion AnalysisFuture work for wall motion analysis will likely benefit from adding more infor-mation, i.e. more data or more priors.119More Training Data?Compared to EF assessment, wall motion analysis is a more challenging task andmay technology beyond the solution proposed for direct EF estimation in Chap-ters 2 and 3. This is because:\u2022 unlike EF, whose variations affect the global LV dynamics, regional detec-tion of WMA requires a more focused, localized and subtle examination ofthe cardiodynamics;\u2022 no clinical segmentation or direct mapping between the pixel data and asso-ciated regional WMSI labels exists; the ground-truth WMA measurementsare acquired entirely semi-quantitatively themselves and are hence noisy;\u2022 WMA measurement relies on more views, namely, A2C, A4C, PLAX, andPSAX (at different levels).Nonetheless, it is possible that increasing the training set of the neural network(e.g. by a factor of ten or more) will enable effective spatio-temporal learning fordirect RWMA detection with WMSI labels. In addition, ensuring the diversity ofsamples in the training set is crucial for the model to prevent biases towards clinicalcohorts.Challenges regarding the increasing size of datasets are non-technical. Dataretrievals put a load on the clinical PACS at VCH, which provides medical ser-vices to five hospitals. Large and frequent retrievals for research increase the riskof crashing servers and should hence be avoided. Furthermore, data query anddownloads (sequence diagrams in Appendix B) requires coding and supervisionby data analyst at regional health authorities (VCH), who have narrow bandwidthsdue to regular responsibilities, especially with the COVID-19 pandemic. There-fore, although ethics-approved clinical data exists on servers, the amount of datadownloaded goes through an infrastructure and staff bottleneck; hence, it is non-trivial to acquire and limited. As we go beyond simple tasks such as EF detection,our group is working towards addressing this issue by designing and establishingsafe data pipelines with high bandwidth for research.120More Prior InformationAlternatively, additional priors can be supplied to the model at training time re-garding the LV and myocardial wall location. Potential pathways forward include:\u2022 acquiring manual annotation of the myocardial wall by echocardiographers;this is an extremely expensive approach;\u2022 acquiring manual pericardial wall annotations by echocardiographers to usetowards a myocardial wall extraction, combined with the presented ML-based endocardial tracking (similar to [131]);\u2022 querying the clinical database for pericardial annotations that may exist forother diagnoses, such as detection of pericardial effusion, to use along theendocardium segmentation presented;\u2022 extracting strain analysis and wall tracking results from advanced echo soft-ware (e.g. syngo Dynamics by Siemens Healthineers) to use for augmentingmyocardial training data; this path requires ethical, legal, and technical con-siderations around the possibility of establishing pipelines to output analysisresults from proprietary software in approved procedures.121Chapter 5Automatic Diastolic DysfunctionDiagnosis from Echo-derivedParameters5.1 Introduction5.1.1 Clinical BackgroundHeart Failure and Diastolic FunctionLeft ventricular diastolic dysfunction (LVDD) refers to abnormalities in the relax-ation phase and refilling of the ventricle in systole. An abnormal diastolic functionrepresents one of many proposed mechanisms underlying the development of heartfailure (HF) in these patients [7, 178]. The degree of severity of LVDD is as-sociated with clinical outcomes of HFpEF [224], leading those with more severediastolic function abnormalities to demonstrate worse outcomes [9, 191], with mor-This chapter is adapted from i) D. F. Yeung*, R. Jiang*, D. Behnami*, J. Jue, R. Sharma,M. Turaga, C. L. Luong, M. Y. Tsang, K. G. Gin, H. Girgis, et al. Impact of the updated dias-tolic function guidelines in the real world. International Journal of Cardiology, 326:124\u2013130, 2021and ii) R. Jiang*, D. Yeung*, D. Behnami*, J. Jue, M. Tsang, K. Gin, C. Luong, P. Nair, H. Girgis,P. Abolmaesumi, et al. Machine learning to facilitate assessment of diastolic function by echocar-diography. Canadian Journal of Cardiology, 35(10):S4\u2013S5, 2019.122tality rates of 10-30% [42]. LVDD is often (but not always) associated with heartfailure with preserved ejection fraction (HFpEF) [203]. Approximately half of allpatients with heart failure have a preserved ejection fraction (EF), but their out-comes are no better than those with reduced EF [174].Diastology Guidelines in EchoThe diastolic cardiac function can be assessed non-invasively with echocardio-graphy (echo). Echocardiographic assessment of diastolic function is complexbut has important diagnostic and prognostic implications in managing heart fail-ure.Detecting and reliable grading of LVDD can significantly impact diagnosingand managing HFpEF [224]. Diastology relies on the grading of diastolic functionbased on category (normal, mild, moderate, severe, indeterminate), which providesa modest correlation with hemodynamic measurements and remains echocardiographer-dependent.Cardiologists often follow the community-standard recent clinical guidelinesto diagnose patients with LVDD, which are complex and rely on the assimila-tion of numerous echocardiographic findings [135, 163]. LVDD is detected al-gorithmically, based on metrics that assess myocardial relaxation, stiffness, andleft ventricular filling pressure [224]. Several protocols and guidelines have beenpublished to facilitate the echocardiographic assessment of diastolic function [4,155, 156, 176, 191]. However, prior algorithms have been limited by significantinter-observer variability and modest correlation with invasive hemodynamic mea-surements [43, 79, 223].LVDD-relevant MeasurementsThe LVDD-related measurements and their description are listed in Table 5.1.These measurements were acquired by the aforementioned Level III echocardiog-raphers based on the most updated Chamber Quantification Guidelines [126, 127]at the time of the exam review. Measurements were retrieved from the FileMakerPro 7 (Apple Inc., Cupertino, California, United States). Figure 5.2 and 5.3 il-lustrate the acquisition of pulsed Doppler at the mitral valve and possible E\/Ascenarios. Diastole consists of an initial phase of rapid (high-velocity) filling123Notation Diastology Input Parameter Label Type- Patient\u2019s age ContinuousLVEF Biplane Left ventricular ejection fraction CategoricalLAVi Left atrium volume index ContinuousVTR Tricuspid valve Regurgitation blood flow velocity CategoricalE E-wave magnitude; trans-mitral inflow early fillingvelocity in early diastoleContinuouse A-wave magnitude in late systole, i.e. atrial contrac-tion phaseContinuousA Mitral annular tissue Doppler velocity Continuouse\u2032lat E\u2019-wave magnitude at the lateral wall Continuouse\u2032sep E\u2019-wave magnitude at the septal wall ContinuousE\/A Ratio of mitral inflow early filling velocity (E-wave)to mitral inflow velocity during atrial contraction(A-wave)ContinuousE\/e\u2032 Ratio of mitral inflow early filling velocity (E-wave)to mitral annular tissue Doppler velocityContinuousTable 5.1: Parameters involved in echo-based diastology. These quantitiesare used as inputs to the guideline algorithms (rule-based and neural net-works) presented in this chapter to determine LVDD.of LV, followed by more gradual filling due to the contraction of the left atrium(LA), which further forces blood into the LV. The peak velocity for the rapid fill-ing acquired from the pulsed Doppler (known as E-wave) and the peak velocity atthe atrial contraction phase (known as the A-wave) are critical measurements forevaluating diastolic function. Using these values, an E\/A ratio can be calculated.The clinical guidelines apply conditions and define criteria for LVDD based on therange of E\/A.5.1.2 Challenges for Detecting LVDDGuideline-based diastology is very complex, and assessing diastolic function ishighly variable because LVDD is both difficult to observe and define. Challengesinvolving deriving echo-based measurements stem from the complex and extensiveanalyses required for performing Guideline-based assessments:124\u2022 Multi-chamber Analysis: LVDD detection depends on measurements fromother heart chambers such as the left atrium (LA), the LV mass, etc., all ofwhich are prone to segmentation and measurement errors.\u2022 Multi-view Analysis: LVDD involves several input views, introducing quality-related challenges in the hard-to-capture views, as discussed in previouschapters\u2022 Multi-modal Analysis: LVDD detection requires examination of B-mode,M-mode, and Doppler modalities, which are highly noisy.What is more, LVDD is also inherently hard to define:\u2022 Coarse Decision Space: As shown in Fig. 5.1, Guidelines suggest a com-plex algorithm that involves several decisions with sharp decision boundaries(e.g. binary (yes-no branches, interval-based conditions, etc.) This discretedecision-making may easily lead to misclassification, especially when ob-served measurements are close to the decision boundaries.\u2022 Indeterminate Output: According to Guidelines, it may not be inherentlypossible to determine if diastolic dysfunction exists, which further empha-sizes that diastolic function assessment is an ill-posed problem.\u2022 Variable Decision Space: With new research, the accepted clinical guide-lines are updated, which may cause diagnoses to change with identical inputparameters.125Figure 5.1: The algorithm for assessing the LVDD based on the updated 2016ASE\/EACVI Guidelines [156].126(a)(b)Figure 5.2: Schematics of mitral flow peak velocity estimation using pulsedDoppler on the mitral valves for determining the E\/A ratio for diastolicfunction analysis. Pulsed Doppler is placed on the mitral valve (a), andvelocity E and A-waves are obtained (b). (Figures are inspired by dia-grams in [1]).127Figure 5.3: Scenarios of E\/A calculated from mitral flow velocities and clin-ical interpretations of normal vs. abnormal diastolic blood flow.1285.1.3 Related WorksSubstantial differences in the grading of diastolic function have been demonstratedwhen prior guidelines and protocols have been compared with one another [164].Therefore, such variability could result in a different diagnosis for the individualpatient and result in differences in the estimated prevalence of LVDD in the pop-ulation [10, 66, 77, 201]. The most recent iteration published by the AmericanSociety of Echocardiography (ASE) and European Association of Cardiovascu-lar Imaging (EACVI) in 2016 [156] aimed to improve on the previous 2009 ASEGuidelines [155] by simplifying the algorithm to focus on the most reliable mea-surements. Figure 5.1 demonstrates our interpretation of the latest clinical guide-lines for LVDD detection, i.e. 2016 ASE\/EACVI Guidelines. While some stud-ies have demonstrated improvements in inter-observer variability [157], invasivehemodynamic correlation [11, 20], clinical diagnosis [197], and outcomes [197],others have shown only modest correlation [160] and have identified unresolvedissues with the updated guidelines [3, 66, 177].Related works involving machine learning (ML) in the context of diastologyare limited. Sengupta et al. proposed early diastolic dysfunction detection withelectrocardiogram (ECG) on a cohort with referrals for coronary computed tomog-raphy (CT) [202]. Yang et al. used convolutional neural networks to detect LVDDbased on heart sound analysis [239]. [175] have explored using a neural networkto phenogroup LVDD and identify patients with preserved ejection fraction (EF)based on hemodynamic measurements and outcome data [175, 224].5.1.4 Chapter SummaryShifting gears from systolic to diastolic cardiac function, in this chapter, we investi-gate the feasibility of using ML for predicting LVDD based on relevant parametersand measurements derived from echo exams (as per clinical guidelines). We firstperform a comparative study between the 2009 ASE [155] for LVDD and 2016ASE\/EACVI [156] Guidelines on a retrospective cohort and study the reclassifi-cation of LVDD according to the latest Guidelines. We then implicitly use neuralnetworks to learn the most recent guidelines for LVDD classification. Finally, thenetwork is modified to perform the regression task to obtain a novel continuous129score describing the severity of diastolic dysfunction. The key contributions in thischapter are the following:\u2022 Presented studies are the most extensive and comprehensive analysis of therecent Guidelines and their impacts compared to the previous iteration (2009)for diastology [73], with significant clinical implications;\u2022 We present the first use of deep learning in diastolic dysfunction to replicateclinical Guidelines on a test set blinded to the 2016 ASE\/EACVI algorithm;and\u2022 We propose a novel methodology for continuous scoring of LVDD severityusing deep learning, which, if adopted, can improve the resolution of theclinical decision space, which is currently discrete and entails sharp decisionboundaries.5.2 Material and Methods5.2.1 Clinical DatabaseThe DFileMaker database (described in Chapter 2) was screened for clinical chartdata relevant to LVDD. The measurements originate from studies performed at fivehospitals in the Vancouver Coastal Health (VCH) jurisdiction within the ethics-approved period of December 30, 1999, to December 31, 2015. All echo measure-ments were obtained by certified cardiac sonographers with commercially availableultrasound systems:\u2022 Vivid I (GE Healthcare, Milwaukee, Wisconsin); and\u2022 iE33 (Philips Medical Imaging, Andover, Massachusetts).Categorization of LVDDThe diastolic function formally reported in these echo studies represents an inter-pretation of diastolic function based on the 2009 Guidelines. LVDD was capturedin the database is categorical with six classes:1301. normal diastolic function;2. mild diastolic dysfunction;3. moderate diastolic dysfunction;4. severe diastolic dysfunction;5. abnormal diastolic function but cannot be graded ;6. indeterminate;The abnormal but cannot be graded classification is not a universally adopted cat-egory. Therefore, this group, which represented 7% of studies, was merged withthe indeterminate category in our subsequent analyses.5.2.2 Comparison of 2009 and 2016 Clinical Guidelines for LVDDOur study aims to determine how the 2009 ASE Guidelines and the 2016 ASE\/EACVIGuidelines compare with respect to the assessment of diastolic function in a large,unselected, real-world cohort of echo studies. To accomplish this, we used the2016 Guidelines to reinterpret the diastolic function in all echo studies previouslyreported based on the 2009 ASE Guidelines by one of six Level III echocardiog-raphers (initials KG, JJ, PL, PN, MYCT, TT). We aim to demonstrate the impactthat a change in Guidelines could make on the assessment of diastolic function ina large, unselected real-world cohort of echo studies. To this end, the algorithm forLVDD detection was implemented in Python 3 to replicate the ASE\/EACVI Guide-lines and objectively assess diastolic function based on previously collected echomeasurements. We assume that all quantifiable variables were measured based onthe most up-to-date Guidelines, even though variables such as LV wall thicknessand left atrial volume can be difficult to measure accurately [78].Study CohortsWe established five separate cohorts of the echo studies identified within the studyperiod for the comparison of the 2009 and 2016 Guidelines:\u2022 Cohort A (Total Cohort) includes all identified studies.131\u2022 Cohort B (Validation Cohort) includes 100 randomly selected echo studiesto validate the proposed algorithm;\u2022 Cohort C (Post Exclusion Cohort) excludes echo studies with features thatwould interfere with diastolic function assessment (listed in Table 5.1);\u2022 Cohort D (Suspected Heart Failure Cohort) includes only patients in Co-hort C who had dyspnea or heart failure listed explicitly as the indication forthe echo study; and\u2022 Cohort E (Preserved LVEF Cohort) includes all patients in Cohort C ex-cept those with evidence of myocardial disease.A breakdown of the cohorts established is shown in Fig. 5.4. Statistics of thecohort demographics and clinical indications can be found in Table 5.2. StudiesFigure 5.4: Overview of clinical data cohorts established for the comparisonof 2009 ASE [155] and 2016 ASE\/EACVI [156] for evaluating left ven-tricular diastolic function.with missing or invalid entries (based on Table 5.1), as well as those containingfeatures that preclude or complicate diastolic function assessment, were excluded;i.e.:\u2022 abnormalities in the patient\u2019s mitral valves (stenosis, rheumatism or pros-thetic valve replacements);132Cohort A B C D EDescription Total Validation Post-exclusionSuspectedHFPreservedLVEFSamples (n) 71,727 100 55,396 2,648 30,854DemographicsAge (years) 62\u00b117 60\u00b115 59\u00b117 65\u00b116 55\u00b11Sex - female 34,366(48%)42 (42%) 26,243(48%)1506(57%)15,791(51%)Study IndicationLVEF - reduced 10,612(15%)11 (11%) 6026(11%)366(14%)0 (0%)LV function -dysfunctional24,541(34%)28 (28%) 18,829(35%)726(27%)9051(29%)Dyspnea or HF 3981(6%)5 (5%) 2648(5%)2648(100%)1315(4%)Table 5.2: Baseline characteristics of each cohort assessed for LV diastolicfunction and filling pressures\u2022 more than moderate mitral regurgitation or Mitral Annular Calcification (MAC);\u2022 non-sinus rhythm complications;\u2022 complications in the pericardial sac (confirmed or suspected build-up of fluid(i.e. tamponade) or constriction, and more than trivial pericardial effusion);\u2022 hypertrophic cardiomyopathy; and\u2022 congenital heart disease.Additionally, we choose the study period February 1, 2010 - March 31, 2016, suchthat one year of adoption is considered for the 2009 ASE Guidelines [155]. Theperiod ends prior to the publication of the 2016 ASE\/EACVI Guidelines [156].Other possible myocardial diseases, including coronary artery disease (CAD),could not be identified with certainty in our study due to the lack of additionalcorresponding clinical history. Instead, only those with overt RWMA other thanabnormal ventricular septal motion were considered to have objective evidence ofmyocardial disease.133Systolic Function CriteriaIn addition, the precise cut-off that constitutes a reduced LVEF has been variablydefined [155, 156]. LVEF was measured by Simpson\u2019s biplane method when theimage quality allowed accurate volumetric assessment or by visual estimate other-wise. We, therefore, defined a reduced LVEF according to the following hierarchy:1. a qualitative description of mild, moderate, or severe LV systolic dysfunctioneven if the quantitative LVEF measures \u2265 50%;2. a quantitative LVEF <52% in men and LVEF <54% in women if no qualita-tive description of LV systolic function was provided;3. the lower value of the LVEF <52% in men and <54% in women if only arange were reported.Validation CriteriaTo validate the proposed algorithm, three Level III echocardiographers (RS, MT,DY), who were not involved in the formal interpretation of any of the echo stud-ies performed within the study period, independently extracted the data from theformal echo reports of 100 randomly selected echo studies from Cohort B (Valida-tion Cohort) and used these measurements to provide an interpretation of diastolicfunction based on the 2016 ASE\/EACVI Guidelines.5.2.3 Modelling Diastolic Dysfunction with Neural NetworksMulti-layer Perceptron for LVDD ClassificationLVDD diagnosis can be modelled using a feed-forward neural network, where theinput vector X\u20d7i for the i-th study is constructed from the LVDD-related parameterslisted in Table 5.1. The neural network used for learning the LVDD guidelines isshown in Fig. 5.5. A network with two hidden layers with 32, 128, and 128 neuronsis used to obtain feature vectors z\u20d7i for sample i. Two hidden layers were empiricallyfound to be the smallest network that led to smooth network convergence. On thefirst hidden layer, a scale-up factor of four was considered, followed by a second134128-neuron layer to further embed the input features. A Rectified Linear (ReLU)activation is used to introduce nonlinearity in the model, i.e.ReLU(x\u20d7i) =max(0, x\u20d7i) (5.1)To classify the LVDD according to the discrete clinical labelling (y = 1 \u2236 4), labelsare one-hot-encoded, and a softmax activation is applied on the output of the lasthidden layer to obtain the likelihood of LVDD classes:so f tmax(z\u20d7i) = ez\u20d7i\u22114j=1 ez\u20d7 j (5.2)a categorical (four-class) cross-entropy is used to optimize the model parameters:`CCE = \u2212\u2211iY DDi \u22c5 logY\u02c6 DDi (5.3)Continuous LVDD GradingAs clinical guidelines rely on discrete decisions, they do not produce continuousscores for LVDD. The coarse decision space may result in misdiagnosis for casesnot close to inter-class decision boundaries. In order to obtain a continuous LVDDscore, we decouple the final layer with a single-node layer with sigmoid activationbounded between 0 and 1;\u03c3(z\u20d7 j) = 11+e\u2212z\u20d7 j (5.4)To map the labels to a continuous output space 0-1, clinical labels are normalizedyDD = Y DD4.0 . The objective function to train the regression model can hence be a rootmean square loss, i.e.`reg =\u221a(yDDi )\u2212 y\u0302DDi )2 (5.5)Model TrainingLVDD labels were determined for the dataset based on the ASE\/EACVI 2016 di-agnostic algorithm (as described earlier). Continuous labels in the inputs werenormalized based on the overall parameter range to construct X\u20d7i. The dataset with135input vectors (X\u20d7i,yDD) was split into training (80%) and validation (20%) sets.These criteria yielded a dataset of the total of 7,728 studies from the total 224,026studies in DFileMaker.136(a)(b)Figure 5.5: Neural network architecture for classification of diastolic dys-function based on diastology parameters, i.e. age, LV EF , LAVi, VT R,E, e\u2032lat , e\u2032sep, E\/A, E\/e\u20321375.3 Results5.3.1 Impacts of Updated Clinical Guidelines on LVDDContinuous variables are reported as means with standard deviations (\u00b5,\u03c3). Cat-egorical variables are reported as absolute values with percentages. Concordancewas evaluated using the Kappa statistic, where p-values <0.05 were consideredstatistically significant.Additionally, 100 echo studies were randomly selected from Cohort A, whichserved as a representative sample (Cohort B, Validation Cohort) for manual val-idation of the coded ASE\/EACVI 2016 algorithm. The characteristics of thesepatients were similar to those of Cohort A (5.2). The characteristics of the othertwo subcohorts, Cohort D and E, are listed in Table 5.2. Disagreement in diastolicfunction grading between the algorithm and echocardiographers was found in 9 ofthe 100 echo studies selected for manual validation (Cohort B). Errors in manualgrading accounted for 6 of the discrepancies. The remaining three discrepancieswere a result of an implementation error in the algorithm. Upon revision of thecode, full concordance in grading was achieved.Diastolic function interpretation reported in the echo studies performed be-tween February 1, 2010, and March 31, 2016, was based on the 2009 ASE Guide-lines. In Cohort A, this resulted in the following distribution of diastolic function:32% normal and 36% indeterminate diastolic function; 23% mild, 8% moderate,and 2% severe diastolic dysfunction (Fig. 5.6). A similar distribution was ob-served for all other cohorts. When the assessment of diastolic function based onthe 2009 ASE Guidelines was compared to that based on the 2016 ASE\/EACVIGuidelines, there were significant differences in the proportion of studies classi-fied as normal (23% vs. 32%) or indeterminate (43% vs. 36%) diastolic function,and mild (23% vs. 23%), moderate (10% vs. 8%), and severe diastolic dysfunc-tion (1% vs. 2%), with a poor agreement between the two methods (Kappa 0.324,95% CI 0.319\u20130.329) (Fig. 5.6). A similar pattern of differences was observed forall other cohorts, with an additional finding of a substantial reclassification frommild diastolic dysfunction to normal diastolic function in the subgroup of patientswith preserved LVEF and no evidence of myocardial disease. When diastolic func-138tion grading was stratified by age, a change in the Guidelines from 2009 to 2016resulted in the following trends:1. an increase in studies graded normal in patients \u2265 55 years old;2. a decrease in studies graded mild diastolic dysfunction in patients \u2265 65 yearsold;3. a reduction in studies with indeterminate diastolic function in patients <55years old; and4. an increase in studies with indeterminate diastolic function in patients 55\u201364years of age (Fig. 5.7).139(a) Total Cohort (A)(b) Validation Cohort (B) (c) Post-exclusion Cohort (C)(d) Suspected HF Cohort (D) (e) Preserved LVEF Cohort (E)Figure 5.6: Comparison of diastolic function grading based on the 2016ASE\/EACVI Guidelines [156] compared to the 2009 ASE Guide-lines [155] in the five established cohorts outlined in Table 5.2.140(a) 2009 ASE [155](b) 2016 ASE\/EACVI [156]Figure 5.7: Diastolic function grading is stratified by age category when as-sessed using the 2009 ASE Guidelines (a) and the 2016 ASE\/EACVIGuidelines (b).141Figure 5.8: Normalized confusion matrix for prediction of LVDD using theproposed neural network v.s. the ASE\/EACVI 2016 Guidelines. Theindeterminate class was excluded in this study.Figure 5.9: Diastolic function scores obtained by the regression LVDDmodel. A continuous output space between severely dysfunctional tonormal function is assumed.When tested on the validation dataset, our neural network reclassified the stud-ies with 99.0% agreement with the 2016 ASE\/EACVI diastolic function grading(confusion matrix shown in Fig. 5.8). In addition, we generated a novel continuousscore (Fig. 5.9) corresponding to an overall diastolic function that correspondedwell with diastolic function grading using the ASE\/EACVI Guidelines [156] (Fig. 5.8).1425.4 Discussion and Conclusion5.4.1 Significance of the Presented ML Use Case for DiastologyIn this chapter, we proposed a multi-layer perceptron that can reliably learn thelatest LVDD detection algorithm and further enable the generation of a continuousdiastolic function score, which is not otherwise possible with discrete categoriza-tion based on guidelines. We presented a novel ML-based diastology modelling bytraining a supervised neural network with echo-based measurements and resultingLVDD labels derived from clinical guidelines. Using relevant echo measurementsas network inputs, we showed that the proposed feed-forward network could ac-curately learn and predict a four-class classification of LVDD. Additionally, weshowed that such ML-based modelling of diastolic dysfunction could enable con-tinuous scoring for the dysfunction severity. This was achieved by replacing thefinal layer with a regression layer to predict a continuous score for LVDD for thefirst time. We showed that the continuous labels correspond well with current rec-ommendations of diastolic function grading. Currently, echocardiographers followlogic-based Guidelines, which have sharp decision boundaries and are inherentlyprone to misclassification. If adopted, the proposed method can have importantdiagnostic and prognostic implications, as it provides a more fine-grained assess-ment of diastolic function, which is associated with risks of heart failure (HFpEF).Whether ML can be used to derive a more precise diastolic function score had notbeen previously demonstrated. The high accuracy of 99% reported for the classifi-cation task on the unseen test set is not surprising because:\u2022 neural networks can learn any arbitrarily complex, linearly non-separabledecision space (universal approximation theorem [93]); and\u2022 the network inputs are arrays of echo-derived measurements, which withrespect to the defined learning task, are not noisy observations. (This isunlike Chapters 2-4, which involved learning the mapping between imagesand measurements.)The proposed framework cannot address the issue of label ambiguity in the clini-cally indeterminate class of diastolic function.1435.4.2 Comparison of GuidelinesWe presented a large comparative study to investigate the impacts of clinical guide-lines for assessing diastolic function.Our results show 2016 ASE\/EACVI Guide-lines lead to a significant reclassification of diastolic function grade. Reclassifica-tion is mainly from indeterminate to normal, with an additional tendency towardsreclassification from mild diastolic dysfunction to normal diastolic function in pa-tients with preserved LVEF and no evidence of myocardial disease. To the best ofour knowledge, this is the most extensive study of diastolic function assessmentand provides a compelling perspective on the impact that a change in guidelinescould make on diagnosing diastolic dysfunction in the real world.Clinical Implications of LVDD ReclassificationAs pointed out by Fraser and Girerd, the presented study is the largest LVDD studythat investigates the impacts of the clinical Guidelines on real-world diastology. Weobserved an improved LVDD specificity with ASE\/EACVI 2016 Guidelines. Manypatients with preserved LVEF previously reported having mild LVDD based on the2009 ASE Guidelines would have been classified as having normal diastolic func-tion using the 2016 ASE\/EACVI Guidelines. Approximately 48% of the 71,727echo studies performed in our single-center study alone would have received a dif-ferent diastolic function interpretation during our six-year study period.Cliniciansshould hence be aware of the tendency for these updated Guidelines to be morespecific and less sensitive in identifying LVDD, as this would have significant im-plications for identifying patients with or at risk for heart failure.Our findings are consistent with prior studies that revealed differences in dias-tolic function interpretation depending on which algorithm is applied [10, 77, 97,197, 201]. These studies had generally demonstrated a decrease in the proportionof patients classified as having diastolic dysfunction when the 2016 ASE\/EACVIalgorithm was used instead of the 2009 ASE [4, 14, 155, 156, 176]. A population-based cohort of 1000 individuals with no known cardiac disease and a preservedLVEF showed a decrease in the classification of diastolic dysfunction from 38.1%to 1.4% [10]. Another population-based cohort of 1485 individuals with no heartfailure and a preserved LVEF demonstrated a decrease in the classification of dias-144tolic dysfunction from 5.9% to 1.3% [97]. The pattern of reclassification observedin these studies has been attributed to an increase in specificity of the updated pro-tocol at the cost of a significant decrease in sensitivity [10, 77]. For example, ina study of 157 patients referred to a heart failure clinic, 49% of patients whosediastolic function interpretation was changed from abnormal to normal due to theupdated Guidelines demonstrated less clinical and biochemical evidence of heartfailure [197]. In another study of 90 patients referred for clinically indicated leftheart catheterization with studies performed immediately prior, the sensitivity ofecho to detect diastolic dysfunction decreased from 79% to 69%. At the sametime, the specificity increased from 70% to 81% when moving from the 2009 tothe 2016 Guidelines [20].5.4.3 ML for Image-based DiastologyThis chapter focused on determining LVDD based on echo-derived parameters.Future work can explore fully-automated diastolic function assessment based onecho images.Required ModulesImage-based LVDD detection requires modules for deriving the measurements inTable 5.1. That is, in addition to LVEF estimation (methods presented in Chapters 2and 3, and segmentation-based methods proposed by Jafari), the following imageanalysis tasks need to be addressed:\u2022 left atrium, or multi-chamber segmentation and quantification (for LA massestimation);\u2022 spatio-temporal Doppler analysis for measuring E, e\u2032, A waves and associ-ated ratios; and\u2022 spatio-temporal Doppler analysis for measuring blood flow velocities (Fig. 5.10).Understanding Sources of LVDD ReclassificationThere are several possible explanations (physiological and algorithmic) for thisreclassification pattern:145(a)(b)Figure 5.10: Example pulsed-wave Doppler on the mitral valve and extract-ing for calculating E\/A ratio. Doppler imaging is a very noisy modal-ity.\u2022 Low VTR: Echo studies might not have been able to fulfill the VT R > 2.8 m\/scriterion due to:\u2013 insufficient TR; or\u2013 elevations in pulmonary pressures in severe diastolic dysfunction [3,10].\u2022 Low LAVi: the left atrial volume index could have been underestimated dueto:\u2013 left atrium segmentation inaccuracies; or\u2013 high body mass index [3].146\u2022 Low e\u2032: Mitral annular velocities e\u2032 might have been reduced, suggestive ofimpaired LV relaxation despite all other parameters being normal, in whichcase, the diastolic function would be considered normal based on the updatedGuidelines [97, 150]. This particularly applies to elderly patients, where aslight reduction in mitral annular velocities may be considered normal dueto aging considerations.Incorporating considerations in the image analysis solution to account for the abovepossible scenario will likely lead to more reliable automatic LVDD detection.147Chapter 6Conclusion and Future Work6.1 Thesis SummaryHeart disease is the leading cause of death globally [233]. Thanks to modernmedicine, several therapies and interventional procedures have been developed formanaging heart disease and improving health outcomes. Early diagnosis of heartdisease is critical for the effective delivery of therapy. Echocardiography (echo)is the most commonly used cardiac modality for assessing cardiac health and di-agnosing heart disease. Echo is harmless, accessible and affordable. Nonetheless,echo is low in signal-to-noise ratio and requires high proficiency for acquisition andinterpretation. This leads to backlogs and insufficient available echo services forthe patients in need. Computer-assisted diagnosis of heart disease in echo can helpincrease the throughput of the clinical services, easing the burden on the patientsand clinicians. With the emergence and ubiquity of machine learning (ML) toolsand libraries, data-driven methodologies are growingly popular for often automat-ing repetitive tasks in the diagnostic clinical workflow. Nonetheless, real-worldecho data is often very noisy and operator-dependent. This thesis investigated thefeasibility of using ML to derive function indices in echo based on labelled clinicaldata. We focused on function evaluation and quantifying the dysfunction sever-ity for systolic function, i.e. left ventricular ejection fraction (EF) and regionalwall motion abnormality (RWMA), as well as left ventricular diastolic dysfunction(LVDD). The main goal was to investigate the use of ML for detecting and grading148dysfunction using available data and annotations. Clinical labels and annotationsare also highly subjective, especially in the presence of disease, in aging and over-weight cohorts, etc. Furthermore, clinical annotations are expensive to acquire,limiting the scalability and extendibility of developing automated ML-based toolsto assist with screening and disease diagnosis.6.1.1 Summary of ContributionsThe contributions made in this thesis are listed below.\u2022 Chapter 2 presented the first successful attempt in literature for segmentation-free direct assessment of EF in echo. We used neural networks for A2C,A4C, and synchronized A2C+A4C views to detect high-risk patients at riskof heart failure with reduced EF (HFrEF) [25, 28]. We used spatio-temporalneural networks consisting of 2D convolutional neural networks (CNNs) forextracting spatial features from echo frames and recurrent neural networks(RNNs) to embed the sequence of extracted features temporally.\u2022 A novel multi-stream multi-tasking framework was proposed in Chapter 3,which used 3D CNNs to extract view-specific spatio-temporal features andmap embeddings to four clinical labels representing EF, i.e. biplane visual,A2C, A4C, and biplane Simpson\u2019s values. The novel pseudo-siamese archi-tecture connected the single or biplane view-specific labels to the appropriatestreams of inputs, ensuring the proper flow of gradients at training time. Thismodel was trained with a hybrid objective function to learn one classificationand three regression tasks based on A2C and A4C input videos.\u2022 Chapter 3 additionally presented an extension to the above network with anovel observer variability modelling framework. In this version, the clinicallabels, shown to have considerable disagreements in describing EF of thesame study, were modelled as a distribution rather than a point estimate. Weconsidered a Gaussian distribution with two independent variables, mean andsigma, that can be implicitly learned. A Gaussian probability distribution(PDF) function was used for regression branches with observer variabilityto optimize the model parameters rather than a mean-squared error. A cu-149mulative distribution function (CDF) was used for the classification branchto obtain likelihoods for the discrete classes. We showed that the variabilitymodelling improved the prediction results. Furthermore, reporting distribu-tions for computer-generated predictions is beneficial as it may reveal thereliability of the predictions. This was the first use of uncertainty modellingin EF estimation for performance enhancement.\u2022 A novel application of ML-predicted view-specific quality scores [136] wasproposed in Chapter 4 to investigate optimal diagnostic view. Over 84,000clinical cine loops belonging to a cohort with wall motion abnormalities anddetermining the optimal views for wall motion analysis were swiftly assessedusing a previously developed view and quality assessment model. In partic-ular, apical planes yielded the best quality, particularly apical four-chamber(A4C). Conversely, parasternal short axis (PSAX-M\/PM) views showed thepoorest quality.\u2022 Chapter 4 also presented a novel fully-automated framework to evaluatethe generalizability of LV segmentation models trained on standard popu-lations [102] when applied to diseased cohorts and visualize regional er-rors [29, 30]. We proposed a novel weak labelling method to localize theLV wall segments automatically. This was done by obtaining ML-predictedendocardial boundaries and predicting the segment locations based on cal-culations from the top two eigenvectors of the LV mask. We additionallyproposed segment-wise metrics (wall distance errors) to obtain regional er-rors at the location of 12 LV wall segments visible in A2C and A4C views.This framework enabled us to automatically visualize the prediction accu-racy, identify failure cases, and further study clinical labels for lab noise. Weobserved that endocardial segmentation is more likely to fail in the presenceof regional systolic dysfunction.\u2022 Chapter 5 presented the most extensive study focusing on the impacts ofupdated clinical guidelines on the detection of LVDD in the relevant liter-ature [241]. Using the most recent guidelines, a novel deep learning-basedmethod was proposed to learn the mapping between echo-based measure-150ments available in clinical databases and the guideline-derived LVDD la-bels [106]. We showed that the multi-layer perceptron could reliably learnthe clinical guidelines and demonstrated for the first time that ML couldbe used to model diastology. Furthermore, we proposed a modified modelto perform a regression task, producing continuous LVDD scores. This isthe first algorithm to obtain non-discrete labels in diastolic function, and ifadopted, it can likely reduce observer variability in LVDD scoring, whichis currently done according to rule-based algorithms with sharp decisionboundaries.6.2 Discussion and Future WorkWe conclude that ML can be used to assist clinicians with automating repetitivevisual analysis tasks (e.g. EF) to increase clinical throughput and interpretationof echocardiographic images (EF and RWMA) and redefining algorithmic cardiacindices (e.g., DD). Considerations and potential future work to unlock ML-assistedmeasurement and disease detection for echo videos are discussed below.6.2.1 Combatting Label NoiseTaking Ground Truth with a Grain of SaltThe ML methodology in this thesis involved supervised learning, where quantitiesare considered as ground-truth y and ML predictions y\u02c6. Nonetheless, ground truthcan be misleading, as even expert echocardiographers\u2019 readings and measurementsare highly variable. This is unlike in the general computer vision literature andsome other medical imaging modalities, where the gold standard value of a quan-tity is known (e.g. LV volume assessment in magnetic resonance (MR) images ofthe heart).The majority of this thesis focused on using ML to learn clinical annotations,despite observer variabilities: Chapter 2 used samples with full agreements of EF.Chapter 3 showed that the data noise could be harnessed with observer variabilitymodelling to obtain better accuracies and interpretability. Chapter 4 evaluated thecloseness of the ML predictions to the clinical labels. However, this paradigm151of replicating clinical labels were switched in Chapter 5, where neural networkswere used to obtain a continuous score for LVDD. This suggests there may be aninteresting opportunity for less obvious use cases of ML, where supervised learningcan help redefine ill-defined subjective quantities or improve their resolutions.Incorporating Additional ModalitiesAlthough the presence of RWMAs is relatively specific for detecting obstructiveCAD, it has low sensitivity at 67%, preventing it from becoming a mainstreammethod of CAD diagnosis despite the increasing accessibility of ultrasound [48,69, 181]. To alleviate the noisy label problem, additional imaging modalities orsources of information can be incorporated into the ML-based decision-makingframework, e.g., angiography, which is the standard of care for detecting arteryblockages and CAD diagnosis. Additionally, ML for analyzing electrocardiograms(ECG) has shown great promise in the past few years for complications such asarrhythmias, valvular disease, as well as low EF [17\u201319, 240]. Incorporating ECGsignals as an additional input modality to echo image analysis will likely benefitdiagnostic tools for functional assessment.Incorporating Patient History and Health OutcomesThe ultimate goal is improving patients\u2019 health outcomes; therefore, optimizingfor accurate outcome prediction may override or supersede visual measurementsacquired from echo. Our group has been working on retrieving such outcome datafrom Population Data (PopData) BC [41] for the patients in our databases whoare BC residents. We have recently obtained ethics and operational approvals toaccess this data and are awaiting data access. PopData contains major cardiacand life incidents, such as a record o prescribed medication, time and reason ofemergency visits, hospitalization and re-hospitalization, stroke, death, etc. Incor-poration Of health outcomes may be approached as multi-task learning of diseaseand outcome, direct detection of outcomes from images, and bypassing noisy mea-surements (Fig. 6.1).152Figure 6.1: Overview of a holistic ML-based framework for heart diagnosis,extending beyond the scope of this thesis (1.11). Next steps involveprediction of outcomes for patient Pi, based on measurements Y ti , andpredicted or diagnosed disease Dti , as well as intervention and therapyreceived history of medication or interventions recorded.6.2.2 More Data, Fewer Problems?In addition to being noisy, medical data are very unstructured. Large datasets mustbe curated to ensure enough data are supplied to the ML models. Practical chal-lenges such as data ethics approval periods, storage considerations, missing imageand label correspondences due to human errors, variable naming conventions atdifferent data centres or imaging devices, errors involved in the de-identificationand re-identification processes, etc. limit the amount of data that can be utilizedfor model training. Incorporating more extensive and diverse datasets in training153will benefit ML developments significantly.Fundamental ML vs. Applied ML DatasetsThe ML models used in the literature nowadays are those borrowed from thecurrently thriving ML and computer vision (CV) research communities (confer-ences such as the CVPR, ICML, ICLR, NeurIPS, etc.), which have establishedbenchmarked image datasets such as ImageNet [54], Common Objects in Context(COCO) [140], CIFAR [121], or video datasets such as UCF-100 [208], Avid [180],Kinetics [39], Moments in Time [151] for action recognition and event detection. Inaddition to having gone through crowd-sourced label validation and data cleaning,these datasets are several orders of magnitude larger than standard medical datasetsor those presented in this thesis. Table 6.1 summarizes dataset sizes in this the-sis, public echo datasets, and contrasts them with the state-of-the-art CV datasets.Acquiring such considerably larger datasets is beneficial to advance automated di-agnostics further and applied ML research. With sufficiently large datasets, futurework will involve the use of state-of-the-art image (vision) and video transform-ers and attention models [15, 64, 84, 249, 250]. These models have accomplishedoptimal performance on public datasets and provide interpretable spatio-temporalattention maps that will allow the user to verify where the model is looking formore confidence in the model\u2019s diagnostic capabilities.How Much Training Data Is Enough?The computer vision community has adopted the arbitrary rule of 1000 samplesper class, inspired by the original ImageNet challenge [54], which contained 1000samples per class. Based on this rule, 4,000 samples can be used to train a four-wayclassification model involved in, e.g. EF, RWMA, or DD dysfunction. Nonethe-less, estimating the amount of data required for training ML models for complexproblems like medical imaging and diagnosis is non-trivial, as it depends on manyparameters, including:\u2022 the number of input features, i.e. image dimensions;\u2022 the number of model parameters being optimized;154Datasets Data and Annotations CountChapter 2 Paired A2C and A4C echo videos withEF labels1,186Chapter 3 Paired A2C and A4c echo videos withmultiple EF labels2,181Chapter 4 Paired A2C, A3C and A4c echo videoswith EF and WMA labels2,910Chapter 5 Echo measurements and LVDD labels 7,728CAMUS [131] A4C and A4C echo videos with multi-chamber myocardial segmentation500HMC-QU [96] A2C or A4C still echo images withwall annotations2,736EchoNet Dy-namic [172]A4C echo cines with LVEF labels 10,000+UCF-101 [208] Human action video clips 13,000+Avid [180] Anonymous videos from diverse coun-tries450,000+Kinetics-700 [39] Human action video clips 650,000+Moments inTime [151]Action and event video clips 830,000+CIFAR-10 [121] Natural images 60,000+MNIST [55] Handwritten digit images 70,000+COCO [140] Common object images 120,000+ImageNet [54] Natural images 14 million+Table 6.1: Summary of datasets used in this thesis compared to publicdatasets of echo and natural image and video datasets. The public nat-ural image datasets are several orders of magnitude larger than the echodatasets. Larger echo training sets will likely enable further ML modeldevelopment for echo-based image analysis and disease detection.\u2022 the amount of noise in the input features;\u2022 the amount of noise in the output labels, the number of output classes, thecoarseness of labels, etc.;\u2022 the algorithm, network architecture, training regimen, etc.;\u2022 complexity of the ML task, the imaged anatomy, the imaging modality, etc.155In statistics, many works focus on sample-size determination methodologies (SS-DMs). These methods can be divided into pre-hoc (model-based) and post-hoc(curve-fitting) categories [21]. Pre-hoc approaches focus on the algorithm char-acteristics, acceptable classification error when generalized and error confidence.A pre-hoc method has been previously proposed for a simple single-layer feed-forward network with k units, d weights and expected \u03b5 error [22, 91]. The mini-mum number of training images N was then proposed as [91]:N \u2265O(d\u03b5log2( k\u03b5 )) (6.1)Another method [85] suggested estimating the training sample size as follows:N =O(d+k\u03b5) (6.2)which in practice leads to [22, 85]:N \u2248 (d\u03b5) (6.3)assuming training and test distributions are similar. Pre-hoc approaches cannotcapture the intricacies involved in bias-variance tuning in ML tasks [21]. Further-more, such methods are not robust with high dimensionality and tasks with highintraclass variability [62], which are common in medical imaging [21], especiallyecho. Post-hoc approaches, on the other hand, focus on empirically evaluating themodel performance (e.g. overall classification accuracy or area under the curve) asa function of training sample size [21]. These methods focus on modelling the rela-tionship between the observed performance and training sample size. They do notmake assumptions about the data distribution; however, they are criticized basedon the additional required data, resources, and experimentation [21].To our knowledge, model-based SSDMs have not been explored in CNN con-texts. However, assuming the above estimation rules extrapolate to the medicalimage domain, we can estimate the number of training samples for echo-basedfunction assessment.For simplicity, let us assume each task to be a classification,with the maximum acceptable error \u03b5 , ultimate accuracy of 1\u2212 \u03b5 on unseen test156data. Considering a safety factor of 2, we aim for \u03b5 = 14C where C is the number ofoutput classes. This leads to acceptable 75% accuracy on unseen data for binaryclassification (C = 2), involved in simple LV segmentation presented in Chapter 4.Input features k can be described ask = x \u22c5y \u22c5c \u22c5V (6.4)where x and y dimensions represent the echo image dimensions, V is the numberof input views (A2C, A4C, etc.) and c is the number of channels; c = 1 for B-modeand c = 3 for Doppler imaging, which entails color data. We assume downsized128\u00d7128 images without a preserved aspect ratio as in Chapter 4, with 123k inputparameters [102]. This yields N = 17M and N = 1.1M according to Equations 6.1and 6.2 respectively. As the pre-hoc SSDM rules were derived for a single-layerneural network, these estimates represent the upper bound of the training samplesize. Empirically, we know the models presented in Chapter 4 for LV segmentationwere successfully trained within acceptable error bounds on approximately 8,000still images. If we assume the 1.1M \u2236 8k factor, we can obtain an estimate for theremainder of ML tasks with a factor of 150 (\u2248 2.2 order of magnitude difference).It follows that a multi-chamber segmentation useful for diastolic dysfunction (C = 5for four chambers and background) would require 20k training samples. For the di-rect video analysis task presented in Chapter 3 for EF estimation, a spatio-temporalmodel with 35M parameters was successfully trained on approximately 1,750 B-mode echo pairs (c = 1, V = 2) of dimensions 128\u00d7128, i.e. approximately 7.5Mper input video stream, as proposed by the original 3D convolutional (C3D) net-work [220]. For a four-way classification of RWMA labels (C = 4), Equations 6.2and 6.4 yield required N = 150B. Normalizing the ratio between samples neededfor C3D vs. a single-layer neural network (1,750 \u2236 150B) yields at least N = 2,630samples with all three views (A2C, A3C, A4C) available. Similarly, if we assumediastolic function assessment involves four-way classification based on six inputstreams, including Doppler imaging (V = 6, c = 3), a minimum of N \u2248 16k train-ing samples may be needed. We can similarly use these equations to predict howadding new samples to the dataset can improve the accuracy. From Equation 6.1,157the error can be modelled as:\u03b5 \u2248 d+kN(6.5)which describes a linear relationship between \u03b5 and sample size N. Based on therelationship in Equation 6.2:\u03b5 \u2248 dNlog2( k\u03b5 ) (6.6)If W(.) denotes the Lambert-W function [50], \u03b5 can be solved for as:\u03b5 = d \u22c5W( k\u22c5N\u22c5log(2)d )n \u22c5 log(2) (6.7)Let \u03b51 and \u03b52 denote the errors associated with N1 and N2 training samples, respec-tively. The error enhancement can be estimated as:\u03b52\u03b51= N1N2\u22c5W( k\u22c5N2log(2)d )W( k\u22c5N1log(2)d ) (6.8)According to this relationship, instance doubling the training data N2 = 2\u00d7N1,where the existing error is \u03b51 = 8.6% (as presented in Chapter 3 with 91.4% ac-curacy for a 35M parameter model) is expected to lead to \u03b52 = 4.9%. A ten-foldincrease yields to \u03b52 = 1.8%. While useful, the estimation above remains na\u0131\u00a8ve anddoes not account for input and label noise, disease prevalence, and spatial correla-tion of input features, all of which compound the problem\u2019s complexity. Additionaldata is needed to perform curve-fitting on the echo analysis problems to better un-derstand the relationship between the training sample size and model performance.Establishing Echo Data PipelinesIncreasing the size and diversity of the training and test set for model training andclinical validation is critical to harnessing the power of state-of-the-art image un-derstanding and learning algorithms. Nonetheless, retrieving data from clinicalPACS is non-trivial as it runs the risk of overloading the hospital servers and healthauthority staff needed for data query and downloads supervision, who do so offthe side of their desk (sequence diagrams in B). Eliminating the data access bot-tleneck and establishing ethics-compliant streamlined data pipelines is a complex158multidisciplinary problem that may unlock ML-driven echo-based diagnostics. Fu-ture work involves establishing data pipelines for a secure and efficient transfer of1M echo data to sites and machines appropriate for model training. Our groupis actively consulting with information technology engineers and staff at UBCand regional health authorities (Vancouver Coastal Health (VCH) and ProvidenceHealthcare (PHC)) to design and implement a solution to create safe and compli-ant large-scale data pipelines and repositories to unlock echo diagnostics researchwith big data. Establishing such a pipeline will be considered a breakthrough andin addition to researchers, it requires multidisciplinary collaboration from severalgroups of stakeholders. Furthermore, this requires substantial resources for build-ing and sustaining the digital infrastructure for retrieving, anonymizing, storingand archiving the data.6.2.3 Towards a Holistic ML-based Heart Disease DiagnosisFrameworkThis thesis focused on systolic and diastolic function assessment. Ongoing re-search involving automatic extraction of measurements or disease detection in-cludes:\u2022 uncertainty modelling for EF estimation [104, 112, 113], landmark detectionfor diameter measurements in parasternal long-axis (PLAX) view [76].\u2022 direct spatio-temporal aortic stenosis assessment in echo [75];\u2022 rhythm analysis for atrial fibrillation detection [60];This is in addition to research involving auxiliary tasks or modules (Fig. 1.13)such as view synchronization [61], view classification with uncertainty estimationwithout sampling [80].Future work involves creating a unified and extendible framework for interpret-ing echo and deriving diagnostic information based on supervised learning withneural networks. By an extendible framework, we refer to one that requires onlyproviding the data associated with the task and retraining the model to obtain thetask-specific set of parameters. To this end, future work may include:159\u2022 developing an auto-encoder model trained on the entire available datasets toavoid training from scratch or potentially preventing overfitting to cohorts;\u2022 left atrium and multi-chamber segmentation and quantification; LA quantifi-cation is needed for detection of LVDD;\u2022 Doppler image analysis to measure blood flow velocities required for LVDD.160Index Views Modality ConsiderationsAorta AC3, A5C, PSAX-A, PLAX, SUP B-Mode, DopplerAortic Prosthesis AC3, A5C, PSAX-A, PLAX, SUP B-Mode, DopplerAortic Regurgitation AC3, A5C, PSAX-A, PLAX, SUP B-Mode, DopplerAortic Valve Function AC3, A5C, PSAX-A, PLAX, SUP B-Mode, DopplerAortic Valve Stenosis Severity AC3, A5C, PSAX-A, PLAX B-ModeAortic Valve Structure AC3, A5C, PSAX-A, PLAX B-ModeBicuspid Aortic Valve (BAV) AC3, A5C, PSAX-A, PLAX B-Mode, Doppler(Diastolic) Filling A2C, A4C B-mode, Doppler LA volume required(Diastolic) Filling Pressure A2C, A4C B-mode, Doppler LA volume requiredHypertrophy PLAX B-mode LV mass and relative wall thick-nessLVEF A2X, A4C, (PLAX) B-mode Most commonMitral Annular Calcification (MAC) A2C, A3C, A4C, PLAX, PSAX-m, B-modeMitral Prosthesis A2C-A5C, PLAX, PSAX B-mode, DopplerMitral Regurgitation A2C-A5C, PLAX, PSAX B-mode, DopplerMitral Valve Function A2C-A5C, PLAX, PSAX B-mode, DopplerMitral Valve Stenosis Severity A2C-A5C, PLAX, PSAX B-mode, DopplerMitral Valve Structure A2C-A5C, PLAX, PSAX B-mode, DopplerPericardial Effusion All B-mode Fluid (blood) around the heartPulmonary Regurgitation PSAX-a B-mode, DopplerRhythm - ECGRV Function PLAX, PSAX-m\/pm, RV-in, A4C,A5C, SUB4,B-modeRV Structure SUB4, A4C, PSAX-a, PLAX B-modeTricuspid Prosthesis RV-in, PSAX-a, A4C, SUB4 B-mode, DopplerTricuspid Regurgitation RV-in, PSAX-a, A4C, SUB4 B-mode, DopplerTricuspid Valve function RV-in, PSAX-a, A4C, SUB4 B-modeTricuspid Valve Structure RV-in, PSAX-a, A4C, SUB4 B-mode, DopplerWall Motion A2C-A5C, PSAX B-modeTable 6.2: Categorical clinical labels in DFileMaker. FileMaker interface snapshots are available in Figs. C.1-C.4.161Table 6.2 lists the categorical cardiac indices available in our clinical dataset(FileMaker) and the cardiac views and modalities required to derive the indices.In addition, snapshots of the FileMaker software interface snapshots are providedinC (Figs. C.1-C.4). Based on experiments performed for this thesis, the followingrecommendations are made for developing ML-based echo analysis tools:\u2022 Before attempting to develop ML models from scratch, study the clinicalproblem, underlying phenomena, and visual features, especially for morecomplicated conditions.\u2022 Where applicable, incorporate clinical knowledge and priors.\u2022 Use architectures with multi-view spatio-temporal analysis, where applica-ble. Multi-plane analyses are computationally expensive but contain richerheart structure and function information.\u2022 Opt for multi-task learning frameworks for detecting conditions, where ap-plicable. For example, allocating output branches for simple measurementssuch as EF may regularize the model and benefit the learning.Limitations of StudiesIn this thesis, we studied the use of ML for echo-derived measurements to eval-uate the heart\u2019s function and investigated model biases concerning subjects\u2019 agegroups, sexes, and body build. We did not investigate or incorporate aspects ex-ternal to the echocardiographic images, such as clinical interventions, i.e. stentplacements, valve replacements and other operations, medications such as bloodthinners, lifestyle factors and changes. The data used in all studies originated fromcart-based ultrasound machines at five VCH hospitals, which serve the Vancou-ver, BC community. The ethnicity demographics of the data are unavailable dueto ethical and privacy reasons; nonetheless, the demographics can be assumed tofollow the Greater Vancouver statistics, i.e. primarily Caucasian (31%), with thedominant visible minorities with Chinese (17%), South and South-east Asian her-itage [37]. Echo data from the indigenous population is hence underrepresented(approximately 2% of the Greater Vancouver population) [24]. Additionally, no ru-ral and point-of-care ultrasound (POCUS) datasets were utilized in this thesis. All162the data used in this research comes from adult cohorts. Hence, pediatric echocar-diography remains another area to be explored for ML-driven diagnostics. Addi-tionally, in most of the studies presented in this thesis, congenital heart disease wasexcluded as they affect the overall baseline of cardiac health and exert different be-haviour in echo cine loops. Future work may include addressing congenital heartdisease.163Bibliography[1] Clinical Echocardiography (e-book). \u2192 pages xxvii, 127[2] A. H. Abdi, C. Luong, T. Tsang, J. Jue, K. Gin, D. Yeung, D. Hawley,R. Rohling, and P. Abolmaesumi. Quality assessment of echocardiographiccine using recurrent neural networks: Feasibility on five standard viewplanes. In International Conference on Medical Image Computing andComputer-Assisted Intervention, pages 302\u2013310. Springer, 2017. \u2192 pages49, 80[3] E. Abergel, S. Lafitte, and N. Mansencal. Evaluation of left ventricularfilling pressure: Updated recommendations lack new evidence and havesevere interpretation issues. Archives of Cardiovascular Diseases, 111(12):707\u2013711, 2018. \u2192 pages 129, 146[4] W. P. Abhayaratna, T. H. Marwick, W. T. Smith, and N. G. Becker.Characteristics of left ventricular diastolic dysfunction in the community:an echocardiographic survey. Heart, 92(9):1259\u20131264, 2006. \u2192 pages123, 144[5] P. Abolmaesumi, Z. Liao, T. Tang, and D. Behnami. Neural network imageanalysis. https:\/\/patents.google.com\/patent\/US20210365786A1, August2021. \u2192 page viii[6] M. Afshin, I. B. Ayed, K. Punithakumar, M. Law, A. Islam, A. Goela,T. Peters, and S. Li. Regional assessment of cardiac left ventricularmyocardial function via MRI statistical features. IEEE Transactions onMedical Imaging, 33(2):481\u2013494, 2014. \u2192 page 19[7] A. Afsin Oktay and S. J Shah. Diagnosis and management of heart failurewith preserved ejection fraction: 10 key lessons. Current CardiologyReviews, 11(1):42\u201352, 2015. \u2192 page 122164[8] M. Alessandrini, B. Heyde, S. Queiro\u00b4s, S. Cygan, M. Zontak,O. Somphone, O. Bernard, M. Sermesant, H. Delingette, D. Barbosa, et al.Detailed evaluation of five 3D speckle tracking algorithms using syntheticechocardiographic recordings. IEEE Transactions on Medical Imaging, 35(8):1915\u20131926, 2016. \u2192 page 16[9] W. AlJaroudi, M. C. Alraies, C. Halley, L. Rodriguez, R. A. Grimm, J. D.Thomas, and W. A. Jaber. Impact of progression of diastolic dysfunctionon mortality in patients with normal ejection fraction. Circulation, 125(6):782\u2013788, 2012. \u2192 page 122[10] J. G. Almeida, R. Fontes-Carvalho, F. Sampaio, J. Ribeiro, P. Bettencourt,F. A. Flachskampf, A. Leite-Moreira, and A. Azevedo. Impact of the 2016ASE\/EACVI recommendations on the prevalence of diastolic dysfunctionin the general population. European Heart Journal - CardiovascularImaging, 19(4):380\u2013386, 2018. \u2192 pages 129, 144, 145, 146[11] O. S. Andersen, O. A. Smiseth, H. Dokainish, M. M. Abudiab, R. C.Schutt, A. Kumar, K. Sato, S. Harb, E. Gude, E. W. Remme, et al.Estimating left ventricular filling pressure by echocardiography. Journal ofthe American College of Cardiology, 69(15):1937\u20131948, 2017. \u2192 page 129[12] E. M. Antman. Time is muscle: translation into practice. Journal of theAmerican College of Cardiology, 52(15):1216\u20131221, 2008. \u2192 page 68[13] Apple Inc. Filemaker Pro 7 data sheet. https:\/\/store.apple.com\/Catalog\/US\/Images\/F100CM02U.pdf?msclkid=6753de49c17511ec9f6eb88712d93fdb,2022. Accessed: 2022-04-18. \u2192 page 202[14] C. P. Appleton. Doppler assessment of left ventricular diastolic function:refinements continue. Journal of the American College of Cardiology, 21(7):1697\u20131700, 1993. \u2192 page 144[15] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Luc\u02c7ic\u00b4, and C. Schmid.Vivit: A video vision transformer. In Proceedings of the IEEE\/CVFInternational Conference on Computer Vision, pages 6836\u20136846, 2021. \u2192page 154[16] P. Asgharzadeh. A deep learning framework for wall motion abnormalitydetection in echocardiograms.https:\/\/open.library.ubc.ca\/collections\/ubctheses\/24\/items\/1.0388865,2020. \u2192 pages xx, 29, 85, 97, 98, 195165[17] Z. I. Attia, S. Kapa, F. Lopez-Jimenez, P. M. McKie, D. J. Ladewig,G. Satam, P. A. Pellikka, M. Enriquez-Sarano, P. A. Noseworthy, T. M.Munger, et al. Screening for cardiac contractile dysfunction using anartificial intelligence\u2013enabled electrocardiogram. Nature Medicine, 25(1):70\u201374, 2019. \u2192 page 152[18] Z. I. Attia, S. Kapa, P. A. Noseworthy, F. Lopez-Jimenez, and P. A.Friedman. Artificial intelligence ecg to detect left ventricular dysfunctionin covid-19: a case series. In Mayo Clinic Proceedings, volume 95, pages2464\u20132466. Elsevier, 2020.[19] Z. I. Attia, D. M. Harmon, E. R. Behr, and P. A. Friedman. Application ofartificial intelligence to the electrocardiogram. European Heart Journal, 42(46):4717\u20134730, 2021. \u2192 page 152[20] B. Balaney, D. Medvedofsky, A. Mediratta, A. Singh, B. Ciszek, E. Kruse,A. P. Shah, K. Addetia, R. M. Lang, and V. Mor-Avi. Invasive validation ofthe echocardiographic assessment of left ventricular filling pressures usingthe 2016 diastolic guidelines: head-to-head comparison with the 2009guidelines. Journal of the American Society of Echocardiography, 31(1):79\u201388, 2018. \u2192 pages 129, 145[21] I. Balki, A. Amirabadi, J. Levman, A. L. Martel, Z. Emersic, B. Meden,A. Garcia-Pedrero, S. C. Ramirez, D. Kong, A. R. Moody, et al.Sample-size determination methodologies for machine learning in medicalimaging research: a systematic review. Canadian Association ofRadiologists Journal, 70(4):344\u2013353, 2019. \u2192 page 156[22] E. Baum and D. Haussler. What size net gives valid generalization?Advances in Neural Information Processing Systems, 1, 1988. \u2192 page 156[23] C. F. Baumgartner, L. M. Koch, M. Pollefeys, and E. Konukoglu. Anexploration of 2D and 3D deep learning techniques for cardiac MR imagesegmentation. arXiv preprint arXiv:1709.04496, 2017. \u2192 page 18[24] BC Campus. Indigenization guide: Urban indigenous peoples anddemographics. https:\/\/bccampus.ca\/2019\/12\/06\/indigenization-guide-urban-indigenous-peoples-and-demographics\/,2019. \u2192 page 162[25] D. Behnami, C. Luong, H. Vaseli, A. Abdi, H. Girgis, D. Hawley,R. Rohling, K. Gin, P. Abolmaesumi, and T. Tsang. Automatic detection ofpatients with a high risk of systolic cardiac failure in echocardiography. In166Deep Learning in Medical Image Analysis and Multimodal Learning forClinical Decision Support, pages 65\u201373. Springer, 2018. \u2192 pagesvi, vii, 26, 66, 67, 72, 114, 149[26] D. Behnami, H. Y. A. Girgis, C. Luong, D. Hawley, R. Rohling, K. Gin,P. Abolmaesumi, and T. Tsang. Artificial intelligence for visual assessmentof left ventricular systolic function in patients with a wide range of ejectionfraction. Journal of the American Society of Echocardiography, 32:121\u2013122, 06 2019. \u2192 page vii[27] D. Behnami, Z. Liao, H. Girgis, C. Luong, R. Rohling, K. Gin, T. Tsang,and P. Abolmaesumi. Dual-view joint estimation of left ventricular ejectionfraction with uncertainty modelling in echocardiograms. In InternationalConference on Medical Image Computing and Computer-AssistedIntervention, pages 696\u2013704. Springer, 2019. \u2192 pages vii, viii, 67, 72[28] D. Behnami, C. Luong, H. Vaseli, H. Girgis, A. Abdi, D. Hawley, K. Gin,R. Rohling, P. Abolmaesumi, and T. Tsang. Automatic cine-baseddetection of patients at high risk of heart failure with reduced ejectionfraction in echocardiograms. Computer Methods in Biomechanics andBiomedical Engineering: Imaging & Visualization, 8(5):502\u2013508, 2020. \u2192pages vii, 26, 72, 114, 149[29] D. Behnami, C. Luong, M. Jafari, N. Van Woudenberg, D. Hawley,R. Rohling, P. Abolmaesumi, and T. Tsang. Can AI-driven LV delineationwithstand regional wall motion abnormalities in TTE echo. Journal of theAmerican Society of Echocardiography (in press), June 2022. \u2192 pagesviii, 150[30] D. Behnami, C. Luong, M. Jafari, N. Van Woudenberg, R. Rohling,T. Tsang, and P. Abolmaesumi. Generalizability of disease-agnosticML-based LV segmentation on a cohort with regional wall motionabnormality. Rebuttal Stage at MICCAI, May 2022. \u2192 page 150[31] E. J. Benjamin, P. Muntner, A. Alonso, M. S. Bittencourt, C. W. Callaway,A. P. Carson, A. M. Chamberlain, A. R. Chang, S. Cheng, S. R. Das, et al.Heart disease and stroke statistics - 2019 update: a report from theAmerican Heart Association. Circulation, 139(10):e56\u2013e528, 2019. \u2192page 3[32] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P.-A. Heng,I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester, et al. Deep learning167techniques for automatic MRI cardiac multi-structures segmentation anddiagnosis: Is the problem solved? IEEE Transactions on Medical Imaging,2018. \u2192 page 18[33] P. Bresser, J. De Beer, and Y. De Wet. A study investigating variability ofleft ventricular ejection fraction using manual and automatic processingmodes in a single setting. Radiography, 21(1):e41\u2013e44, 2015. \u2192 pages14, 19, 31[34] I. G. Burwash, A. Basmadjian, D. Bewick, J. B. Choy, B. Cujec, D. S.Jassal, S. MacKenzie, P. Nair, L. G. Rudski, E. Yu, et al. 2010 CanadianCardiovascular Society\/Canadian Society of Echocardiography guidelinesfor training and maintenance of competency in adult echocardiography.Canadian Journal of Cardiology, 27(6):862\u2013864, 2011. \u2192 page 69[35] I. Bytyc\u00b8i and G. Bajraktari. Mortality in heart failure patients. AnatolianJournal of Cardiology, 15(1):63, 2015. \u2192 page 4[36] M. Cameli, S. Mondillo, M. Solari, F. M. Righini, V. Andrei, C. Contaldi,E. De Marco, M. Di Mauro, R. Esposito, S. Gallina, et al.Echocardiographic assessment of left ventricular systolic function: fromejection fraction to torsion. Heart Failure Reviews, 21(1):77\u201394, 2016. \u2192pages 8, 14, 30, 31, 32[37] Canada Population. Vancouver population 2021\/2022.https:\/\/canadapopulation.org\/vancouver-population\/, 2022. \u2192 page 162[38] G. Carneiro and J. C. Nascimento. Combining multiple dynamic modelsand deep learning architectures for tracking the left ventricle endocardiumin ultrasound data. IEEE Transactions on Pattern Analysis and MachineIntelligence, 35(11):2592\u20132607, 2013. \u2192 page 31[39] J. Carreira, E. Noland, C. Hillier, and A. Zisserman. A short note on thekinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019.\u2192 pages 154, 155[40] Centers for Disease Control and Prevention. Heart disease facts.https:\/\/www.cdc.gov\/heartdisease\/facts.htm, 2022. Accessed: 2022-03-13.\u2192 pages 4, 68[41] Centers for Disease Control and Prevention. Population Data (PopData)BC. https:\/\/www.popdata.bc.ca, 2022. Accessed: 2022-04-13. \u2192 page 152168[42] M. M. Chan and C. S. Lam. How do patients with heart failure withpreserved ejection fraction die? European Journal of Heart Failure, 15(6):604\u2013613, 2013. \u2192 page 123[43] C. B. Chapman, S. M. Ewer, A. F. Kelly, K. M. Jacobson, M. A. Leal, andP. S. Rahko. Classification of left ventricular diastolic function usingAmerican Society of Echocardiography guidelines: agreement amongechocardiographers. Echocardiography, 30(9):1022\u20131031, 2013. \u2192 page123[44] Z. Chen, F. Contijoch, and E. McVeigh. Development of deep learningpipeline for direct observation of wall motion abnormality from 4DCT. InMedical Imaging 2022: Biomedical Applications in Molecular, Structural,and Functional Imaging, volume 12036, pages 429\u2013439. SPIE, 2022. \u2192page 73[45] F. Chollet et al. Keras. https:\/\/github.com\/keras-team\/keras, 2015. \u2192 page44[46] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation ofgated recurrent neural networks on sequence modeling. arXiv preprintarXiv:1412.3555, 2014. \u2192 page 43[47] G. D. Cole, N. M. Dhutia, M. J. Shun-Shin, K. Willson, J. Harrison, C. E.Raphael, M. Zolgharni, J. Mayet, and D. P. Francis. Defining thereal-world reproducibility of visual grading of left ventricular function andvisual estimation of left ventricular ejection fraction: impact of imagequality, experience and accreditation. The International Journal ofCardiovascular Imaging, 31(7):1303\u20131314, 2015. \u2192 pages 14, 19, 31[48] S. M. Colquhoun, J. R. Carapetis, J. H. Kado, B. M. Reeves, B. Remenyi,W. May, N. J. Wilson, and A. C. Steer. Pilot study of nurse-led rheumaticheart disease echocardiography screening in Fiji - a novel approach in aresource-poor setting. Cardiology in the Young, 23(4):546\u2013552, 2013. \u2192pages 4, 152[49] C. B. Compas, E. Y. Wong, X. Huang, S. Sampath, B. A. Lin, P. Pal,X. Papademetris, K. Thiele, D. P. Dione, M. Stacy, et al. Radial basisfunctions for combining shape and speckle tracking in 4Dechocardiography. IEEE Transactions on Medical Imaging, 33(6):1275\u20131289, 2014. \u2192 page 18169[50] R. M. Corless, G. H. Gonnet, D. E. Hare, D. J. Jeffrey, and D. E. Knuth. Onthe lambertw function. Advances in Computational mathematics, 5(1):329\u2013359, 1996. \u2192 page 158[51] J. K. Dave, M. E. Mc Donald, P. Mehrotra, A. R. Kohut, J. R. Eisenbrey,and F. Forsberg. Recent technological advancements in cardiac ultrasoundimaging. Ultrasonics, 2017. \u2192 page 2[52] M. A. Degel, N. Navab, and S. Albarqouni. Domain and geometry agnosticCNNs for left atrium segmentation in 3D ultrasound. arXiv preprintarXiv:1805.00357, 2018. \u2192 page 18[53] A. Degerli, M. Zabihi, S. Kiranyaz, T. Hamid, R. Mazhar, R. Hamila, andM. Gabbouj. Early detection of myocardial infarction in low-qualityechocardiography. IEEE Access, 9:34442\u201334453, 2021. \u2192 page 73[54] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: Alarge-scale hierarchical image database. In 2009 IEEE Conference onComputer Vision and Pattern Recognition, pages 248\u2013255. Ieee, 2009. \u2192pages 43, 154, 155[55] L. Deng. The mnist database of handwritten digit images for machinelearning research. IEEE Signal Processing Magazine, 29(6):141\u2013142,2012. \u2192 page 155[56] R. C. Deo, J. Zhang, L. A. Hallock, et al. An end-to-end computer visionpipeline for automated cardiac function assessment by echocardiography.CoRR, 2017. URL http:\/\/arxiv.org\/abs\/1706.07342. \u2192 pages 18, 31[57] R. Dey and F. M. Salem. Gate-variants of gated recurrent unit (GRU)neural networks. In 2017 IEEE 60th International Midwest Symposium onCircuits and Systems (MWSCAS), pages 1597\u20131600. IEEE, 2017. \u2192 page83[58] F. T. Dezaki, N. Dhungel, A. H. Abdi, et al. Deep residual recurrent neuralnetworks for characterisation of cardiac cycle phase from echocardiograms.In Deep Learning in Medical Image Analysis, pages 100\u2013108. Springer,2017. \u2192 pages 25, 32, 38, 50[59] F. T. Dezaki, Z. Liao, C. Luong, H. Girgis, N. Dhungel, A. H. Abdi,D. Behnami, K. Gin, R. Rohling, P. Abolmaesumi, et al. Cardiac phasedetection in echocardiograms with densely gated recurrent neural networksand global extrema loss. IEEE Transactions on Medical Imaging, 38(8):1821\u20131832, 2018. \u2192 pages 25, 194170[60] F. T. Dezaki, T. Ginsberg, C. Luong, H. Vaseli, R. Rohling, K. Gin,P. Abolmaesumi, and T. Tsang. Echo-rhythm net: Semi-supervisedlearning for automatic detection of atrial fibrillation in echocardiography.In 2021 IEEE 18th International Symposium on Biomedical Imaging(ISBI), pages 110\u2013113. IEEE, 2021. \u2192 pages 50, 159[61] F. T. Dezaki, C. Luong, T. Ginsberg, R. Rohling, K. Gin, P. Abolmaesumi,and T. Tsang. Echo-syncnet: Self-supervised cardiac view synchronizationin echocardiography. IEEE Transactions on Medical Imaging, 40(8):2092\u20132104, 2021. \u2192 pages 25, 50, 159[62] K. K. Dobbin and R. M. Simon. Sample size planning for developingclassifiers using high-dimensional dna microarray data. Biostatistics, 8(1):101\u2013117, 2007. \u2192 page 156[63] S. Dong, G. Luo, G. Sun, et al. A left ventricular segmentation method on3D echocardiography using deep learning and snake. In Computing inCardiology Conference (CinC), 2016, pages 473\u2013476. IEEE, 2016. \u2192pages 18, 31[64] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. Animage is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. \u2192 page 154[65] X. Du, W. Zhang, H. Zhang, J. Chen, Y. Zhang, J. C. Warrington,G. Brahm, and S. Li. Deep regression segmentation for cardiac bi-ventricleMR images. IEEE Access, 6:3828\u20133838, 2018. \u2192 page 32[66] T. Edvardsen and O. A. Smiseth. Evaluation of diastolic function byechocardiography: important progression, but issues to be resolved.European Heart Journal-Cardiovascular Imaging, 2018. \u2192 page 129[67] J. Ehrhardt, M. Wilms, H. Handels, and D. Sa\u00a8ring. Automatic detection ofcardiac remodeling using global and local clinical measures and randomforest classification. In International Workshop on Statistical Atlases andComputational Models of the Heart, pages 199\u2013207. Springer, 2015. \u2192page 18[68] K. Ellenberger, P. Jeyaprakash, S. Sivapathan, S. Sangha, J. Kitley,A. Darshni, D. Chen, K. Negishi, and F. Pathan. The effect of obesity onechocardiographic image quality. Heart, Lung and Circulation, 31(2):207\u2013215, 2022. \u2192 pages 114, 115171[69] D. Engelman, J. H. Kado, B. Reme\u00b4nyi, S. M. Colquhoun, J. R. Carapetis,N. J. Wilson, S. Donath, and A. C. Steer. Screening for rheumatic heartdisease: quality and agreement of focused cardiac ultrasound by brieflytrained health workers. BMC Cardiovascular Disorders, 16(1):1\u20139, 2016.\u2192 pages 4, 152[70] T. A. Foley, S. V. Mankad, N. S. Anavekar, C. R. Bonnichsen, M. F.Morris, T. D. Miller, and P. A. Araoz. Measuring left ventricular ejectionfraction-techniques and potential pitfalls. European Cardiology, 8(2):108\u2013114, 2012. \u2192 pages 8, 55, 67[71] Food and Drug Administration. Class 2 device recall: GE Vscan Extend.https:\/\/www.accessdata.fda.gov\/scripts\/cdrh\/cfdocs\/cfRes\/res.cfm?ID=173162,2019. Accessed: 2022-02-28. \u2192 page 119[72] Food and Drug Administration. Class 2 device recall: Butterfly iQ. https:\/\/www.accessdata.fda.gov\/scripts\/cdrh\/cfdocs\/cfres\/res.cfm?id=180221,2020. Accessed: 2022-02-28. \u2192 page 119[73] A. G. Fraser and N. Girerd. Cardiovascular imaging \u201cguidelines\u201d\u2013hightime for a paradigm shift. International Journal of Cardiology, 329:246\u2013248, 2021. \u2192 pages 130, 144[74] P. Gifani, H. Behnam, A. Shalbaf, and Z. A. Sani. Automatic detection ofend-diastole and end-systole from echocardiography images using manifoldlearning. Physiological Measurement, 31(9):1091, 2010. \u2192 page 32[75] T. Ginsberg, R.-e. Tal, M. Tsang, C. Macdonald, F. T. Dezaki, J. v. d. Kuur,C. Luong, P. Abolmaesumi, and T. Tsang. Deep video networks forautomatic assessment of aortic stenosis in echocardiography. InInternational Workshop on Advances in Simplifying Medical Ultrasound,pages 202\u2013210. Springer, 2021. \u2192 page 159[76] J. A. D. Goco, M. H. Jafari, C. Luong, T. Tsang, and P. Abolmaesumi. Anefficient deep landmark detection network for plax ef estimation usingsparse annotations. In Medical Imaging 2022: Image-Guided Procedures,Robotic Interventions, and Modeling, volume 12034, pages 129\u2013134. SPIE,2022. \u2192 page 159[77] M. F. Gottbrecht, M. Salerno, and G. P. Aurigemma. Evolution of diastolicfunction algorithms: Implications for clinical practice. Echocardiography,35(1):39\u201346, 2018. \u2192 pages 129, 144, 145172[78] J. S. Gottdiener, S. V. Livengood, P. S. Meyer, and G. A. Chase. Shouldechocardiogfraphy be performed to assess effects of antihypertensivetherapy? Test-retest reliability of echocardiography for measurement of leftventricular mass and function. Journal of the American College ofCardiology, 25(2):424\u2013430, 1995. \u2192 page 131[79] A. D. Grant, K. Negishi, T. Negishi, P. Collier, S. R. Kapadia, J. D.Thomas, T. H. Marwick, B. P. Griffin, and Z. B. Popovic\u00b4. Grading diastolicfunction by echocardiography: hemodynamic validation of existingguidelines. Cardiovascular Ultrasound, 13(1):1\u20139, 2015. \u2192 page 123[80] A. N. Gu, C. Luong, M. H. Jafari, N. V. Woudenberg, H. Girgis,P. Abolmaesumi, and T. Tsang. Efficient echocardiogram viewclassification with sampling-free uncertainty estimation. In InternationalWorkshop on Advances in Simplifying Medical Ultrasound, pages 139\u2013148.Springer, 2021. \u2192 page 159[81] B. Gu, Y. Shan, V. S. Sheng, et al. Sparse regression with outputcorrelation for cardiac ejection fraction estimation. Information Sciences,423:303\u2013312, 2018. \u2192 pages 32, 56[82] P. Gudmundsson, E. Rydberg, R. Winter, et al. Visually estimated leftventricular ejection fraction by echocardiography is closely correlated withformal quantitative methods. International Journal of Cardiology, 101(2):209\u2013212, 2005. \u2192 pages 14, 32[83] O. Hamila, S. Ramanna, C. J. Henry, S. Kiranyaz, R. Hamila, R. Mazhar,and T. Hamid. Fully automated 2D and 3D convolutional neural networkspipeline for video segmentation and myocardial infarction detection inechocardiography. arXiv preprint arXiv:2103.14734, 2021. \u2192 page 73[84] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao,C. Xu, Y. Xu, et al. A survey on vision transformer. IEEE Transactions onPattern Analysis and Machine Intelligence, 2022. \u2192 page 154[85] S. Haykin and N. Network. A comprehensive foundation. NeuralNetworks, 2(2004):41, 2004. \u2192 page 156[86] X. He and A. M. Chiang. Wall motion analyzer, Dec. 2 2014. US Patent8,900,149. \u2192 page 19[87] Health CostHelper. How much does a CT scan cost?https:\/\/health.costhelper.com\/ct-scan.html, 2022. \u2192 page 7173[88] Health CostHelper. How much does a cardiac MRI cost?https:\/\/health.costhelper.com\/heart-mri.html, 2022. \u2192 page 7[89] Heart and Stroke Foundation. The Heart and Stroke Foundation 2016report on the health of Canadians: Burden of heart failure.https:\/\/www.heartandstroke.ca\/-\/media\/pdf-files\/canada\/2017-heart-month\/heartandstroke-reportonhealth-2016.ashx?la=en,2017. \u2192 page 3[90] K. O. Hensel, M. Roskopf, L. Wilke, and A. Heusch. Intraobserver andinterobserver reproducibility of m-mode and b-mode acquired mitralannular plane systolic excursion (mapse) and its dependency onechocardiographic image quality in children. PloS one, 13(5):e0196614,2018. \u2192 page 115[91] W. Hitzl, H. Reitsamer, K. Hornykewycz, A. Mistlberger, and G. Grabner.Application of discriminant, classification tree and neural network analysisto differentiate between potential glaucoma suspects with and withoutvisual field defects. Journal of Theoretical Medicine, 5(3-4):161\u2013170,2003. \u2192 page 156[92] S. Hochreiter and J. Schmidhuber. Long short-term memory. NeuralComputation, 9(8):1735\u20131780, 1997. \u2192 pages 79, 83[93] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforwardnetworks are universal approximators. Neural networks, 2(5):359\u2013366,1989. \u2192 page 143[94] G. Huang, Z. Liu, K. Q. Weinberger, et al. Densely connectedconvolutional networks. In IEEE Conference on Computer Vision andPattern Recognition (CVPR), 2017. \u2192 pages xxiii, 42, 48, 79, 83[95] M.-S. Huang, C.-S. Wang, J.-H. Chiang, P.-Y. Liu, and W.-C. Tsai.Automated recognition of regional wall motion abnormalities through deepneural network interpretation of transthoracic echocardiography.Circulation, 142(16):1510\u20131520, 2020. \u2192 pages 72, 73[96] M.-S. Huang et al. Automated recognition of regional wall motionabnormalities through deep neural network interpretation of transthoracicechocardiography. Circulation, 142(16):1510\u20131520, 2020. \u2192 pages119, 155174[97] O. Huttin, A. G. Fraser, S. Coiro, E. Bozec, C. Selton-Suty, Z. Lamiral,Z. Frikha, P. Rossignol, F. Zannad, and N. Girerd. Impact of changes inconsensus diagnostic recommendations on the echocardiographicprevalence of diastolic dysfunction. Journal of the American college ofcardiology, 69(25):3119\u20133121, 2017. \u2192 pages 144, 145, 147[98] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep networktraining by reducing internal covariate shift. In International Conferenceon Machine Learning, pages 448\u2013456. PMLR, 2015. \u2192 page 79[99] F. Isensee, P. Jaeger, P. M. Full, I. Wolf, S. Engelhardt, and K. H.Maier-Hein. Automatic cardiac disease assessment on cine-MRI viatime-series segmentation and domain specific features. arXiv preprintarXiv:1707.00587, 2017. \u2192 page 31[100] M. H. Jafari. Towards a more robust machine learning framework forcomputer-assisted echocardiography. PhD thesis, University of BritishColumbia, 2021. \u2192 pages xx, 25, 29, 145[101] M. H. Jafari, H. Girgis, Z. Liao, D. Behnami, A. Abdi, H. Vaseli, C. Luong,R. Rohling, K. Gin, T. Tsang, et al. A unified framework integratingrecurrent fully-convolutional networks and optical flow for segmentation ofthe left ventricle in echocardiography data. In Deep Learning in MedicalImage Analysis and Multimodal Learning for Clinical Decision Support,pages 29\u201337. Springer, 2018. \u2192 pages 18, 31, 55, 72, 194[102] M. H. Jafari, H. Girgis, N. Van Woudenberg, Z. Liao, R. Rohling, K. Gin,P. Abolmaesumi, and T. Tsang. Automatic biplane left ventricular ejectionfraction estimation with mobile point-of-care ultrasound using multi-tasklearning and adversarial training. International Journal of ComputerAssisted Radiology and Surgery, 14(6):1027\u20131037, 2019. \u2192 pagesxxiii, 72, 75, 86, 87, 150, 157[103] M. H. Jafari, C. Luong, M. Tsang, A. N. Gu, N. Van Woudenberg,R. Rohling, T. Tsang, and P. Abolmaesumi. U-land: Uncertainty-drivenvideo landmark detection. IEEE Transactions on Medical Imaging, 2021.\u2192 pages 67, 72[104] M. H. Jafari, N. Van Woudenberg, C. Luong, P. Abolmaesumi, andT. Tsang. Deep bayesian image segmentation for a more robust ejectionfraction estimation. In 2021 IEEE 18th International Symposium onBiomedical Imaging (ISBI), pages 1264\u20131268. IEEE, 2021. \u2192 page 159175[105] M. H. Jafari, N. Van Woudenberg, C. Luong, P. Abolmaesumi, andT. Tsang. Deep bayesian image segmentation for a more robust ejectionfraction estimation. In 2021 IEEE 18th International Symposium onBiomedical Imaging (ISBI), pages 1264\u20131268. IEEE, 2021. \u2192 pages 67, 72[106] R. Jiang*, D. Yeung*, D. Behnami*, J. Jue, M. Tsang, K. Gin, C. Luong,P. Nair, H. Girgis, P. Abolmaesumi, et al. Machine learning to facilitateassessment of diastolic function by echocardiography. Canadian Journal ofCardiology, 35(10):S4\u2013S5, 2019. \u2192 pages ix, 151[107] K. W. Johnson, J. T. Soto, B. S. Glicksberg, K. Shameer, R. Miotto, M. Ali,E. Ashley, and J. T. Dudley. Artificial intelligence in cardiology. Journal ofthe American College of Cardiology, 71(23):2668\u20132679, 2018. \u2192 page 20[108] T. Joyce, A. Chartsias, and S. A. Tsaftaris. Deep multi-class segmentationwithout ground-truth labels. Open Review, 2018. \u2192 page 18[109] A. Kabani and M. R. El-Sakka. Estimating ejection fraction and leftventricle volume using deep convolutional networks. In Image Analysisand Recognition, pages 678\u2013686. Springer, 2016. \u2192 page 32[110] A. Kabani and M. R. El-Sakka. Ejection fraction estimation using a wideconvolutional neural network. In Image Analysis and Recognition, pages87\u201396. Springer, 2017. \u2192 pages 32, 52, 56, 66[111] K. Kalam, P. Otahal, and T. H. Marwick. Prognostic implications of globalLV dysfunction: a systematic review and meta-analysis of globallongitudinal strain and ejection fraction. Heart, 100(21):1673\u20131680, 2014.\u2192 pages 13, 16[112] X. Kang, M. H. Jafari, M. M. Kazemi, C. Luong, T. Tsang, andP. Abolmaesumi. A light-weight deep video network: towards robustassessment of ejection fraction on mobile devices. In Medical Imaging2022: Image-Guided Procedures, Robotic Interventions, and Modeling,volume 12034, pages 332\u2013337. SPIE, 2022. \u2192 page 159[113] M. M. Kazemi Esfehani, C. Luong, D. Behnami, T. Tsang, andP. Abolmaesumi. A deep Bayesian video analysis framework: towards amore robust estimation of ejection fraction. In International Conference onMedical Image Computing and Computer-Assisted Intervention, pages582\u2013590. Springer, 2020. \u2192 pages xx, 29, 67, 72, 159, 195176[114] A. Kendall and Y. Gal. What uncertainties do we need in Bayesian deeplearning for computer vision? In Conference on Neural InformationProcessing Systems, pages 5574\u20135584, 2017. \u2192 page 57[115] M. Khened, V. A. Kollerathu, and G. Krishnamurthi. Fully convolutionalmulti-scale residual densenets for cardiac segmentation and automatedcardiac diagnosis using ensemble of classifiers. arXiv preprintarXiv:1801.05173, 2018. \u2192 page 18[116] C. Kim, J. Hur, B. S. Kang, et al. Can an offsite expert remotely evaluatethe visual estimation of ejection fraction via a social network video call?Journal of Digital Imaging, 30(6):718\u2013725, 2017. \u2192 pages 14, 32[117] M. Kinno, P. Nagpal, S. Horgan, and A. H. Waller. Comparison ofechocardiography, cardiac magnetic resonance, and computed tomographicimaging for the evaluation of left ventricular myocardial function: Part 2(diastolic and regional assessment). Current Cardiology Reports, 19(1):6,2017. \u2192 pages 6, 12, 15, 69[118] S. Ko, Y. Kim, J. Park, and N. Choi. Assessment of left ventricular ejectionfraction and regional wall motion with 64-slice multidetector CT: acomparison with two-dimensional transthoracic echocardiography. TheBritish Journal of Radiology, 83(985):28\u201334, 2010. \u2192 page 16[119] B. Kong, Y. Zhan, M. Shin, T. Denny, and S. Zhang. Recognizingend-diastole and end-systole frames via deep temporal regression network.In International Conference on Medical Image Computing and ComputerAssisted Intervention, pages 264\u2013272. Springer, 2016. \u2192 page 32[120] C. Krittanawong, A. Tunhasiriwet, H. Zhang, Z. Wang, M. Aydar, andT. Kitai. Deep learning with unsupervised feature in echocardiographicimaging. Journal of the American College of Cardiology, 69(16):2100\u20132101, 2017. \u2192 page 18[121] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features fromtiny images. 2009. \u2192 pages 154, 155[122] K. Kusunose, K. Shibayama, H. Iwano, M. Izumo, N. Kagiyama,K. Kurosawa, H. Mihara, H. Oe, T. Onishi, T. Onishi, et al. Reducedvariability of visual left ventricular ejection fraction assessment withreference images: the Japanese Association of Young EchocardiographyFellows multicenter study. Journal of Cardiology, 72(1):74\u201380, 2018. \u2192page 80177[123] K. Kusunose, T. Abe, A. Haga, D. Fukuda, H. Yamada, M. Harada, andM. Sata. A deep learning approach for assessment of regional wall motionabnormality from echocardiographic images. Cardiovascular Imaging, 13(2 Part 1):374\u2013381, 2020. \u2192 page 72[124] K. Kusunose, A. Haga, N. Yamaguchi, T. Abe, D. Fukuda, H. Yamada,M. Harada, and M. Sata. Deep learning for assessment of left ventricularejection fraction from echocardiographic images. Journal of the AmericanSociety of Echocardiography, 33(5):632\u2013635, 2020. \u2192 page 72[125] J. E. Lambrecht, K. Zhang, D. M. Tierney, P. Millner, D. Giovannini,K. Barron, W. Novak, S. A. Patel, R. Dversdal, E. J. Cox, et al. Integrationof point-of-care ultrasound education into the internal medicine coreclerkship experience. Journal of Ultrasound in Medicine, 41(1):33\u201340,2022. \u2192 page 69[126] R. M. Lang, M. Bierig, R. B. Devereux, F. A. Flachskampf, E. Foster, P. A.Pellikka, M. H. Picard, M. J. Roman, J. Seward, J. S. Shanewise, et al.Recommendations for chamber quantification: a report from the AmericanSociety of Echocardiography\u2019s guidelines and standards committee and thechamber quantification writing group, developed in conjunction with theeuropean association of echocardiography, a branch of the EuropeanSociety of Cardiology. Journal of the American Society ofEchocardiography, 18(12):1440\u20131463, 2005. \u2192 page 123[127] R. M. Lang, L. P. Badano, V. Mor-Avi, J. Afilalo, A. Armstrong,L. Ernande, F. A. Flachskampf, E. Foster, S. A. Goldstein, T. Kuznetsova,et al. Recommendations for cardiac chamber quantification byechocardiography in adults: an update from the American Society ofEchocardiography and the European Association of CardiovascularImaging. Journal of the American Society of Echocardiography, 28(1):1\u201339, 2015. \u2192 pages xv, 4, 8, 14, 15, 16, 19, 69, 70, 71, 77, 123[128] C. A. Lawson, F. Zaccardi, I. Squire, S. Ling, M. J. Davies, C. S. Lam,M. A. Mamas, K. Khunti, and U. T. Kadam. 20-year trends incause-specific heart failure outcomes by sex, socioeconomic status, andplace of diagnosis: a population-based study. The Lancet Public Health, 4(8):e406\u2013e420, 2019. \u2192 page 17[129] R. Lebeau, K. Serri, M. Di Lorenzo, C. Sauve\u00b4, V. Soulie`res, M. El-Rayes,M. Page\u00b4, C. Za\u0131\u00a8ani, J. Garot, F. Poulin, et al. Assessment of LVEF using a178new 16-segment wall motion score in echocardiography. Echo Researchand Practice, 5(2):63\u201369, 2018. \u2192 pages 8, 15, 70[130] S. Leclerc, T. Grenier, F. Espinosa, et al. A fully automatic andmulti-structural segmentation of the left ventricle and the myocardium onhighly heterogeneous 2D echocardiographic data. In UltrasonicsSymposium (IUS), 2017 IEEE International, pages 1\u20134. IEEE, 2017. \u2192pages 18, 31, 55[131] S. Leclerc, E. Smistad, J. Pedrosa, A. \u00d8stvik, F. Cervenansky, F. Espinosa,T. Espeland, E. A. R. Berg, P.-M. Jodoin, T. Grenier, et al. Deep learningfor segmentation using an open large-scale dataset in 2D echocardiography.IEEE transactions on medical imaging, 38(9):2198\u20132210, 2019. \u2192 pages72, 121, 155[132] K. C. Lee, S. Liu, P. Callahan, T. Green, T. Jarrett, J. Cochran, Y. Mei,S. Mobasseri, H. Sayegh, V. Rangarajan, et al. Routine use of contrast onadmission transthoracic echocardiography for heart failure reduces the rateof repeat echocardiography during index admission. Journal of theAmerican Society of Echocardiography, 34(12):1253\u20131261, 2021. \u2192 page115[133] P. Leeson. (Deep) learning your left from your right. CardiovascularImaging, 13(2 Part 1):382\u2013384, 2020. \u2192 page 118[134] R. Leischik, B. Dworrak, and K. Hensel. Intraobserver and interobserverreproducibility for radial, circumferential and longitudinal strainechocardiography. The Open Cardiovascular Medicine Journal, 8:102,2014. \u2192 page 16[135] S. J. Lester, A. J. Tajik, R. A. Nishimura, J. K. Oh, B. K. Khandheria, andJ. B. Seward. Unlocking the mysteries of diastolic function: decipheringthe rosetta stone 10 years later. Journal of the American College ofCardiology, 51(7):679\u2013689, 2008. \u2192 page 123[136] Z. Liao, H. Girgis, A. Abdi, H. Vaseli, J. Hetherington, R. Rohling, K. Gin,T. Tsang, and P. Abolmaesumi. On modelling label uncertainty in deepneural networks: Automatic estimation of intra-observer variability in 2Dechocardiography quality assessment. IEEE Transactions on MedicalImaging, 39(6):1868\u20131883, 2019. \u2192 pagesxxiii, 25, 27, 74, 78, 79, 80, 116, 150179[137] Z. Liao, M. H. Jafari, H. Girgis, K. Gin, R. Rohling, P. Abolmaesumi, andT. Tsang. Echocardiography view classification using quality transfer stargenerative adversarial networks. In International Conference on MedicalImage Computing and Computer-Assisted Intervention, pages 687\u2013695.Springer, 2019. \u2192 pages 72, 86, 87[138] N. Liel-Cohen, Y. Tsadok, R. Beeri, P. Lysyansky, Y. Agmon, M. S.Feinberg, W. Fehske, D. Gilon, I. Hay, R. Kuperstein, et al. A new tool forautomatic assessment of segmental wall motion based on longitudinal 2Dstrainclinical perspective: a multicenter study by the Israeliechocardiography research group. Circulation: Cardiovascular Imaging, 3(1):47\u201353, 2010. \u2192 pages 15, 17, 69[139] J. Lin, G. Sahebzamani, C. Luong, F. T. Dezaki, M. Jafari, P. Abolmaesumi,and T. Tsang. Reciprocal landmark detection and tracking with extremelyfew annotations. In Proceedings of the IEEE\/CVF Conference on ComputerVision and Pattern Recognition, pages 15170\u201315179, 2021. \u2192 page 72[140] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,P. Dolla\u00b4r, and C. L. Zitnick. Microsoft COCO: Common Objects inContext. In European Conference on Computer Vision, pages 740\u2013755.Springer, 2014. \u2192 pages 154, 155[141] G. Luo, S. Dong, K. Wang, and H. Zhang. Cardiac left ventricular volumesprediction method based on atlas location and deep learning. InInternational Conference on Bioinformatics and Biomedicine (BIBM),pages 1604\u20131610. IEEE, 2016. \u2192 page 18[142] C. Luong, Z. Liao, A. Abdi, H. Girgis, R. Rohling, K. Gin, J. Jue,D. Yeung, E. Szefer, D. Thompson, et al. Automated estimation ofechocardiogram image quality in hospitalized patients. The InternationalJournal of Cardiovascular Imaging, 37(1):229\u2013239, 2021. \u2192 pages114, 115[143] C. Luong, S. Saboktakin Rizi, K. Gin, J. Jue, D. F. Yeung, M. Y. Tsang,E. C. Sayre, and T. S. Tsang. Prevalence of left ventricular systolicdysfunction by single echocardiographic view: towards an evidence-basedpoint of care cardiac ultrasound scanning protocol. The InternationalJournal of Cardiovascular Imaging, pages 1\u20138, 2021. \u2192 pages 109, 114[144] J. Mantilla, M. Garreau, J. J. Bellanger, and J. L. Paredes. SVM-basedclassification of LV wall motion in cardiac MRI with the assessment of180STE. In 10th International Symposium on Medical Information Processingand Analysis, volume 9287, page 92870N. International Society for Opticsand Photonics, 2015. \u2192 page 19[145] R. Maragna, C. M. Giacari, M. Guglielmo, A. Baggiano, L. Fusini, A. I.Guaricci, A. Rossi, M. Rabbat, and G. Pontone. Artificial intelligencebased multimodality imaging: a new frontier in coronary artery diseasemanagement. Frontiers in Cardiovascular Medicine, 8, 2021. \u2192 page 73[146] P. H. Mayo, Y. Beaulieu, P. Doelken, D. Feller-Kopman, C. Harrod,A. Kaplan, J. Oropello, A. Vieillard-Baron, O. Axler, D. Lichtenstein, et al.American college of Chest Physicians\/la socie\u00b4te\u00b4 de re\u00b4animation de langueFranc\u00b8aise statement on competence in critical care ultrasonography. Chest,135(4):1050\u20131060, 2009. \u2192 page 78[147] S. Mazaheri, P. S. B. Sulaiman, R. Wirza, F. Khalid, S. Kadiman, M. Z.Dimon, and R. M. Tayebi. Echocardiography image segmentation: asurvey. In Advanced Computer Science Applications and Technologies(ACSAT), 2013 International Conference, pages 327\u2013332. IEEE, 2013. \u2192page 18[148] S. Mazaheri, R. Wirza, P. S. Sulaiman, M. Z. Dimon, F. Khalid, and R. M.Tayebi. Segmentation methods of echocardiography images for leftventricle boundary detection. Journal of Computer Science, 11(9):957,2015. \u2192 page 18[149] D. Medvedofsky, V. Mor-Avi, I. Byku, A. Singh, L. Weinert, M. Yamat,E. Kruse, B. Ciszek, A. Nelson, K. Otani, et al. Three-dimensionalechocardiographic automated quantification of left heart chamber volumesusing an adaptive analytics algorithm: feasibility and impact of imagequality in nonselected patients. Journal of the American Society ofEchocardiography, 30(9):879\u2013885, 2017. \u2192 page 80[150] S. S. Mitter, S. J. Shah, and J. D. Thomas. A test in context: E\/a and e\/e toassess diastolic dysfunction and LV filling pressure. Journal of theAmerican College of Cardiology, 69(11):1451\u20131464, 2017. \u2192 page 147[151] M. Monfort, A. Andonian, B. Zhou, K. Ramakrishnan, S. A. Bargal,T. Yan, L. Brown, Q. Fan, D. Gutfreund, C. Vondrick, et al. Moments intime dataset: one million videos for event understanding. IEEETransactions on Pattern Analysis and Machine Intelligence, 42(2):502\u2013508, 2019. \u2192 pages 154, 155181[152] V. Mor-Avi, R. M. Lang, L. P. Badano, M. Belohlavek, N. M. Cardim,G. Derumeaux, M. Galderisi, T. Marwick, S. F. Nagueh, P. P. Sengupta,et al. Current and evolving echocardiographic techniques for thequantitative evaluation of cardiac mechanics: ASE\/EAE consensusstatement on methodology and indications endorsed by the JapaneseSociety of Echocardiography. European Journal of Echocardiography, 12(3):167\u2013205, 2011. \u2192 pages 16, 19[153] R. Muraki, A. Teramoto, K. Sugimoto, K. Sugimoto, A. Yamada, andE. Watanabe. Automated detection scheme for acute myocardial infarctionusing convolutional neural network and long short-term memory. PublicLibrary of Science (PloS) One, 17(2):e0264002, 2022. \u2192 page 73[154] Y. Nagata, Y. Kado, T. Onoue, K. Otani, A. Nakazono, Y. Otsuji, andM. Takeuchi. Impact of image quality on reliability of the measurements ofleft ventricular systolic function and global longitudinal strain in 2Dechocardiography. Echo Research and Practice, 5(1):27\u201339, 2018. \u2192 page80[155] S. F. Nagueh, C. P. Appleton, T. C. Gillebert, P. N. Marino, J. K. Oh, O. A.Smiseth, A. D. Waggoner, F. A. Flachskampf, P. A. Pellikka, andA. Evangelisa. Recommendations for the evaluation of left ventriculardiastolic function by echocardiography. European Journal ofEchocardiography, 10(2):165\u2013193, 2009. \u2192 pagesxxvii, xxviii, 123, 129, 132, 133, 134, 140, 141, 144[156] S. F. Nagueh, O. A. Smiseth, C. P. Appleton, B. F. Byrd, H. Dokainish,T. Edvardsen, F. A. Flachskampf, T. C. Gillebert, A. L. Klein,P. Lancellotti, et al. Recommendations for the evaluation of left ventriculardiastolic function by echocardiography: an update from the AmericanSociety of Echocardiography and the European Association ofCardiovascular Imaging. European Journal of Echocardiography, 17(12):1321\u20131360, 2016. \u2192 pagesxxvii, xxviii, 123, 126, 129, 132, 133, 134, 140, 141, 142, 144[157] S. F. Nagueh, T. P. Abraham, G. P. Aurigemma, J. J. Bax, C. Beladan,A. Browning, M. A. Chamsi-Pasha, V. Delgado, G. Derumeaux, G. Dolci,et al. Interobserver variability in applying American Society ofEchocardiography\/European Association of Cardiovascular Imaging 2016guidelines for estimation of left ventricular filling pressure. Circulation:Cardiovascular Imaging, 12(1):e008122, 2019. \u2192 page 129182[158] M. Nasr-Esfahani, M. Mohrekesh, M. Akbari, S. Soroushmehr,E. Nasr-Esfahani, N. Karimi, S. Samavi, and K. Najarian. Left ventriclesegmentation in cardiac MR images using fully convolutional network.arXiv preprint arXiv:1802.07778, 2018. \u2192 page 18[159] National Heart, Blood and Lung Institute (NHBLI). Heart tests.https:\/\/www.nhlbi.nih.gov\/health\/heart-tests, 2022. \u2192 pages 6, 7, 8[160] J. F. Nauta, Y. M. Hummel, P. van der Meer, C. S. Lam, A. A. Voors, andJ. P. van Melle. Correlation with invasive left ventricular filling pressuresand prognostic relevance of the echocardiographic diastolic parametersused in the 2016 ESC heart failure guidelines and in the 2016 ASE\/EACVIrecommendations: a systematic review in patients with heart failure withpreserved ejection fraction. European Journal of Heart Failure, 20(9):1303\u20131311, 2018. \u2192 page 129[161] L. Nelson, H. Vaseli, M. Mahdavi, F. Taheri Dezaki, C. Luong, D. Yeung,K. Gin, M. Tsang, P. Nair, J. Jue, P. Abolmaesumi, and T. Tsang. Deeplearning model for accurate diagnosis of atrial fibrillation onechocardiography. Journal of the American Society of Echocardiography(in press), June 2022. \u2192 page 50[162] T. A. Ngo, Z. Lu, and G. Carneiro. Combining deep learning and level setfor the automated segmentation of the left ventricle of the heart fromcardiac cine magnetic resonance. Medical Image Analysis, 35:159\u2013171,2017. \u2192 pages 18, 31[163] R. A. Nishimura and A. J. Tajik. Evaluation of diastolic filling of leftventricle in health and disease: Doppler echocardiography is the clinician\u2019srosetta stone. Journal of the American College of Cardiology, 30(1):8\u201318,1997. \u2192 page 123[164] S. Nistri, P. Ballo, D. Mele, B. Papesso, M. Galderisi, S. Mondillo, G. B.Zito, and M. Y. Henein. Effect of echocardiographic grading of leftventricular diastolic dysfunction by different classifications in primarycare. The American Journal of Cardiology, 116(7):1144\u20131152, 2015. \u2192page 129[165] J. A. Noble and D. Boukerroui. Ultrasound image segmentation: a survey.IEEE Transactions on Medical Imaging, 25(8):987\u20131010, 2006. \u2192 page 18[166] Y. Nosir, W. B. Vletter, E. Boersma, et al. The apical long-axis rather thanthe two-chamber view should be used in combination with the183four-chamber view for accurate assessment of left ventricular volumes andfunction. European Heart Journal, 18(7):1175\u20131185, 1997. \u2192 pages 46, 49[167] I. Oksuz, B. Ruijsink, E. Puyol-Anto\u00b4n, M. Sinclair, D. Rueckert, J. A.Schnabel, and A. P. King. Automatic left ventricular outflow tractclassification for accurate cardiac mr planning. In International Symposiumon Biomedical Imaging (ISBI), pages 462\u2013465. IEEE, 2018. \u2192 page 32[168] O. Oktay. Learning anatomical image representations for cardiac imaging.PhD thesis, Imperial College London, 2017. \u2192 page 19[169] O. Oktay, E. Ferrante, K. Kamnitsas, M. Heinrich, W. Bai, J. Caballero,S. A. Cook, A. de Marvao, T. Dawes, D. P. O\u2018Regan, et al. Anatomicallyconstrained neural networks (ACNNs): application to cardiac imageenhancement and segmentation. IEEE Transactions on Medical Imaging,37(2):384\u2013395, 2018. \u2192 page 19[170] H. A. Omar, J. S. Domingos, A. Patra, R. Upton, P. Leeson, and J. A.Noble. Quantification of cardiac bull\u2019s-eye map based on principal strainanalysis for myocardial wall motion assessment in stress echocardiography.In Biomedical Imaging (ISBI 2018), 2018 IEEE 15th InternationalSymposium on, pages 1195\u20131198. IEEE, 2018. \u2192 pages 4, 17, 19, 69, 72[171] D. Ouyang, B. He, A. Ghorbani, N. Yuan, J. Ebinger, C. P. Langlotz, P. A.Heidenreich, R. A. Harrington, D. H. Liang, E. A. Ashley, et al.Video-based ai for beat-to-beat assessment of cardiac function. Nature, 580(7802):252\u2013256, 2020. \u2192 page 67[172] D. Ouyang et al. Echonet-dynamic: a large new cardiac motion video dataresource for medical machine learning. In Conference on NeuralInformation Processing Systems (NeurIPS) Machine Learning for HealthWorkshop (ML4H): Vancouver, BC, Canada, 2019. \u2192 pages67, 72, 119, 155[173] N. Ouzir, O. Lairez, A. Basarab, and J.-Y. Tourneret. Tissue motionestimation using dictionary learning: Application to cardiac amyloidosis.In Ultrasonics Symposium (IUS), 2017 IEEE International, pages 1\u20134.IEEE, 2017. \u2192 page 18[174] T. E. Owan, D. O. Hodge, R. M. Herges, S. J. Jacobsen, V. L. Roger, andM. M. Redfield. Trends in prevalence and outcome of heart failure withpreserved ejection fraction. New England Journal of Medicine, 355(3):251\u2013259, 2006. \u2192 page 123184[175] A. Pandey, N. Kagiyama, N. Yanamala, M. W. Segar, J. S. Cho, M. Tokodi,and P. P. Sengupta. Deep-learning models for the echocardiographicassessment of diastolic dysfunction. Cardiovascular Imaging, 14(10):1887\u20131900, 2021. \u2192 page 129[176] W. J. Paulus, C. Tscho\u00a8pe, J. E. Sanderson, C. Rusconi, F. A. Flachskampf,F. E. Rademakers, P. Marino, O. A. Smiseth, G. De Keulenaer, A. F.Leite-Moreira, et al. How to diagnose diastolic heart failure: a consensusstatement on the diagnosis of heart failure with normal left ventricularejection fraction by the heart failure and echocardiography associations ofthe European Society of Cardiology. European Heart Journal, 28(20):2539\u20132550, 2007. \u2192 pages 123, 144[177] A. C. Pearson. Issues with estimating \u201cdiastolic function\u201d and leftventricular filling pressure using the new guidelines. Journal of theAmerican College of Cardiology, 70(9):1197\u20131198, 2017. \u2192 page 129[178] M. Penicka, M. Vanderheyden, and J. Bartunek. Diagnosis of heart failurewith preserved ejection fraction: role of clinical Doppler echocardiography.Heart, 100(1):68\u201376, 2014. \u2192 page 122[179] D. Peressutti, M. Sinclair, W. Bai, T. Jackson, J. Ruijsink, D. Nordsletten,L. Asner, M. Hadjicharalambous, C. A. Rinaldi, D. Rueckert, et al. Aframework for combining a motion atlas with non-motion information tolearn clinically useful biomarkers: application to cardiac resynchronisationtherapy response prediction. Medical Image Analysis, 35:669\u2013684, 2017.\u2192 page 19[180] A. Piergiovanni and M. Ryoo. Avid dataset: Anonymized videos fromdiverse countries. Advances in Neural Information Processing Systems, 33:16711\u201316721, 2020. \u2192 pages 154, 155[181] D. Plummer, D. Brunette, R. Asinger, and E. Ruiz. Emergency departmentechocardiography improves outcome in penetrating cardiac injury. Annalsof Emergency Medicine, 21(6):709\u2013712, 1992. \u2192 pages 4, 152[182] J. Pore\u00b4e, M. Baudet, F. Tournoux, G. Cloutier, and D. Garcia. A dualtissue-Doppler optical-flow method for speckle tracking echocardiographyat high-frame-rate. IEEE Transactions on Medical Imaging, 2018. \u2192 page19[183] A. R. Porras, M. Alessandrini, O. Mirea, J. D\u2019hooge, A. F. Frangi, andG. Piella. Integration of multi-plane tissue Doppler and B-mode185echocardiographic images for left ventricular motion estimation. IEEETransactions on Medical Imaging, 35(1):89\u201397, 2016. \u2192 page 19[184] E. Potter and T. H. Marwick. Assessment of left ventricular function byechocardiography: case for routinely adding global longitudinal strain toejection fraction. JACC: Cardiovascular Imaging, 11(2):260\u2013274, 2018. \u2192page 8[185] Public Health Agency of Canada. Report from the Canadian chronicdisease surveillance system: Heart disease in Canada.https:\/\/www.canada.ca\/en\/public-health\/services\/publications\/diseases-conditions\/heart-disease-canada-fact-sheet.html, 2018. \u2192pages 3, 4, 75[186] E. Puyol-Anto\u00b4n, M. Sinclair, B. Gerber, M. S. Amzulescu, H. Langet,M. De Craene, P. Aljabar, P. Piro, and A. P. King. A multimodalspatiotemporal cardiac motion atlas from MR and ultrasound data. MedicalImage Analysis, 40:96\u2013110, 2017. \u2192 page 19[187] E. Puyol-Anto\u00b4n, M. Sinclair, B. Gerber, M. S. Amzulescu, H. Langet,M. De Craene, P. Aljabar, J. A. Schnabel, P. Piro, and A. P. King.Multiview machine learning using an atlas of cardiac cycle motion. InInternational Workshop on Statistical Atlases and Computational Modelsof the Heart, pages 3\u201311. Springer, 2017. \u2192 page 19[188] E. Puyol-Anto\u00b4n, B. Ruijsink, W. Bai, H. Langet, M. De Craene, J. A.Schnabel, P. Piro, A. P. King, and M. Sinclair. Fully automated myocardialstrain estimation from cine MRI using convolutional neural networks. InInternational Symposium on Biomedical Imaging (ISBI), pages 1139\u20131143.IEEE, 2018. \u2192 page 19[189] Radiology Info for Patients. Cardiac CT for calcium scoring.https:\/\/www.radiologyinfo.org\/en\/info\/ct calscoring, 2020. \u2192 page 7[190] U. Raghavendra, H. Fujita, A. Gudigar, R. Shetty, K. Nayak, U. Pai,J. Samanth, and U. R. Acharya. Automated technique for coronary arterydisease characterization and classification using DD-DTDWT in ultrasoundimages. Biomedical Signal Processing and Control, 40:324\u2013334, 2018. \u2192page 73[191] M. M. Redfield, S. J. Jacobsen, J. C. Burnett, D. W. Mahoney, K. R. Bailey,and R. J. Rodeheffer. Burden of systolic and diastolic ventriculardysfunction in the community: appreciating the scope of the heart failure186epidemic. Journal of the American Medical Association, 289(2):194\u2013202,2003. \u2192 pages 122, 123[192] V. L. Roger, S. A. Weston, M. M. Redfield, J. P. Hellermann-Homan,J. Killian, B. P. Yawn, and S. J. Jacobsen. Trends in heart failure incidenceand survival in a community-based population. Journal of the AmericanMedical Association, 292(3):344\u2013350, 2004. \u2192 page 3[193] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networksfor biomedical image segmentation. In International Conference onMedical Image Computing and Computer-assisted Intervention, pages234\u2013241. Springer, 2015. \u2192 page 87[194] S. Sabour, N. Frosst, and G. E. Hinton. Dynamic routing between capsules.In Advances in Neural Information Processing Systems, pages 3859\u20133869,2017. \u2192 pages 42, 48[195] G. Sahebzamani. A weakly supervised landmark detection for automaticmeasurement of left ventricular diameter in videos of plax from cardiacultrasound. https:\/\/open.library.ubc.ca\/soa\/cIRcle\/collections\/ubctheses\/24\/items\/1.0396334, 2021. \u2192 pages xx, 25, 29[196] S. Sanchez-Martinez, N. Duchateau, T. Erdei, A. G. Fraser, B. H. Bijnens,and G. Piella. Characterization of myocardial motion patterns byunsupervised multiple kernel learning. Medical Image Analysis, 35:70\u201382,2017. \u2192 page 18[197] L. Sanchis, R. Andrea, C. Falces, S. Poyatos, B. Vidal, and M. Sitges.Differential clinical implications of current recommendations for theevaluation of left ventricular diastolic function by echocardiography.Journal of the American Society of Echocardiography, 31(11):1203\u20131208,2018. \u2192 pages 129, 144, 145[198] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern recognition, pages4510\u20134520, 2018. \u2192 page 83[199] G. Savarese and L. H. Lund. Global public health burden of heart failure.Cardiac Failure Review, 3(1):7, 2017. \u2192 pages 2, 3[200] K. Seetharam, D. Brito, P. D. Farjo, and P. P. Sengupta. The role ofartificial intelligence in cardiovascular imaging: state of the art review.Frontiers in Cardiovascular Medicine, page 374, 2020. \u2192 page 118187[201] J. Selmeryd, E. Henriksen, J. Leppert, and P. Hedberg. Interstudyheterogeneity of definitions of diastolic dysfunction severely affectsreported prevalence. European Heart Journal - Cardiovascular Imaging,17(8):892\u2013899, 2016. \u2192 pages 129, 144[202] P. P. Sengupta, H. Kulkarni, and J. Narula. Prediction of abnormalmyocardial relaxation from signal processed surface ecg. Journal of theAmerican College of Cardiology, 71(15):1650\u20131660, 2018. \u2192 page 129[203] A. M. Shah, M. Cikes, N. Prasad, G. Li, S. Getchevski, B. Claggett,A. Rizkala, I. Lukashevich, E. O\u2019Meara, J. J. Ryan, et al.Echocardiographic features of patients with heart failure and preserved leftventricular ejection fraction. Journal of the American College ofCardiology, 74(23):2858\u20132873, 2019. \u2192 page 123[204] J. F. Silva, J. M. Silva, A. Guerra, et al. Ejection fraction classification intransthoracic echocardiography using a deep learning approach. InSymposium on Computer-Based Medical Systems, pages 123\u2013128. IEEE,2018. \u2192 pages 56, 66, 67[205] O. A. Smiseth, H. Torp, A. Opdahl, K. H. Haugaa, and S. Urheim.Myocardial strain imaging: how useful is it in clinical decision making?European Heart Journal, 37(15):1196\u20131207, 2015. \u2192 page 16[206] E. Smistad, A. \u00d8stvik, et al. 2D left ventricle segmentation using deeplearning. In Ultrasonics, pages 1\u20134. IEEE, 2017. \u2192 pages 18, 55[207] M. A. Soliman Hamad, A. H. van Straten, J. P. Scho\u00a8nberger, J. F. terWoorst, A. M. de Wolf, E. J. Martens, and A. A. van Zundert. Preoperativeejection fraction as a predictor of survival after coronary artery bypassgrafting: comparison with a matched general population. Journal ofCardiothoracic Surgery, 5(1):1\u20138, 2010. \u2192 page 31[208] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 humanactions classes from videos in the wild. arXiv preprint arXiv:1212.0402,2012. \u2192 pages 154, 155[209] T. Stanton, R. Leano, and T. H. Marwick. Prediction of all-cause mortalityfrom global longitudinal speckle strain: comparison with ejection fractionand wall motion scoring. Circulation: Cardiovascular Imaging, pages356\u2013364, 2009. \u2192 page 16188[210] J. V. Stough, J. DiPalma, Z. Ma, B. K. Fornwalt, and C. M. Haggerty.Ventricular segmentation and quantitative assessment in cardiac MR usingconvolutional neural networks. In Medical Imaging 2018: BiomedicalApplications in Molecular, Structural, and Functional Imaging, volume10578, page 1057826. International Society for Optics and Photonics,2018. \u2192 page 18[211] A. Suinesiaputra, A. F. Frangi, T. A. Kaandorp, H. J. Lamb, J. J. Bax, J. H.Reiber, and B. P. Lelieveldt. Automated detection of regional wall motionabnormalities based on a statistical model applied to multislice short-axiscardiac mr images. IEEE Transactions on Medical Imaging, 28(4):595\u2013607, 2009. \u2192 pages 19, 73[212] A. Suinesiaputra, P. Ablin, X. Alba, M. Alessandrini, J. Allen, W. Bai,S. Cimen, P. Claes, B. R. Cowan, J. D\u2019hooge, et al. Statistical shapemodeling of the left ventricle: myocardial infarct classification challenge.IEEE Journal of Biomedical and Health Informatics, 22(2):503\u2013515, 2018.\u2192 page 19[213] F. Taheri Dezaki. A machine learning framework for spatio-temporalcardiac assessment of echocardiographic cines. PhD thesis, University ofBritish Columbia, 2021. \u2192 pages xx, 29[214] A. J. Tajik. Machine learning for echocardiographic imaging: embarkingon another incredible journey. Journal of the American College ofCardiology, 2016. \u2192 pages 17, 20[215] L. K. Tan, Y. M. Liew, E. Lim, and R. A. McLaughlin. Cardiac leftventricle segmentation using convolutional neural network regression. InBiomedical Engineering and Sciences (IECBES) and Engineering inMedicine and Biology Society Conference (EMBS), pages 490\u2013493. IEEE,2016. \u2192 page 56[216] S. Targ, D. Almeida, and K. Lyman. ResNet in ResNet: Generalizingresidual architectures. arXiv preprint arXiv:1603.08029, 2016. \u2192 page 83[217] J. J. Thaden, M. Y. Tsang, C. Ayoub, R. Padang, V. T. Nkomo, S. F. Tucker,C. S. Cassidy, M. Bremer, G. C. Kane, and P. A. Pellikka. Associationbetween echocardiography laboratory accreditation and the quality ofimaging and reporting for valvular heart disease. Circulation:Cardiovascular Imaging, 10(8):e006140, 2017. \u2192 page 115189[218] K. Thygesen, J. S. Alpert, H. D. White, et al. Universal definition ofmyocardial infarction. Journal of the American College of Cardiology, 50(22):2173\u20132195, 2007. \u2192 page 4[219] D. A. Tighe, M. Rosetti, C. S. Vinch, D. Chandok, D. Muldoon, B. Wiggin,S. T. Dahlberg, and G. P. Aurigemma. Influence of image quality on theaccuracy of real time three-dimensional echocardiography to measure leftventricular volumes in unselected patients: a comparison withgated-SPECT imaging. Echocardiography, 24(10):1073\u20131080, 2007. \u2192pages 80, 115[220] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learningspatiotemporal features with 3D convolutional networks. In Proceedings ofthe IEEE International Conference on Computer Vision, pages 4489\u20134497,2015. \u2192 pages 84, 157[221] D. Tran, H. Wang, L. Torresani, et al. A closer look at spatiotemporalconvolutions for action recognition. In IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pages 6450\u20136459, 2018. \u2192 pages60, 67, 84[222] C. W. Tsao, A. W. Aday, Z. I. Almarzooq, A. Alonso, A. Z. Beaton, M. S.Bittencourt, A. K. Boehme, A. E. Buxton, A. P. Carson,Y. Commodore-Mensah, et al. Heart disease and stroke statistics - 2022update: a report from the American Heart Association. Circulation, 145(8):e153\u2013e639, 2022. \u2192 page 3[223] S. Unzek, Z. B. Popovic, T. H. Marwick, and D. G. C. Investigators. Effectof recommendations on interobserver consistency of diastolic functionevaluation. JACC: Cardiovascular Imaging, 4(5):460\u2013467, 2011. \u2192 page123[224] M. A. Vannan, E. Argulian, and P. Lancellotti. Phenogrouping diastolicdysfunction by artificial intelligence: Learning from what we teach themachines. Cardiovascular Imaging, 14(10):1901\u20131903, 2021. \u2192 pages122, 123, 129[225] H. Vaseli, Z. Liao, A. H. Abdi, H. Girgis, D. Behnami, C. Luong, F. T.Dezaki, N. Dhungel, R. Rohling, K. Gin, et al. Designing lightweight deeplearning models for echocardiography view classification. In MedicalImaging 2019: Image-Guided Procedures, Robotic Interventions, andModeling, volume 10951, page 109510F. International Society for Opticsand Photonics, 2019. \u2192 pages 25, 62, 66, 78, 80, 195190[226] D. M. Vigneault, W. Xie, D. A. Bluemke, and J. A. Noble. Feature trackingcardiac magnetic resonance via deep learning and spline optimization. InInternational Conference on Functional Imaging and Modeling of theHeart, pages 183\u2013194. Springer, 2017. \u2192 page 19[227] D. M. Vigneault, W. Xie, C. Y. Ho, D. A. Bluemke, and J. A. Noble. \u03c9-net(omega-net): Fully automatic, multi-view cardiac MR detection,orientation, and segmentation with deep neural networks. Medical ImageAnalysis, 48:95\u2013106, 2018. \u2192 page 18[228] S. S. Virani, A. Alonso, H. J. Aparicio, E. J. Benjamin, M. S. Bittencourt,C. W. Callaway, A. P. Carson, A. M. Chamberlain, S. Cheng, F. N. Delling,et al. Heart disease and stroke statistics - 2021 update: a report from theAmerican Heart Association. Circulation, 143(8):e254\u2013e743, 2021. \u2192page 3[229] Wikimedia Commons, the free media repository. File:heartdiagram-en.svg. https:\/\/commons.wikimedia.org\/w\/index.php?title=File:Heart diagram-en.svg&oldid=573514806, 2021. \u2192 pages xvii, 2[230] Wikimedia Commons, the free media repository. File:systolevsdiastole.png. https:\/\/commons.wikimedia.org\/w\/index.php?title=File:Systolevs Diastole.png&oldid=565092215, 2021. \u2192 pages xvii, 3[231] Wikimedia Commons, the free media repository. File:wiggers diagram.svg.https:\/\/commons.wikimedia.org\/w\/index.php?title=File:Wiggers Diagram.svg&oldid=578036966, 2021. \u2192 pages xvii, 3[232] World Health Organization (WHO). Global status report onnoncommunicable diseases 2014.https:\/\/apps.who.int\/iris\/handle\/10665\/148114, 2014. \u2192 page 32[233] World Health Organization (WHO). Global Health Observatory (GHO).http:\/\/www.who.int\/gho\/mortality burden disease\/causes death\/top 10\/en\/,2017. \u2192 pages 2, 148[234] N. V. Woudenberg, Z. Liao, A. H. Abdi, H. Girgis, C. Luong, H. Vaseli,D. Behnami, H. Zhang, K. Gin, R. Rohling, et al. Quantitativeechocardiography: real-time quality estimation and view classificationimplemented on a mobile android device. Simulation, Image Processing,and Ultrasound Systems for Assisted Diagnosis and Navigation, pages74\u201381, 2018. \u2192 pages 25, 36, 80, 195191[235] W. Xue, A. Islam, M. Bhaduri, and S. Li. Direct multitype cardiac indicesestimation via joint representation and regression learning. IEEETransactions on Medical Imaging, 36(10):2057\u20132067, 2017. \u2192 page 32[236] W. Xue, A. Lum, A. Mercado, et al. Full quantification of left ventricle viadeep multitask learning network respecting intra-and inter-task relatedness.In International Conference on Medical Image Computing and ComputerAssisted Intervention, pages 276\u2013284. Springer, 2017. \u2192 pages 32, 56[237] W. Xue, G. Brahm, S. Pandey, S. Leung, and S. Li. Full left ventriclequantification via deep multitask relationships learning. Medical ImageAnalysis, 43:54\u201365, 2018. \u2192 page 32[238] X. Yang, C. Bian, L. Yu, D. Ni, and P.-A. Heng. Class-balanced deepneural network for automatic ventricular structure segmentation. InInternational Workshop on Statistical Atlases and Computational Modelsof the Heart, pages 152\u2013160. Springer, 2017. \u2192 page 18[239] Y. Yang, X.-M. Guo, H. Wang, and Y.-N. Zheng. Deep learning-basedheart sound analysis for left ventricular diastolic dysfunction diagnosis.Diagnostics, 11(12):2349, 2021. \u2192 page 129[240] X. Yao, D. R. Rushlow, J. W. Inselman, R. G. McCoy, T. D. Thacher, E. M.Behnken, M. E. Bernard, S. L. Rosas, A. Akfaly, A. Misra, et al. Artificialintelligence\u2013enabled electrocardiograms for identification of patients withlow ejection fraction: a pragmatic, randomized clinical trial. NatureMedicine, 27(5):815\u2013819, 2021. \u2192 page 152[241] D. F. Yeung*, R. Jiang*, D. Behnami*, J. Jue, R. Sharma, M. Turaga, C. L.Luong, M. Y. Tsang, K. G. Gin, H. Girgis, et al. Impact of the updateddiastolic function guidelines in the real world. International Journal ofCardiology, 326:124\u2013130, 2021. \u2192 pages ix, 150[242] L. A. Zadeh. Toward a theory of fuzzy information granulation and itscentrality in human reasoning and fuzzy logic. Fuzzy sets and systems, 90(2):111\u2013127, 1997. \u2192 page 55[243] D. Zhang, I. Icke, B. Dogdas, S. Parimal, S. Sampath, J. Forbes, A. Bagchi,C.-L. Chin, and A. Chen. A multi-level convolutional LSTM model for thesegmentation of left ventricle myocardium in infarcted porcine cine MRimages. In IEEE International Symposium on Biomedical Imaging (ISBI),pages 470\u2013473. IEEE, 2018. \u2192 page 18192[244] D. Zhang, I. Icke, B. Dogdas, S. Parimal, S. Sampath, J. Forbes, A. Bagchi,C.-L. Chin, and A. Chen. Segmentation of left ventricle myocardium inporcine cardiac cine mr images using a hybrid of fully convolutional neuralnetworks and convolutional LSTM. In Medical Imaging 2018: ImageProcessing, volume 10574, page 105740A. International Society for Opticsand Photonics, 2018. \u2192 page 18[245] J. Zhang, S. Gajjala, P. Agrawal, et al. A web-deployed computer visionpipeline for automated determination of cardiac structure and function anddetection of disease by two-dimensional echocardiography.arXiv:1706.07342, 2017. \u2192 pages 18, 31, 55[246] X. Zhen, Z. Wang, A. Islam, et al. Multi-scale deep networks andregression forests for direct bi-ventricular volume estimation. MedicalImage Analysis, 30:120\u2013129, 2016. \u2192 page 32[247] X. Zhen, H. Zhang, A. Islam, M. Bhaduri, I. Chan, and S. Li. Direct andsimultaneous estimation of cardiac four chamber volumes by multioutputsparse regression. Medical Image Analysis, 36:184\u2013196, 2017. \u2192 page 32[248] Q. Zheng, H. Delingette, N. Duchateau, and N. Ayache. 3D consistent &robust segmentation of cardiac images by deep learning with spatialpropagation. IEEE Transactions on Medical Imaging, 2018. \u2192 page 18[249] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng.Deepvit: Towards deeper vision transformer. arXiv preprintarXiv:2103.11886, 2021. \u2192 page 154[250] X. Zhu, D. Cheng, Z. Zhang, S. Lin, and J. Dai. An empirical study ofspatial attention mechanisms in deep networks. In Proceedings of theIEEE\/CVF International Conference on Computer Vision, pages6688\u20136697, 2019. \u2192 page 154[251] X. Zhuang, L. Li, C. Payer, D. Stern, M. Urschler, M. P. Heinrich, J. Oster,C. Wang, O. Smedby, C. Bian, et al. Evaluation of algorithms formulti-modality whole heart segmentation: An open-access grand challenge.arXiv preprint arXiv:1902.07880, 2019. \u2192 page 55[252] M. Zolgharni, M. Negoita, N. M. Dhutia, M. Mielewczik, K. Manoharan,S. Sohaib, J. A. Finegold, S. Sacchi, G. D. Cole, and D. P. Francis.Automatic detection of end-diastolic and end-systolic frames in 2Dechocardiography. Echocardiography, 34(7):956\u2013967, 2017. \u2192 pages32, 50193Appendix ARelated Co-authored PublicationsIn addition to the first-author publications listed in the Preface (i.e. in Ch. ), theauthor of this dissertation D. Behnami contributed to the data preparation for theinitial LV segmentation study by [101], as well as phase detection by Dezaki et al.:\u2022 M. H. Jafari, H. Girgis, Z. Liao, D. Behnami, A. Abdi, H. Vaseli, C. Luong,R. Rohling, K. Gin, T. Tsang, et al. A unified framework integrating recur-rent fully-convolutional networks and optical flow for segmentation of theleft ventricle in echocardiography data. In Deep Learning in Medical Im-age Analysis and Multimodal Learning for Clinical Decision Support, pages29\u201337. Springer, 2018 ([101] in Bibliography).\u2022 F. T. Dezaki, Z. Liao, C. Luong, H. Girgis, N. Dhungel, A. H. Abdi, D. Behnami,K. Gin, R. Rohling, P. Abolmaesumi, et al. Cardiac phase detection inechocardiograms with densely gated recurrent neural networks and globalextrema loss. IEEE Transactions on Medical Imaging, 38(8):1821\u20131832,2018 ([59] in Bibliography).D. Behnami contributed to related research, including problem formulation anddata preparation for the team\u2019s view and quality classification model used in Chap-ters 2- 4. The results were published in:\u2022 N. V. Woudenberg, Z. Liao, A. H. Abdi, H. Girgis, C. Luong, H. Vaseli,D. Behnami, H. Zhang, K. Gin, R. Rohling, et al. Quantitative echocardiog-raphy: real-time quality estimation and view classification implemented on a194mobile android device. Simulation, Image Processing, and Ultrasound Sys-tems for Assisted Diagnosis and Navigation, pages 74\u201381, 2018 ([234] inBibliography).\u2022 H. Vaseli, Z. Liao, A. H. Abdi, H. Girgis, D. Behnami, C. Luong, F. T.Dezaki, N. Dhungel, R. Rohling, K. Gin, et al. Designing lightweight deeplearning models for echocardiography view classification. In Medical Imag-ing 2019: Image-Guided Procedures, Robotic Interventions, and Modeling,volume 10951, page 109510F. International Society for Optics and Photon-ics, 2019 ([225] in Bibliography).D. Behnami contributed to problem formulation, data preparation, and method de-velopment for regional wall motion analysis published in Asgharzadeh\u2019s masterthesis:\u2022 P. Asgharzadeh. A deep learning framework for wall motion abnormality de-tection in echocardiograms. https:\/\/open.library.ubc.ca\/collections\/ubctheses\/24\/items\/1.0388865, 2020.As a continuation of uncertainty modelling for left ventricular ejection fraction,D. Behnami contributed to the initial problem formulation, data preparation, andmanuscript preparation, i.e. writing results analysis and presentation in:\u2022 M. M. Kazemi Esfehani, C. Luong, D. Behnami, T. Tsang, and P. Abolmae-sumi. A deep Bayesian video analysis framework: towards a more robust es-timation of ejection fraction. In International Conference on Medical ImageComputing and Computer-Assisted Intervention, pages 582\u2013590. Springer,2020 ([113] in Bibliography).195Appendix BSupporting Materials: Echo DataRetrieval PipelinesData for this thesis was queried and retrieved from Vancouver Coastal Health(VCH). The diagrams below illustrate the sequences involved in:\u2022 echo data acquisition (via general physician\u2019s referral in Fig. B.1 or emer-gency visit in Fig. B.2;\u2022 approval processes via BC\u2019s research ethics board (REB) in Fig. B.3;\u2022 data query through VCH in Fig. B.4;\u2022 and data retrieval, download and processing in Fig. B.4.196Figure B.1: Sequence of planned echocardiographer visit in Vancouver, BC.197Figure B.2: Sequence of emergency echocardiographer visit in Vancouver, BC.198Figure B.3: Process for obtaining ethics and operational approvals for retrieving retrospective clinical echocardiogra-phy data for research and development at UBC.199Figure B.4: Echocardiography data access process.200Figure B.5: Retrieval process for the retrospective clinical echo data used for research and development at UBC.201Appendix CFileMaker Clinical ReportingInterfaceSnapshots of the FileMaker Pro 7 (Apple Inc., Cupertino, California, United States) [13]interface used for preparing echo reports at clinical workstations are shown inFigs. C.1-C.4, courtesy of Vancouver Coastal Health (VCH).202Figure C.1: FileMaker reporting interface on the main page, including pa-tient information (name, ID, sex, etc.), disease indications, patientlifestyle and history indications, study type and modality, text-basedtechnician comments, etc.203Figure C.2: FileMaker reporting interface for ventricular function and struc-ture assessment, including left and right ventricular ejection fraction(EF), systolic wall motion analysis, diastolic function, as well as hy-pertrophy, and thrombus.204Figure C.3: FileMaker reporting interface for atrial, aortic and pericardial as-sessment, including left and right atrial chamber quantification, aorticassessment, pericardial effusion, tamponade and congenital indications.205Figure C.4: FileMaker reporting interface for valvular assessment, includingaortic valve (AV), mitral valve (MV), tricuspid valve (TV), and pul-monary valve assessment.206","type":"literal","lang":"en"}],"http:\/\/www.europeana.eu\/schemas\/edm\/hasType":[{"value":"Thesis\/Dissertation","type":"literal","lang":"en"}],"http:\/\/vivoweb.org\/ontology\/core#dateIssued":[{"value":"2022-11","type":"literal","lang":"en"}],"http:\/\/www.europeana.eu\/schemas\/edm\/isShownAt":[{"value":"10.14288\/1.0417279","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/language":[{"value":"eng","type":"literal","lang":"en"}],"https:\/\/open.library.ubc.ca\/terms#degreeDiscipline":[{"value":"Electrical and Computer Engineering","type":"literal","lang":"en"}],"http:\/\/www.europeana.eu\/schemas\/edm\/provider":[{"value":"Vancouver : University of British Columbia Library","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/publisher":[{"value":"University of British Columbia","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/rights":[{"value":"Attribution-NonCommercial-NoDerivatives 4.0 International","type":"literal","lang":"*"}],"https:\/\/open.library.ubc.ca\/terms#rightsURI":[{"value":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/","type":"literal","lang":"*"}],"https:\/\/open.library.ubc.ca\/terms#scholarLevel":[{"value":"Graduate","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/contributor":[{"value":"Abolmaesumi, Purang","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/title":[{"value":"Machine learning for diagnosing functional heart disease in echocardiography","type":"literal","lang":"en"}],"http:\/\/purl.org\/dc\/terms\/type":[{"value":"Text","type":"literal","lang":"en"}],"https:\/\/open.library.ubc.ca\/terms#identifierURI":[{"value":"http:\/\/hdl.handle.net\/2429\/82322","type":"literal","lang":"en"}]}}