Applied Science, Faculty of
Biomedical Engineering, School of
DSpace
UBCV
Zhu, Hongzhi
Doctor of Philosophy - PhD
University of British Columbia
Despite the rapid advancement in medical imaging technologies, health care systems in Canada as well as in many other countries still cannot provide patients with sufficient medical imaging resources due to the ever-growing demands, lack of imaging devices, and fully trained medical practitioners. What makes the situation worse are persistent and prevalent work-related injury situations for medical practitioners, especially sonographers. To help these problems, the overall objective of this thesis is to improve the user interface design for ultrasound machines and contribute to automated medical imaging-based diagnosis with the help of the gaze tracking technology. Multi-themed study and research were conducted to achieve the objectives. On one hand, for the interface design improvement, we started by analyzing the characteristics of gaze signals through statistical modelling. Then we surveyed among sonographers to understand their daily usage of the ultrasound machines. Also, we analyzed different kinds of ultrasound machines to understand common design patterns and suboptimal control logic. To improve the suboptimal control logic and observed usage difficulties on ultrasound machines, we designed and implemented a gaze tracking contingent ultrasound machine. For the validation of our design ideas and machine effectiveness, comparative user studies were conducted. Additionally, we enhanced the user experience by solving the gaze tracker accuracy deterioration problem during normal usage of the devices by proposing a disruption-free auto-recalibration method. On the other hand, for the automated medical image diagnosis, with the integration of the gaze tracking dataset, we proposed deep learning methods that not only can provide a label that predicts which kind of abnormality is observed in the medical image, but also some reasoning that can guide humans to understand how the prediction is made. Multi-task learning methods and neural network attention mechanisms were used for enhanced automated diagnostic performance and better interpretability of the deep learning models. Our study has demonstrated the usefulness of integrating gaze tracking with human-computer interaction and image understanding in the medical field. Future work can be done to further processing and quantify gaze tracking data.
Gaze Tracking for Human-Ultrasound Machine Interaction andMedical Image UnderstandingbyHongzhi ZhuB.Eng., Tianjin University, 2014M.Eng., McGill University, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES(Biomedical Engineering)The University of British Columbia(Vancouver)August 2022© Hongzhi Zhu, 2022The following individuals certify that they have read, and recommend to the Faculty of Graduateand Postdoctoral Studies for acceptance, the thesis entitled:Gaze Tracking for Human-Ultrasound Machine Interaction and Medical ImageUnderstandingsubmitted by Hongzhi Zhu in partial fulfillment of the requirements for the degree of Doctorof Philosophy in Biomedical Engineering.Examining Committee:Professor Robert Rohling, Electrical and Computer Engineering, Mechanical Engineering, andBiomedical Engineering, UBCCo-SupervisorProfessor Septimiu Salcudean, Electrical and Computer Engineering, and Biomedical Engineer-ing, UBCCo-SupervisorProfessor Jane Wang, Electrical and Computer Engineering, UBCSupervisory Committee MemberAssistant Professor Lyndia Wu, Mechanical Engineering, UBCSupervisory Committee MemberProfessor Terry Peters, Robarts Research Institute, Western UniversityExternal examinerProfessor Alla Sheffer, Computer Science, UBCUniversity examinerAssociate Professor Miriam Spering, Ophthalmology and Visual Sciences, UBCUniversity examineriiAbstractDespite the rapid advancement in medical imaging technologies, health care systems in Canadaas well as in many other countries still cannot provide patients with sufficient medical imagingresources due to the ever-growing demands, lack of imaging devices, and fully trained medicalpractitioners. What makes the situation worse are persistent and prevalent work-related injurysituations for medical practitioners, especially sonographers. To help these problems, the over-all objective of this thesis is to improve the user interface design for ultrasound machines andcontribute to automated medical imaging-based diagnosis with the help of the gaze trackingtechnology.Multi-themed study and research were conducted to achieve the objectives. On one hand,for the interface design improvement, we started by analyzing the characteristics of gaze signalsthrough statistical modelling. Then we surveyed among sonographers to understand their dailyusage of the ultrasound machines. Also, we analyzed different kinds of ultrasound machinesto understand common design patterns and suboptimal control logic. To improve the subop-timal control logic and observed usage difficulties on ultrasound machines, we designed andimplemented a gaze tracking contingent ultrasound machine. For the validation of our designideas and machine effectiveness, comparative user studies were conducted. Additionally, we en-hanced the user experience by solving the gaze tracker accuracy deterioration problem duringnormal usage of the devices by proposing a disruption-free auto-recalibration method. On theother hand, for the automated medical image diagnosis, with the integration of the gaze track-ing dataset, we proposed deep learning methods that not only can provide a label that predictswhich kind of abnormality is observed in the medical image, but also some reasoning that canguide humans to understand how the prediction is made. Multi-task learning methods and neu-ral network attention mechanisms were used for enhanced automated diagnostic performanceand better interpretability of the deep learning models.Our study has demonstrated the usefulness of integrating gaze tracking with human-computerinteraction and image understanding in the medical field. Future work can be done to furtherprocessing and quantify gaze tracking data.iiiLay SummaryThis thesis focuses on the application of gaze tracking technology for improved medical pro-cess efficiency and ergonomics. Through field observation, survey, interview, and statisticalmodelling, we analyze gaze behaviors and interaction on ultrasound machines performed bysonographers. Based on investigation results, we enhanced interaction efficiency and user ex-perience by introducing gaze tracking (with auto-calibration) to assist with user interaction onultrasound systems. We also discovered that gaze-contingent multi-task learning and attentionformation in deep convolutional neural networks can outperform existing methods for sometasks in disease classification and medical image interpretation.ivPrefaceResearch work presented in this thesis was conducted at the Robotic and Control Laboratory,University of British Columbia, Vancouver, BC, Canada.Some of the research processes involve the collection and processing of human data, whichwas approved by the Office of Research Ethics at the University of British Columbia with iden-tification number H15-02969.Major contributions in this thesis have been reported to multiple publications as listed be-low:• The work discussed in Chapter 2 has been published in[247] Hongzhi Zhu, Septimiu Salcudean, and Robert Rohling. “The Neyman Pearsondetection of microsaccades with maximum likelihood estimation of parameters.” Journalof Vision 19.13 (2019): 17-17.I am the main contributor to the research. I was mainly responsible for the concept forma-tion, experiment design/implementation, data analysis and manuscript writing. SeptimiuSalcudean and Robert Rohling supervised and advised the research. All authors partici-pated in the manuscript revision.• The work discussed in Chapter 3 has been published in[250] Hongzhi Zhu, Yasmin Halwani, Robert Rohling, Sidney Fels, and Septimiu Salcud-ean. “A unified representation of control logic in human-ultrasound machine interaction.”IEEE Journal of Biomedical and Health Informatics (2022).I carried on the research after Yasmin Halwani’s preliminary investigation. I was primar-ily responsible for the concept formation, data analysis and manuscript writing. YasminHalwani conducted questionnaire design and data collection. Septimiu Salcudean, Sid-ney Fels, and Robert Rohling supervised and advised the research. Septimiu Salcudean,Robert Rohling and myself participated in manuscript revision.• The work discussed in Chapter 4 has been published inv[248] Hongzhi Zhu, Septimiu Salcudean, and Robert Rohling. “A novel gaze-supportedmultimodal human–computer interaction for ultrasound machines.” International Journalof Computer Assisted Radiology and Surgery 14.7 (2019): 1107-1115.I am the lead investigator to this study. I was responsible for experiment design/imple-mentation, data collection/analysis, and manuscript writing. Septimiu Salcudean formu-lated the study concept. Septimiu Salcudean and Robert Rohling supervised the researchand revised the manuscript.• The work discussed in Chapter 5 has been published in[249] Hongzhi Zhu, Robert Rohling, and Septimiu Salcudean. “Hand-eye coordination-based implicit re-calibration method for gaze tracking on ultrasound machines: a statisti-cal approach.” International Journal of Computer Assisted Radiology and Surgery, 15.5(2020): 837-845.I led this study. I contributed to major areas of concept formation, experiment design/im-plementation, data collection/analysis, and manuscript writing. Septimiu Salcudean andRobert Rohling supervised the research and revised the manuscript.• The work discussed in Chapter 6 has been submitted asHongzhi Zhu, Septimiu Salcudean, and Robert Rohling. “Jointly Boosting Saliency Pre-diction and Disease Classification on Chest X-ray Images with Multi-task UNet.” In An-nual Conference on Medical Image Understanding and Analysis (2022): 594-608Hongzhi Zhu, Robert Rohling, and Septimiu Salcudean. “Gaze-Guided Class ActivationMapping: Leveraging Human Attention for Network Attention in Chest X-rays Classifi-cation.” (in press)I managed this study. I was primarily responsible for the concept formation, experimentdesign/implementation, data analysis, and manuscript writing. Septimiu Salcudean andRobert Rohling supervised the research and revised the manuscripts.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 Eye movements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.2 Gaze Tracking Methods . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.3 Gaze Tracking and Biomedical Engineering . . . . . . . . . . . . . . . 41.1.4 Interaction methods for US machines . . . . . . . . . . . . . . . . . . 71.2 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.3 Objectives and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.4 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Statistical Modelling and Processing of Eye-gaze Data . . . . . . . . . . . . . . . 112.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Statistical Modeling of Gaze Data . . . . . . . . . . . . . . . . . . . . . . . . 132.2.1 Model forH0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2.2 Model forH1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14vii2.3 Proposed Detection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.3.1 Parameter Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.3.2 Test Statistics for NPD . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.3 NPD Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252.4.1 Test with Simulated Data . . . . . . . . . . . . . . . . . . . . . . . . . 252.4.2 Test with real data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 Survey on Human-computer Interaction Methods for Ultrasound Machines . . . 373.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.2.1 Field observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.2.2 Survey and interview . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.2.3 US machine investigation . . . . . . . . . . . . . . . . . . . . . . . . 403.2.4 UML diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.3 Survey Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.3.1 Demographic characteristics of survey respondents . . . . . . . . . . . 413.3.2 Usage frequency of US machine operations and functions . . . . . . . 413.4 UML control logic representation . . . . . . . . . . . . . . . . . . . . . . . . 433.4.1 Function classification . . . . . . . . . . . . . . . . . . . . . . . . . . 433.4.2 UML diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.5 Control logic evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.5.1 Advantages of using a consistent control logic . . . . . . . . . . . . . . 493.5.2 Potential improvements to the consistent control logic . . . . . . . . . 503.5.3 Applicability to intelligent HCI . . . . . . . . . . . . . . . . . . . . . 513.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524 Human-Ultrasound Machine Interface Assisted by Gaze Tracking . . . . . . . . 544.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544.2 System Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.2.1 Common HCI design features of US machines . . . . . . . . . . . . . 554.2.2 Usage difficulties and design ideas . . . . . . . . . . . . . . . . . . . . 564.2.3 HCI arrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.3 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.3.1 Goals and hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . 604.3.2 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604.3.3 Data analysis and results . . . . . . . . . . . . . . . . . . . . . . . . . 62viii4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655 Disruption-Free Auto-Recalibration for Gaze Trackers . . . . . . . . . . . . . . 665.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.2 Statistical Modelling and re-calibration . . . . . . . . . . . . . . . . . . . . . . 675.2.1 Statistical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.2.2 Gaussian Process Regression (GPR) Framework . . . . . . . . . . . . 715.2.3 Implicit Recalibration with GPR . . . . . . . . . . . . . . . . . . . . . 735.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745.3.1 Re-calibration performance evaluation standard . . . . . . . . . . . . . 745.3.2 User study and Data Acquisition . . . . . . . . . . . . . . . . . . . . . 755.3.3 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 775.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786 Human Attention for Attention Networks in Deep Learning for Medical Images 816.1 Multi-task UNet: Jointly Boosting Saliency Prediction and Disease Classifica-tion on Chest X-ray Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826.1.3 Multi-task Learning Method . . . . . . . . . . . . . . . . . . . . . . . 836.1.4 Training and Evaluation Methods . . . . . . . . . . . . . . . . . . . . 876.1.5 Experiments and Result . . . . . . . . . . . . . . . . . . . . . . . . . 886.1.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.2 Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Net-work Attention in Chest X-rays Classification . . . . . . . . . . . . . . . . . . 916.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916.2.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 926.2.3 GG-CAM: Proposed Approach . . . . . . . . . . . . . . . . . . . . . . 946.2.4 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . 986.2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1047 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1057.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1057.2 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 1067.2.1 Qualitative versus quantitative analysis . . . . . . . . . . . . . . . . . 1067.2.2 Static versus dynamic gaze data representation . . . . . . . . . . . . . 1077.2.3 Medical behavioral data availability . . . . . . . . . . . . . . . . . . . 1077.2.4 Active gaze assumption . . . . . . . . . . . . . . . . . . . . . . . . . 107ixBibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108A Derivation of the Equations (2.27) and (2.29) . . . . . . . . . . . . . . . . . . . . 128B US machine function usage frequency scores . . . . . . . . . . . . . . . . . . . . 130C Nasa task load index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131xList of TablesTable 2.1 MSE of MLE estimated parameters . . . . . . . . . . . . . . . . . . . . . . 26Table 2.2 Summary of data information (for testing) in [14]. . . . . . . . . . . . . . . 31Table 2.3 AUC and sensitivity index for each dataset of the CNN (with pretrained“combined weights”) and NPD methods. . . . . . . . . . . . . . . . . . . . 31Table 2.4 Data information in [152] and NPD parameter estimates . . . . . . . . . . . 32Table 2.5 Summary of the detection results. . . . . . . . . . . . . . . . . . . . . . . . 32Table 2.6 Suggested parameter ranges based on empirical experience (for gaze signalsampled at 1000Hz). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Table 3.1 US machine details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Table 3.2 Kendall tau correlation and statistical test results between pairs of functionusage frequency categories. . . . . . . . . . . . . . . . . . . . . . . . . . . 43Table 3.3 Function complexity classification and minimum number of step(s) requiredfor the task completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Table 4.1 US machine details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56Table 5.1 Statistics on ATBs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77Table 6.1 Performance comparison between saliency prediction models. Values arepresented as median statistics followed by standard deviations behind the ±sign. Metrics with up-pointing arrow ↑ indicate greater values reflect betterperformances, and vice versa. KL, CC, and HI stand for Kullback–Leiblerdivergence, Pearson’s correlation coefficient, and histogram intersection, re-spectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Table 6.2 Performance comparison between multi-task learning schemes. The metricsare presented as median statistics followed by standard deviations behindthe ± sign. The up-pointing arrow ↑ indicates larger value stands for betterperformance, and vice versa. . . . . . . . . . . . . . . . . . . . . . . . . . 89xiTable 6.3 Visualization of visual heat map Ψ and network attention ΩY , (Y = 1,2,3),for each CNN. The heat maps are plotted on top of the CXR image for bet-ter visualization. The heat maps are color coded where red/orange indicateshigh concentration of attention and blue/green indicates low attention distri-bution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101Table 6.4 Performance comparison between CNNs. Values are presented as medianstatistics followed by the standard deviation after the ± sign. The best met-rics are highlighted in bold font. . . . . . . . . . . . . . . . . . . . . . . . . 103xiiList of FiguresFigure 1.1 Application-field based gaze tracking application classifications . . . . . . 5Figure 2.1 Generalized exponential function from equations (2.2) and the interpreta-tion of parameters in equations (2.3), (2.4), (2.5), (2.6), (2.9) and (2.10).Comparing the functions 1 (in blue) and 2 (in red), we can notice that θ2controls onset of the saccadic event. Comparing the functions 2 (in red) and3 (in yellow), we can notice that θ3 controls the abruptness of the amplitudechange. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Figure 2.2 Demonstration of a simulated gaze position measurements and the twostatesH0 andH1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Figure 2.3 Demonstration of σ2w and σ2v estimation . . . . . . . . . . . . . . . . . . . 19Figure 2.4 The distribution of statistics T with different SNR . . . . . . . . . . . . . . 27Figure 2.5 ROC curves for Algorithm 2 with known parameters . . . . . . . . . . . . 28Figure 2.6 Demonstration of the NPD output . . . . . . . . . . . . . . . . . . . . . . 29Figure 2.7 ROC curves of NPD and CNN for each dataset . . . . . . . . . . . . . . . 35Figure 2.8 Main Sequence with NPD and BIM . . . . . . . . . . . . . . . . . . . . . 36Figure 3.1 UML diagram example: Pointing and Selecting . . . . . . . . . . . . . . . 40Figure 3.2 Demographic characteristics of survey respondents as compared to thatfrom sonographers in British Columbia (BC), Canada, and in the USA . . . 42Figure 3.3 Survey result statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Figure 3.4 Function usage frequency (top 10 in each category) . . . . . . . . . . . . . 44Figure 3.5 UML diagram for binary functions . . . . . . . . . . . . . . . . . . . . . . 45Figure 3.6 UML diagram: 1D functions . . . . . . . . . . . . . . . . . . . . . . . . . 46Figure 3.7 UML diagram: pointing . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Figure 3.8 UML diagram: Measurement . . . . . . . . . . . . . . . . . . . . . . . . . 47Figure 3.9 UML diagram: TGC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Figure 3.10 UML diagrams for annotation and ROI selection . . . . . . . . . . . . . . 49xiiiFigure 3.11 UML diagram: Measurement (optimized), where “select”, “cancel”, and“end” are abbreviated button operations to select a point, cancel a pointselection, and finish the measurement, respectively. . . . . . . . . . . . . . 52Figure 4.1 System demonstration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Figure 4.2 Gaze supported HCI demonstration . . . . . . . . . . . . . . . . . . . . . 60Figure 4.3 Demonstration of the shooting game . . . . . . . . . . . . . . . . . . . . . 62Figure 4.4 One-way ANOVA for task duration . . . . . . . . . . . . . . . . . . . . . 62Figure 4.5 One-way ANOVA for measurement and selection errors . . . . . . . . . . 64Figure 4.6 NASA TLX score for each workload components . . . . . . . . . . . . . . 64Figure 4.7 One-way ANOVA on NASA TLX overall score . . . . . . . . . . . . . . . 65Figure 5.1 Hand-eye coordination from two datasets (best viewed in color) . . . . . . 69Figure 5.2 Heatmap of selection positions on the screen . . . . . . . . . . . . . . . . 71Figure 5.3 System setup (reproduced from [248]) . . . . . . . . . . . . . . . . . . . . 76Figure 5.4 Data visualization (best viewed in color). For (a), the light blue dots on theleft show the initial bias for each trial, and the corresponding end bias isplotted at the time instance that reflects the duration of the gaze trackingperiod. The thick blue line marks the median trend of all selected trials,whose slope is 0.55 mm/min, and the intercept is the median initial bias(1.53 cm). For (b), we use dual y-axes to present the score (right y-axis)and percentage values (left y-axis) separately. . . . . . . . . . . . . . . . . 79Figure 5.5 Demonstration of robustnesss to noise (best viewed in color) . . . . . . . . 80Figure 6.1 MT-UNet architecture. The solid blocks represent 3D tensors, RF×H×W ,where F , H, and W denote feature (channel), height and width dimensions,respectively. The solid circles represent 1D tensors. Arrows denote opera-tions to the tensors. Numbers above some of the solid blocks stand for thenumber features in tensors. . . . . . . . . . . . . . . . . . . . . . . . . . . 84Figure 6.2 Training process visualization with Equation (6.1) . . . . . . . . . . . . . 86Figure 6.3 Comparison with benchmark classification networks. Box plots are usedto demonstrate the statistics of the metrics values. The red horizontal lineindicates the median values; the top and bottom lines in blue boxes denotethe 75th and 25th percentile, respectively; and the shorter black bars at thefar ends of each box plots show the maximum and minimum values [231]. . 88Figure 6.4 Multi-task learning schemes comparison . . . . . . . . . . . . . . . . . . . 90Figure 6.5 Comparison between generic classification CNN and CAM-CNN. . . . . . 95Figure 6.6 Distribution of CAM elements. . . . . . . . . . . . . . . . . . . . . . . . . 97xivFigure 6.7 Training process visualization for GG-CAM modified EfficientNetV2. Logscale for the y-axis is used for all plots. Color blue is used for plots related toclassification. Color orange is used for plots related to attention supervision. 98Figure 6.8 Visualization of network performance for different initial values for σce andσsm. The AUC values are color coded through color brightness, wherebright (white or yellow) colors correspond to large values and dark (redor black) colors indicate small values. The values shown in this figure arebased on a single training. The best metrics are emphasized with bold, un-derlined fonts. Metrics from the standard initialization are underlined andshown in blue. Log scale is used in both figures due to the high dynamicrange for σ values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102xvGlossaryAI Artificial intelligenceANN Artificial neural networkATB Averaged tracking biasAUC Area under curveBIM Bayesian inference methodCAM Class activation mappingCDF Cumulative distribution functionCNN Convolutional neural networkCXR Chest X-rayDCNN Deep convolutional neural networkDDDM Data-driven detection methodDICOM Digital imaging and communications in medicineEOG Electro-oculographyFBDM Feature based detection methodsFSPD Fetal standardized plane detectionGG-CAM Gaze-guided class activation mappingGPR Gaussian process regressionGSA GE scan assistantHCI Human-computer interactionKLD Kullback-Leibler divergenceMSE Mean square errorMSK musculoskeletalNPD Neyman-Pearson detectorOB/GYN obstetrics and gynecologyPC Personal computerPCC Pearson’s correlation coefficientPCP Physical control panelPDF Probability density functionxviPFA Probability of false alarmPOG Point of gazeROC Receiver operating characteristicROI Region of interestSDM Statistical detection methodSGD Stochastic gradient descentSNR Signal-to-noise ratioTAD Tracking accuracy deteriorationTGC Time gain compensationTLX Task load indexUML Unified modelling languageUS UltrasoundxviiAcknowledgmentsThe pursuit of a PhD degree is more than a mere academic advancement and also a journeyof self-discovery and self-fulfillment. I am grateful to all the support and company that I wasluckily provided during the journey.First of all, I would like to express my gratitude to my supervisors: Prof. Robert Rohlingand Prof. Tim Saculdean. Their dedication to academic excellence has motivated me and willcontinue to inspire my enthusiasm for academic and career development. Their resourceful,considerate and flexible supervisory style has released my creativity and opened many excitingpossibilities. It was a meaningful, fruitful, and pleasant journey with my supervisors.Secondly, I would like to thank my parents. My advancement cannot leave their endeavor toprovide me a sound and carefree environment to study and research. It is their unconditional andwholehearted love that compassed my growth, passing through joys and vicissitudes. Thoughthey did not directly participate in my PhD study, their beliefs in scientific advancements arethe building blocks for my academic advancement.Lastly, I need to thank my friends, fellow lab-mates, and many other people whom I metduring my PhD study. All their companies have created colorful memories and lasting inspira-tions. Many of my friends also showed their support by directly participating in my research.Special thanks should also be delivered to an expert sonographer, Vickie Lessoway, who has apassion to help student researchers with her own expertise and knowledge in ultrasonography.No achievement can be made without their support.xviiiChapter 1IntroductionIn this chapter, we start by introducing background pertinent to this thesis, which includes eyemovement mechanisms and functionality, gaze tracking methods and their applications, andinteraction methods for ultrasound (US) machines. Next, we identify motivations and objectivesfor your study by showing knowledge gaps in the current literature and propose solutions forimprovements. Lastly, we state the structure of this thesis.1.1 Background1.1.1 Eye movementsIn our daily life, a myriad of visual information (around 10×1010 bit/s) reaches our retina,and it is with our visual system that we can select, transmit, and process that vast amount ofinformation [6, 175]. Eye movements (voluntary and involuntary) are integral to the properfunctioning of our visual system. There are five basic types of eye movements: saccades,smooth pursuit movements, vergence movements, vestibulo-ocular movements, and fixations[174, 228].Saccades are rapid binocular movements that shift the point of fixations, helping the explo-ration and examination of visual scenes [32, 174]. As ballistic movements, the amplitude ofsaccades is measured via angle of rotation. It is one of the most rapid movements exercised bythe human body, with peak angular velocity reaching 700 °/s [68]. A saccade may last 20 to200 ms depending on the amplitude and the underlying visual task [11]. For the efficient andtimely execution of saccades, neural passages have more direct control of eye muscles, avoidingdelays due to neural signal transmission and processing [63].Smooth pursuit movements, different from saccades, are much smoother and slower, allow-ing continued fixations on a moving object [174, 229]. Usually, two stages are involved for asmooth pursuit. The first state is the “open-loop pursuit”, when a moving object first attracts1one’s attention, and the visual system is preparing a pursuit to respond to the visual stimulus[123]. The second state is the “closed-loop pursuit”, when one’s eyes are moving with the targetobject with desired direction and velocity until the movement terminates [236]. For fast movingobjects such that a smooth pursuit movement cannot fully follow, catch-up saccades are alsoneeded to jointly accomplish the gaze pursuit of a moving object [48].Vergence movements usually occur when an object of interest is moving towards or awayfrom one, or when one tries to shift visual focus between near and far objects. During vergencemovements, the pupil on each eye rotates closer or further away from each other [174].Vestibulo-ocular movements serve to stabilize gaze positions during head movement withthe involvement of one’s vestibular system (sensing balance and orientation) [174]. Time effi-ciency is critical to vestibulo-ocular movements as one’s vision can be blurred as soon as headmovement is not fully counterbalanced. Therefore, the vestibulo-ocular reflex is deployed forthe movement, and it is one of the fastest reflexes (less than 10 ms lag) observed in human kind[201].Fixation is the type of eye movement when one tries to focus his or her eye position ona certain location of a scene. Even during fixation, one can hardly hold his or her eyes still,as various involuntary eye movements are always presented, including microsaccade (a loweramplitude saccade event) and drifts (slow and random displacement of eye positions). Usually,a person can have high visual acuity during fixation, which is important for more detailed visualinformation perception and processing [45].1.1.2 Gaze Tracking MethodsFour principle gaze tracking methods have been developed, which are electro-oculography(EOG) based tracking, scleral search coils, infrared oculography and video oculography. EOGand scleral search coils are intrusive gaze tracking methods, where direct equipment contactwith one’s body is required. Infrared oculography and video oculography are non-intrusivetracking methods, where a single or multiple cameras for facial/eye images with optional lightsources are involved for non-contact gaze tracking.Intrusive trackingIntrusive tracking methods were primarily used for eye and gaze related research in the lastcentury.One of the methods, EOG based tracking, takes the measured electrical potential difference(in Volts) between the front and back of one’s eye, and uses the potential difference to estimateits rotational movement [145]. To record EOG, electrodes are placed around each eye, yetmany factors can influence the recording accuracy, i.e., environmental noise, and non-stationaryelectromagnetic properties of one’s skin [22]. In order to convert EOG to eye movements,2model-based calibration is also required. As a result, the estimated gaze position based on EOGis susceptible to recording inaccuracies, as well as individual differences, model inaccuraciesand calibration error. The tracking inaccuracy can reach several degrees of visual angle.Another tracking method, scleral search coils, requires one to wear contact lenses whereinductive coils are embedded [213]. The embedded inductive coil is used to trace real-timemagnetic flux through the coil. When placed inside a reference magnetic field, the changes inmagnetic flux can reflect the positional information of the inductive coils, and thus tracing themovement of one’s eyes. Scleral search coils are more accurate than EOG based tracking witha tracking inaccuracy fewer than 1.0° of visual angle [96, 213]. Yet, search coils need to beworn which can be unnatural and uncomfortable for some people, and the external magneticfield imposes restrictions on body and head movements.Non-intrusive trackingInfrared oculography and video oculography are two of the most applied non-intrusive trackingmethods. The essence of infrared oculography and video oculography are the same, which is toextract features from facial/eye images for point of gaze (POG) estimation [107].Before the popularization of machine learning approaches, engineered image features wereused for POG estimation, and the most important of which are pupil centers on the image. Whilecommon RGB (red, green, and blue) images/videos used in video oculography can captureclear images of one’s face and eyes, the distinct reflection characteristics of one’s pupil andcornea can be used to enhance eye features. Thus, infrared oculography is developed, whereinfrared light sources light up one’s pupil and cornea, and infrared cameras capture infraredreflections while filtering out other lights [127]. Traditionally, at least two cameras are requiredfor the triangulation of pupil positions. In [90], a new model is proposed, allowing for singlecamera gaze tracking. Normal commercial non-intrusive gaze trackers, i.e., GP3 gaze tracker(GazePoint, Vancovuer, Canada) have tracking inaccuracy around 1.0° of visual angle, andsampling frequencies less than 100 Hz. Trackers with low sampling rate (less than 100 Hz) aremore affordable but may fail to capture fine movement of one’s eyes, as many movements, i.e.,saccades, are completed very swiftly. More advanced gaze trackers, i.e., Eyelink 1000 plus (SRResearch, Kanata, Canada), can achieve 2000 Hz sampling rate with tracking inaccuracy lessthan 0.5° of visual angle.Advancements in computer vision and computational speed have enabled more diversifiedgaze tracking approaches even without dedicated devices. In [189, 238], webcam based gazetracking is developed. Though they are usually less accurate than a gaze tracker, they are morecost-efficient and flexible. Deep learning is one of the most prominent driving forces for newgaze tracking solutions, yet reliability and accuracy are the primary concerns for deep learningbased tracking algorithms [176].31.1.3 Gaze Tracking and Biomedical EngineeringThe non-invasiveness and accessibility of the video oculography technology has enabled theconvenient applications of gaze tracking to myriads of fields, psychology, physiology, healthcare, etc. [120]. Various engineering, statistics, and computer science methods, i.e., signal pro-cessing, machine learning, internet of things, and cloud computing, are involved in the appli-cation, which enable more possibilities and dynamics in this interdisciplinary field [120, 148].In this section, emphasis is put on reviewing gaze tracking applications related to biomedicalengineering. To achieve this, we start by discussing classification methods on the gaze trackingapplications. Then we introduce recent advancements in utilizing gaze tracking for biomedicalengineering purposes.Application classificationAs an interdisciplinary field, there are four existing methods to classify gaze tracking applica-tions. The first method classifies applications by the tracking method (scleral search coil, in-frared oculography, EOG, or video oculography) used in the application [151, 155]. However,as users usually prefer non-invasive tracking methods, more recent studies commonly choosevideo oculography, and thus, the first classification method gradually becomes inefficient touse. Therefore, the second and third classification methods emerge, which differentiates appli-cations through application fields (health care, gaming, etc.) or applied technologies (machinelearning, signal processing, etc.), respectively. In [144], different from the previous approach,Majaranta and Bulling proposed the classification based on how the eye-gaze data are used inthe system. Ranging from intentionally using gaze signals for real-time command and control,to passively monitoring eye movement, Majaranta and Bulling suggests that applications canfit into the continuum of intentional to unintentional (passive) gaze tracking enabled systems.Examples of intentional systems are gaze tracking enabled HCI systems and assistive technolo-gies; Examples of unintentional systems are user behavior monitoring systems for marketing orpsychological analysis.In this thesis, we adopt the application-field based classification method (second method),as it helps identifying applications related to biomedical engineering, which by itself, is also aninterdisciplinary field. Besides the application-field based classification method, the applied-technology based classification method (third method) is also worth mentioning, as differentkinds of technologies have their unique impacts to the gaze tracking applications. Noticeably,with the booming of machine learning methods, novel ideas appear, reshaping the developmentcourse of gaze-tracking applications. Therefore, development in gaze tracking applicationsbased on machine learning will also be discussed while we review applications in each field.Figure 1.1 shows the non-exhaustive application-field based gaze tracking application clas-sifications. In the engineering field, we further classify applications into sub-categories: aug-4Gaze tracking applicationsSportsMarketingNeural ScienceHealth CarePhysiologyEngineeringPsychologyAugmented/Virtual RealityGamingRoboticsMedical InstrumentationEducationWorkflow OptimizationDiagnosisCompetency EvaluationTrainingGaze trackingin BiomedicalEngineeringFigure 1.1: Application-field based gaze tracking application classificationsmented/virtual reality, gaming, robotics, and medical instrumentation. In health care, sub-categories are workflow optimization, diagnosis, competency evaluation, and training. Thehighlighted green area in Figure 1.1 are the application fields pertinent to biomedical engi-neering. In the following subsections, we discuss gaze tracking applications to each field inbiomedical engineering.Application in Medical InstrumentationAs many medical operations involve bimanual tasks under a sterile environment, conventionalmedical instrument control with hands touching a control panel can be inefficient, and/or unsan-itary. Therefore, as an indicator of one’s attention and state of mind, gaze data is used to helpwith the touch-free control of medical equipment, either directly as an explicit input signal, orpassively for machine optimization.When explicitly used, the gaze signal serves as a control signal for surgical robotics andultrasound machines. In robotic surgeries, gaze signals are employed to control ablation area[200], robotic probe [160], medical endoscope [108], and robotic laparoscope [237]. Besidesconventional robotic surgeries, tele-operated robotic surgeries also benefited from gaze tracking.In [102] and [103], certain machine settings in a tele-operated Da Vinci surgical system aredirectly controlled by gaze signals. For US scans, the gaze signal is used to zoom and pan theimage on the US machine [78]. Despite the validation of the feasibility of gaze signal basedexplicit medical instrument control, the aforementioned applications usually require longer taskcompletion time or extra efforts from the operators, compared to traditional control methods.Other than explicit use, the gaze signal is also less explicitly exploited to alter or optimizemachine settings. Image perception optimization is one the most applied tasks for medicalinstruments with the help from eye-gaze. Images displayed in a medical instrument usuallyrequire reconstruction and post-processing from raw signal sources, i.e., US image and 3D re-constructed image. As computationally expensive tasks, image display may suffer from heavytime-lag or precision loss. To solve this problem, the gaze signal is used such that regions with5the operator’s attention are processed with extra efforts, optimizing the allocation of the com-putation resources [2, 42, 158, 218]. Gaze signal is passively used for other tasks as well. In[159], minimally invasive surgery accuracy and performance are boosted by using gaze signalsto reference the hepatic constraints for the surgical controls. In [79], by acquiring the gaze po-sitions of both the sonographer and the patient, this patented method can intelligently optimizethe screen position for each individual’s point of view. In [70], the gaze position is acquired tocheck the alignment of body parts with medical devices.Application in Workflow OptimizationWorkflow optimization with gaze tracking is a relatively new area of research that tries to iden-tify and perfect mental and behavioral states during medical practices. To achieve this goal, adatabase framework is proposed that records the operator’s gaze data as well as all operationson medical instruments [179].In the field of vision science, research has attempted to understand the visual behaviorbetween expert and novice medical practitioners for medical procedures, as experts typicallymasters highly efficient visual search and perception ability yet may or may not be fully recog-nized by themselves [125, 126]. Therefore, the primitive objective for vision science targetedfor medical imaging procedures is to identify the highly efficient visual perception and cog-nition strategies and better avoid errors, both perceptual and cognitive, in medical procedures[125, 210]. The observed expert’s strategy can then in return used for expertise level assessmentand training [49, 180]. Artificial intelligence also plays important roles in assisting with the vi-sion science for medical imaging procedures, as neural network is better at modelling nonlinearrelations and handle stochastic dataset [51, 199].Additionally, deeper insights can be drawn from the user’s gaze data when employed tofacilitate the interface design for medical devices. In [8], better workstation design ideas wereproposed according to the best visual search patterns extracted from the radiologists’ gaze datawhen viewing medical images for diagnosis. In [12], computer interface for surgeries wasenhanced based on attention distributions among operators. In [118], the physician’s intentionto use the electronic medical record was predicted, helping with the intelligent organization anduse of the records for accelerated diagnosis.Other than assisting interface design, visual patterns reflected by gaze data are importantindicators for diagnostic bias or error. In [56], systematic bias in fetus head circumference mea-surement with US images is discovered, as sonographers usually refer to the standard fetus headgrowth chart and adjust their own measurement values towards the standard circumference be-fore formally reporting the measurement. In [65, 153], visualization tools for the prescribing ofantibiotics to patients are built based on visual and other behaviors from physicians, better help-ing with the physician’s decision making process. In [72], eye movement metrics are evaluated6to discover the sources for diagnosis failures during an intensive care unit visit.Application in Medical DiagnosisVarious medical imaging techniques, i.e., X-ray, US, and magnetic resonance imaging, haveenabled more accurate diagnosis for medical practitioners. With the popularization of machinelearning methods, especially deep learning, automated medical diagnosis methods have beenextensively studied in recent years. In this subsection, we describe methods in automated med-ical diagnosis that are assisted by gaze tracking.Ahmed and Noble pioneered the integration of gaze tracking with medical diagnosis throughmanually inspecting the distribution of expert sonographers’ gaze on US images. They manu-ally engineered better features based on gaze analysis that can be applied in existing computervision algorithms for improved fetal standardized plane detection (FSPD) [3]. However, themanual inspection process for feature extraction is cumbersome and time consuming. There-fore, in [26], Droste et al. proposed a deep learning based FSPD that uses gaze heatmap as anadditional input to the network. The deep learning network can automatically integrate infor-mation from the gaze signal and the US image for feature extraction. The detection achievedhigher accuracy in [26], yet the network relies on gaze signal as input, making it difficult to usefor generic US scans where gaze tracking is not used. To avoid the additional input, in [25],Cai et al. proposed a generative adversarial network that tried to generate gaze heatmap for theFSPD task.More generally, the gaze signal is found to be useful in extracting generic features for med-ical images [55] and drive strong representation learning and transfer learning for related tasks[54]. Therefore, medical tasks not limited to FSPD can benefit from gaze tracking, i.e., tumorregion segmentation [150], medical image registration [40], abnormality detection [50], imageclassification [4]. More recently, gaze tracking is found to be helpful in network distillation andknowledge transferring tasks [171].1.1.4 Interaction methods for US machinesIn this subsection, we briefly review existing technologies applied for human-US-machine in-teraction, and their impacts to the sonographers. US imaging is one of the most frequently usedimaging techniques for medical diagnosis and intervention, because it is non-invasive, low-costand ionizing-radiation free [95]. In spite of the growing diversity of US systems, ergonomicsand productivity have always been criticized for US system design [7]. With several decades ofdevelopment, the functionality and human-computer interaction (HCI) methods of US systemsare rapidly evolving.Traditionally, cart-based US machines are used for diagnostic US scans. Though portableUS systems are proliferating, the US machine market is still dominated by cart-based US sys-7tems, which accounts for 71% of total equipment sales in 2018 globally [132]. The HCI com-ponents on cart-based US machines are typically composed of two separated parts: 1) the mainscreen, for the display of US images and some machine parameters; and 2) the control panel,for the adjustment of machine parameters [248]. Usually, a trackball, a touch screen, severalbuttons, knobs, and sliders are observed on the control panel.Newer designs also employ voice control and scan automation to enhance the usability ofthe HCI for US machines.Voice control enabled HCI, thanks to the advancement in natural language processing, is anovel method aiming at reducing physical repetitiveness and increasing control efficiency forUS systems [173]. US machine model iU22, by Philips, is the first commercially availablesystem that employs voice control for US systems. However, studies show that the use of voicecontrol activated commands can interfere with the normal conversation with the patient, and thevoice interpretation error by the machine can further hinder the fluent use of voice commands[77].Scan automation is a computer assisted technique that helps with the completion of US scantasks through customizable machine control protocols and parameter adjustment. To reduce theworkload during scans, many US machine manufactures developed scan automation methodson their system. The GE Scan Assistant (GSA), by GE Healthcare, is one of the commerciallyavailable scan automation methods. It is found that the employment of GSA can reduce therepetitiveness of body movement when adjusting machine parameters, as well as can simplifythe steps needed on machine control [88]. Other scan automation methods, such as in [198]by University of Science and Technology; [184], [202] by Philips; and [142] by Samsung canachieve similar outcomes.Given the prevalence of work-related injuries (WRI) for sonographers, ergonomics of theUS systems is extensively studied. In [64, 69, 75], general ergonomic guidelines regarding theoperator’s gestures and operation procedures are provided. Other factors, including lightingconditions [130], machine layout design [16], control panel design [10], and chair selection[224], are also analyzed to facilitate better uses of US systems. Measurements on the ergonomicperformance of the system are proposed in [177], and in [204], factors contributing to WRIamong sonographers are quantitatively rated for reflecting their severity and importance. Inspite of many dedicated research studies on improving the ergonomics of US systems, WRI,including musculoskeletal disorders and cumulative trauma disorders, is still prevalent amongsonographers [166, 187]. Reports show that around 90% of sonographers experienced WRI intheir career [156].81.2 MotivationsAdvancements in convenient and economical gaze tracking methods have inaugurated myriadsof attention and/or vision related research [57]. As a direct measurement of points of gaze, thegaze tracking technology can measure the spatial-temporal human gaze behavior as time series,facilitating the quantitative mathematical modelling and statistical processing [34]. The uniquespatial and attentional information conveyed in tracked gaze time series have contributed tomany successful applications in human computer interaction (HCI), machine learning, market-ing, cognitive science, linguistics, and many other fields [5, 57]. However, integration of gazetracking to the medical field, especially medical imaging, is largely missing, partially due to thestochasticity of the gaze signal and the lack of the delegated research efforts.Medical imaging, i.e., ultrasound (US), X-ray, and magnetic resonance imaging, is a fre-quently used diagnostic and interventional method for its glowingly exceptional capabilitiesto visualize the interior condition of the human body noninvasively [242]. As an inalienablepart of health care, the need for medical imaging continues to mount throughout the years[197]. However, the diagnostic processes with medical imaging systems can be error-prone andinjury-inducing for the operators. It has been found that the repetitiveness and sustained stressduring the US image acquisition processes can lead to work-related musculoskeletal disordersfor many sonographers [77], and delayed diagnosis and/or misdiagnosis can hardly be avoideddue to the deliberate efforts required to examine medical images [208]. Therefore, more effi-cient and ergonomic HCI for medical imaging, and more intelligent image analysis methods areneeded to increase the throughput and reduce the work required by medical practitioners.What’s more, as a spatial-temporal measurement of one’s visual behavior, the gaze signalis a direct measure of one’s attention. Therefore, during the medical image acquisition andexamination procedures, the unique spatial-temporal allocation of visual attention over the im-ages can be truthfully recorded via gaze tracking [234]. As the method that an expert adopts toallocate his or her visual attention itself is a useful source of information for the understandingof the diagnostic strategies, such information can be extracted via advanced signal processingand/or machine learning approaches to facilitate more intelligent medical image understandingand diagnosis.1.3 Objectives and ContributionsIn this thesis, we investigate a central hypothesis:Gaze tracking technology can be used to facilitate human-ultrasound machine inter-action and to assist automatic medical image understanding methods.9The validation of the hypothesis can help alleviate the workload for medical practitioners andincrease the throughput of medical procedures. To device validation methods, four objectivesare set:Obj 1 Statistical modelling of gaze signals: Construct and validate statistical models for the fil-tering and processing of gaze signals.Obj 2 User experience survey: Understand image acquisition procedures and usage difficultieson existing US machines.Obj 3 Novel HCI design: Create and validate eye-gaze contingent HCI for US machines.Obj 4 AI integration: Applying gaze data for augmented medical image interpretation with ma-chine learning.Among the 4 objectives, Obj 1 is the fundamental step for us to have a deeper understanding ofthe stochastic nature of eye movements and tracking inaccuracies that usually impede the gaze-signal based analysis. The construction of a proper model for the gaze signal can be used forsignal filtering and feature extraction, which can be used for later phases of our research. Obj 2focuses on the investigation on gaze patterns and usage routines during US image acquisition.This is an important step for user-centered designs, as we can acquire firsthand knowledge andexperiences directly from end users. Once Obj 1 and Obj 2 are completed, we can identify thepossibilities and challenges in integrating the gaze tracking technologies to the HCI designs aswell as for intelligent medical image analysis. Therefore, we can proceed to Obj 3 and Obj 4,where we propose gaze-tracking contingent HCIs and deep learning methods for medical imagebased diagnosis. Both Obj 3 and Obj 4 aim to increase the efficiency of medical procedures,where the former tries to reduce the time and efforts on medical device operation and medicalimage acquisition, and the latter focuses on accelerating the diagnostic procedures.1.4 Thesis StructureThe rest of the thesis is organized as follows. First, in Chapter 2, we focus on Obj 1, and pro-pose statistical models for gaze signals. The proposed statistical model is validated throughapplication in microsaccade detection. Next, in Chapter 3, we aim to achieve Obj 2, where weinvestigate usage routines on US machines during US exams and identify points of improve-ments on HCIs. In Chapters 4 and 5, we fulfill Obj 3, such that gaze contingent HCIs for USmachines are designed for better time-efficient and disruption-free operations. After that, withChapter 6, we propose deep learning methods with gaze tracking for improved image diagnosis(classification with better network interpretability) and saliency prediction. Lastly, we concludethis thesis in Chapter 7.10Chapter 2Statistical Modelling and Processing ofEye-gaze Data2.1 IntroductionVision is the primary method for us to perceive the world and communicate with each other[53]. Recent advancements in non-invasive eye-gaze tracking technologies have further in-creased interest in vision research, as we can exploit gaze position to gain knowledge about,for instance, human allocation of attention and inform human-computer interaction (HCI) tech-nologies [37, 183, 248]. Yet gaze-tracking for functional and cognitive research of the visualsystem is challenging because of the gaze-tracking inaccuracies as well as the stochastic natureof eye movements.People’s eyes are never still, not even during deliberate fixation. In [41, 219], it is found thatour eyes exhibit three types of involuntary movements: tremor, drift and microsaccades, duringfixation. The tremor is a high frequency, very low-amplitude movement firstly discovered in[1]. It is a highly rhythmical and oscillatory movement with peak frequency around 80Hz. Eyetracking equipment with very high accuracy is needed to record the tremor [15], and hence,common commercial eye tracking devices cannot capture the existence of tremor successfully[181]. Drift movements have low amplitudes and low velocities: for prolonged fixations, drifthas amplitudes in the range between 1′ and 8′ of visual angle, and velocities usually below30 ′ s−1 [52, 181]. The microsaccade, which occurs a few times per second, each with a du-ration spanning from 5 or 6 ms to about 20-30 ms, is a high-velocity eye movement with veryshort duration [60]. Microsaccades and saccades generally form a continuum of oculomotor be-haviors [163, 251]. Commonly in recent years, microsaccades are differentiated from saccadesas the events with amplitude below 1° of vision angle [196]. Some microsaccades can be muchsmaller than that, sometimes coming close to the measurement noise of many eye tracking de-11vices (for instance, the EyeLink II video-based infrared eye tracker reports a precision of 0.01°of visual angle) [60, 146]. Their small size combined with their relatively short duration ac-count partly for the difficulty of microsaccade detection, which represents an ongoing researchproblem.Past research on the topic of detecting a microsaccade can be roughly classified into threecategories: feature based detection methods (FBDMs), statistical detection methods (SDMs),and more recently, data-driven detection methods (DDDMs). The commonly used features byFBDMs are the velocity and acceleration of a microsaccade. In [60], the velocity thresholdmethod is proposed to detect microsaccades, and in [162], the unsupervised clustering methodbased on multiple features (peak velocity and acceleration) is used to differentiate a microsac-cade from other types of eye movements. A major disadvantage of employing FBDMs is theirvulnerability to noise. Therefore, in [193], FBDMs suitable for high-noise regime are pro-posed. Also, several SDMs have been developed to handle the noise in the tracked eye-gazepositions. Particle filtering, with a small number of effective particles, was proposed to detect(micro)saccades [47]. More recently, the hidden Markov model was proposed in [152], andthe Bayesian Inference Method (BIM) was derived accordingly as the microsaccade detector.A reliable and robust detection method would inevitably rely on an accurate statistical model.However, due to the differences between individuals, the problem of finding the general statisti-cal distributions for several characteristic quantities of a microsaccade, e.g., maximum velocity,duration and amplitude, are not fully solved yet. Though based on principled statistical infer-ence, the BIM has the following limitations: First, a constant velocity model for microsaccadeis proposed in BIM, which is not the case in real data. Such a model simplifies the calcula-tion at a cost of losing model accuracy. Second, the parameter estimation method in the BIMmay lead to biases in the detection results. Last but not least, BIM is effective when a full setof gaze data has already been acquired, yet real-time BIM implementation is difficult. There-fore, BIM detectors are difficult to use in HCI applications. Efforts have also been made indeveloping data-driven approaches to solve the problem of microsaccade detection. In [14], theCNN is trained using labelled gaze signals. To the best of our knowledge, this CNN method isthe current state-of-the-art method, providing, according to the authors, “human-level detectionperformance”. However, in contrast to a process or statistical model, a data-driven approach isagnostic to a possible underlying process and it does not rely on or directly infer microsaccadecharacteristics.To solve the aforementioned limitations of the BIM and the CNN approach, and to betterdetect a microsaccade during eye fixation, we propose an alternative SDM, NPD, for microsac-cade detection. To elaborate our method, the rest of the chapter is organized as follows. Wepropose our statistical model for gaze data during eye fixation in Section: Statistical Modelingof Gaze Data. Based on the proposed model, in Section: Proposed Detection Method, we derive12NPD for the detection of microsaccades. To test the validity of our model as well as to compareour NPD with BIM and the CNN method, results are presented in Section: Results, and lastly,we conclude this chapter in Section: Discussion and Conclusion. Our algorithm is publiclyavailable at https://github.com/hz-zhu/NPD-micro-saccade-detection.2.2 Statistical Modeling of Gaze DataIn our proposed model, we only employ the well-developed statistics for fixational eye move-ment. Our proposed model classifies a sequence of gaze data into the following binary states: 1)the state of pure drifting movement, denoted asH0, and 2) the state of microsaccade, denoted asH1. The tremor component during eye fixation is not taken into consideration as it is very lowin amplitude and very few gaze trackers can measure it reliably [121]. We present our statisticalmodel forH0, first. Then, the model forH1 is discussed with several of its properties derived.2.2.1 Model forH0Let k denote the discrete time index for gaze position measurements and k ∈ Z+, where Z+denotes the set of positive integers, i.e., {1,2,3, ...}; Xk denote the actual (noise-free) gazeposition as a visual angle at time k; and Zk denote the measured eye gaze position in visualangle at time k. It has been shown that the correlated random walk constitutes an accuratemodel for the drifting eye gaze movement [61, 149]. However, the correlation factors in [61]are not trivial to estimate from the dataset itself. Thus, we propose the use of a simplifiedyet more straightforward model based on [61], where only first-order correlation is considered.More specifically, we propose the following model for stateH0:Xk = Xk−1+WkZk = Xk +Vk (2.1)where Wk ∼N (0,σ2w) is the noise that models the drifting movement of the eye; Vk ∼N (0,σ2v )is the measurement noise; Wk and Vk are independent, ∀k ∈ Z+; σw, σv ∈ R+ represent thestandard deviation for Wk and Vk respectively; andN (β ,γ) stands for the Gaussian distributionwith mean β and variance γ .132.2.2 Model forH1In [81], the generalized exponential function was proposed to model the shape of a saccadestarting at time t = 0:h(t;θ ) =θ1{1− exp[−(tθ2)θ3]}, t ≥ 00, t < 0(2.2)where θ = [θ1,θ2,θ3]T ∈ R3 is a parameter vector, t ∈ R is the continuous time index, andT denotes the transpose operation. We need θ1 ∈ R, θ2 > 0 and θ3 > 1. As saccades andmicrosaccades are thought to share the same generating neural mechanism [147], we adoptthe same generalized exponential function to model a microsaccade. Figure 2.1 shows threeexamples of h(t;θ ) with different parameters. We can see that the parameter θ1 controls theamplitude of the function; θ2 and θ3 control the shape of the function. It is also demonstratedthat the velocity profile of a microsaccade can be captured by function (2.2),v(t;θ ) =∂h(t;θ )∂ t=θ1θ3θ2exp[−(tθ2)θ3]( tθ2)θ3−1 (2.3)which is a very good approximation of the actual velocity profile v(t;θ ) of a microsaccadeas studied in [214]. From equation (2.2) and Figure 2.1, we also notice the following facts:1) h(t;θ ) converges to θ1, as t → +∞; and 2) the time that v(t;θ ) reaches its maximum isdependent on θ . Therefore, we need to define the beginning and end of a microsaccade eventexplicitly based on equation (2.2). We define the duration of a microsaccade, denoted as d, thatfollows equation (2.2), as the time span [t1, t2] that its amplitude raises from 0.5θ1(1− dm) to0.5θ1(1+dm), where dm ∈ (0,1) is a scaling factor. Therefore, during the defined duration of amicrosaccade, dm×100% of the total amplitude θ1 has finished. Figure 2.1 helps demonstratethe relation of θ , dm, d, t1 and t2. It can be calculated that:t1 = θ2[− ln(0.5+0.5dm)] 1θ3 (2.4)t2 = θ2[− ln(0.5−0.5dm)] 1θ3 (2.5)d = t2− t1. (2.6)A value close to 1 should be chosen for dm, and in this chapter, we choose dm = 0.95. That isbecause for a saccade event following equation (2.2), which is an exponential function, it takesan infinite amount of time to reach the theoretical maximum gaze position displacement, θ1,140 10 20 30 40 50 60 70t00.10.20.30.40.50.60.70.80.91Amplitudeddm 1(t1, 0.5 1(1-dm))(t2, 0.5 1(1+dm))(tv, vmax)Function 11=1, 2=50, 3=20Function 21=1, 2=20, 3=20Function 31=1, 2=20, 3=5Velocity for function 1Figure 2.1: Generalized exponential function from equations (2.2) and the interpretationof parameters in equations (2.3), (2.4), (2.5), (2.6), (2.9) and (2.10). Comparingthe functions 1 (in blue) and 2 (in red), we can notice that θ2 controls onset of thesaccadic event. Comparing the functions 2 (in red) and 3 (in yellow), we can noticethat θ3 controls the abruptness of the amplitude change.and thus, we cannot use the time that the gaze position reaches the maximum displacement asthe duration of a saccade event. Therefore, dm is used, through which we define the saccadeduration as the time the gaze position displacement reaches 95% of the maximum displacement,0.95θ1. Figure 2.1 can also help demonstrating the relation between parameters t1, t2, d, h(t;θ )and v(t;θ ).It is natural to modify h(t;θ ) based on the defined starting and end of a microsaccade, as:hs(t;θ ) = h(t+ t1;θ ) (2.7)where hs(t;θ ) is the shifted function of h(t;θ ), such that the defined starting time of a mi-crosaccade coincides with the time t = 0. For sampled data with sampling interval ∆ ∈ R+, wedefineH(k;θ ) = hs(k∆;θ ), k ∈ Z+ (2.8)By letting∂v(t;θ )∂ t= 0, we can also calculate the time, tv, at which a microsaccade reachesits maximum velocity, vmax:tv = θ2(θ3−1θ3) 1θ3(2.9)vmax =θ1θ3θ2exp(−θ3+1θ3)(θ3−1θ3) θ3−1θ3(2.10)H(k;θ ) can capture the amplitude and velocity profile of a microsaccade parametrically.15Therefore, we can build the statistical model as:Xk = Xk−1+Wk +Hd(k;θ ,ω)Zk = Xk +Vk (2.11)whereHd(k;θ ,ω) = H(k−ω;θ )−H(k−ω−1;θ ). (2.12)and ω ∈ Z is the start time of a microsaccade; 0 < k+ω ≤ d, such that equation (2.11) isonly valid during the defined duration of a microsaccade; the drifting noise Wk and measure-ment noise Vk follow the same distribution as specified by equation (2.1); and the starting valueXω = 0. Comparing equation (2.11) with equation (2.1), we can note the only difference comesfrom the term Hd(k;θ ,ω), and the essence of Hd(k;θ ,ω) is that it models a deterministic dis-placement to represent the microsaccade. It is not hard to see that:E [Xk] = H(k+ω;θ ) (2.13)where E [·] represents the theoretical expectation.2.3 Proposed Detection MethodIn the last section, we built the statistical model for gaze data, and in this section, the statisticaldetection method for microsaccades is derived based on this model. We apply the widely usedNeyman Pearson detection [89, 215] method from signal processing to the problem of saccadicevent detection, specifically to test the hypothesis whether the eye is in stateH0 (drift) or stateH1 (saccadic event). While the Bayesian method makes use of prior distributions over theparameters, the Neyman Pearson method uses the likelihood ratio test for the two eye states andgives equal weight to the two possibilities. Therefore, in certain situations, such as datasets withhigh individual differences in microsaccade generation, priors over parameters might not befeasible, and the Neyman-Pearson approach may provide a better alternative for microsaccadedetection. Based on our model, a sequence of gaze data can be sectioned into chunks with twopossible states. A demonstration is shown in Figure 2.2, where the data points can be sectionedinto three chunks. Data in the left and right chunks follow stateH0 and the data in the middlechunk follow state H1. A microsaccade detector needs to provide us with the position of theboundary between different state chunks. Therefore, NPD can be utilized naturally to solve thetwo-state hypothesis testing problem. We can declare a detection of a microsaccade if stateH1is produced by NPD. Yet, before implementing NPD, several parameters need to be estimated.To demonstrate the proposed detection method, first, we show the moment based parameter16Figure 2.2: Demonstration of a simulated gaze position measurements and the two statesH0 andH1estimation method for σ2v and σ2w, as well as the MLE method for estimating θ . Then, wederive the test statistics for NPD under the Neyman Pearson detection framework. Lastly, weunify the parameter estimation method with NPD to perform the task of detecting saccadicevents, where some post processing steps are applied to further enhance the performance of thedetector.2.3.1 Parameter EstimationEstimation of σ2v and σ2wWe employ the moment based estimation method for σ2v and σ2w, which follows the methoddeveloped in [245], where the unknown variance is estimated using linear regression. Thedetailed application of the method in [245] is presented next.Under stateH0, we have:Zk−Zk−n =n∑i=1Wk+i+Vk+n−Vk (2.14)where n ∈ Z+, and therefore, we have:Zk−Zk−n ∼N(0,nσ2w+2σ2v). (2.15)With equation (2.15), we know that if we can estimate the variance of Zk−Zk−n, we can thensolve for σ2v and σ2w. Furthermore, we can also notice that VAR [Zk−Zk−n] is a linear functionwith respect to n, where VAR [·] is the empirical variance. Hence, σ2v and σ2w can be estimatedvia linear regression, such that the regression slope is σ2w, and one half of the linear regression17Algorithm 1 Estimate σ2v and σ2w1: Get a vector of measured gaze position Zk0:k1 =[Zk0 ,Zk0+1, ...,Zk1]T2: Section Zk0:k1 into m ∈ Z+ chunks, such that [Z1, ...,Zm]T = Zk0:k13: Pick a value n0 ∈ Z+. Construct n0 empty sets R1:n0 , and an all-zero vector G1:n04: for i = 1,2, ...,m do5: Let Yk denotes the kth element in Z i, and Z i has N elements in total6: for j = 1,2, ...,n0 do7: Calculate vector D = Y1+ j:N−Y1:N− j, and calculate σ2D = VAR [D]8: Add σ2D to the set V j9: for j = 1,2, ...,n0 do10: Let G j = min[V j]11: Perform linear regression G j = α j+β with respect to j12: The estimate of σ2v is σˆ2v = 0.5β13: The estimate of σ2w is σˆ2w = αintersection is σ2v .Under stateH1, we have:Zk−Zk−n =n∑i=1Wk+i+Vk+n−Vk+H(k+ω;θ )−H(k−n+ω;θ ).(2.16)Therefore, we can find that:E [Zk−Zk−n] = H(k+ω;θ )−H(k−n+ω;θ ) (2.17)andVAR [Zk−Zk−n] = nσ2w+2σ2v . (2.18)The distribution for Zk−Zk−n under state H1 has non-zero mean, which is different fromthat under state H0. Based on the distribution differences of Zk− Zk−n under two states, wepropose Algorithm 1 to estimate σ2v and σ2w.Let the empirical mean and variance of elements in vector D be denoted as:E [D] =1MM∑i=1Di (2.19)VAR [D] =1M−1M∑i=1(Di−E [D])2 (2.20)where M > 1, M ∈ Z+, is the size of D, and Di is the ith element.180 10 20 30 40 50 60 70 80 90 100n6789101112nw2+2v210-4Emperical varianceTheoretical variance(a)5 10 15 20 25 30n6789nw2+2v210-4datafitted curve(b)Figure 2.3: Demonstration of σ2w and σ2v estimationAlgorithm 1 attempts to calculate the variance of Zk−Zk−n empirically, ∀k0 ≤ k ≤ k1, fordifferent values of n. The key point in Algorithm 1 is the step 10 where we calculate theminimum G j = min[V j]of the elements V j . From equations (2.14) to (2.18), we know thatthe empirical variance will be larger if (Zk − Zk−n) follows the state H1, compared to thatwhen (Zk− Zk−n) follows the state H0. Therefore, the calculation of the minimum varianceguarantees that the estimation of the variance is from a subsection of the stateH0, and notH1.Figure 2.3 shows the estimated nσ2w+2σ2v with respect to different n values using Algorithm1 with the real gaze data provided in [152]. The dotted line in Figure 2.3a depicts the estimatedvariance from the dataset, and the dash line in Figure 2.3a shows the theoretical variance fromfitted data points. We note from Figure 2.3a that, as n increases, the estimated value does notalways exhibit a linear increment of the same rate. That is due to the fact that measured gazeposition is in the unit of visual angle, which cannot exceed 180°. Therefore, the random walkmodel can only fit the data when n is small, and in our case, n ≤ 30. Also, we notice that theestimated value at n = 1 is an obvious outlier from the rest of the data points. Its existence isdue to the limitation of the operation min[·]. When n is very small, the value of E [Zk−Zk−nH1]is very close to E [Zk−Zk−nH0] = 0, making it harder to differentiate between two states bycalculating the min[V j]. Therefore, in practice, we only use the n value range from 2 to 10 toperform the estimation. Figure 2.3b shows the linear regression results for n = 2,3, ...,30. Theresulting R-squared score is 0.9942, suggesting that our linear model fits the data very well.19MLE for θ1, θ2 and θ3Now we know how to estimate σ2v and σ2w from the data, and based on the estimated σ2v and σ2w,we can exploit the MLE framework to estimate θ . Suppose from k0 to k1, the measured gazedata, Zk0:k1 , is at the stateH1. Based on model (2.11), we know that:Ck = Zk−Zk−1=Wk +Vk−Vk+1+Hd(k;θ ,k0)∼N(µk,σ2c), k0 < k ≤ k1(2.21)where µk = Hd(k;θ ,k0), σ2c = σ2w + 2σ2v . Hence Ck0+1:k1 follows the multivariate Gaussiandistribution, such that:Ck0+1:k1 ∼N (µ ,Σ) (2.22)whereµ =[µk0+1, ...,µk1]T (2.23)andΣ =σ2c −σ2v 0 . . . 0−σ2v σ2c −σ2v . . . 00 −σ2v −σ2c . . . 0.......... . ....0 0 0 . . . σ2c. (2.24)We have the covariance matrix Σ as in equation (2.24) due to the fact that, in our model, onlythe first order correlation is considered. Thus, all off-diagonal elements are zero except forthe elements directly adjacent to the diagonal elements. Mathematically, we can see that thecorrelation between Ck and Ck+1 is due to the common factor Vk+1. Ck and Ck+ j, j > 1, areuncorrelated as they share no correlated factor. Let k0 + 1 = k2, we can write the probabilitydensity function (PDF) of Ck2:k1 as:f(Ck2:k1θ ,H1)= det(2piΣ)−12 exp[−12(Ck2:k1−µ)T·Σ−1 (Ck2:k1−µ)] . (2.25)The log-likelihood function is:ln f(Ck2:k1θ ,H1)=−12lndet(2piΣ)− 12[(Ck2:k1−µ)T·Σ−1 (Ck2:k1−µ)] . (2.26)20From equation (2.26), we can have the maximum likelihood estimator for θ as:θˆML = argmaxθln f(Ck2:k1θ ,H1)= argmaxθ(2µ TΣ−1Ck2:k1−µ TΣ−1µ) (2.27)To find out the maximum likelihood estimator for θ1, θˆ1ML, in equation (2.27), we can solvethe equation:∂ ln f(Ck2:k1θ ,H1)∂θ1= 0 (2.28)Hence, we can find that:θˆ1ML =µ¯ TΣ−1Ck2:k1µ¯ TΣ−1µ¯(2.29)Where µ¯ = 1θ1 µ . A Detailed derivation of Equations (2.27) and (2.29) is shown in the Appen-dices.We use the Simplex algorithm [129] to estimate θˆ2ML and θˆ3ML, which is an efficient algo-rithm to find the minima without employing the function derivatives. Once we calculated theθˆ2ML and θˆ3ML, we can use equation (2.29) to find out θˆ1ML.2.3.2 Test Statistics for NPDThe detection of microsaccades, as mentioned before, in our settings, can be solved as a hy-pothesis testing problem under the Neyman Pearson detection framework. Suppose that Zk1:k2is a section of measured gaze positions from k1 to k2. Hence the detection of a microsaccade inZk1:k2 can be considered as the problem of determining the correct hypothesis betweenH0 andH1. We can therefore rephrase our detection problem as:H0 :Zk1:k2 follows the model (2.1)H1 :Zk1:k2 follows the model (2.11).Or equivalently:H0 : C = Zk1+1:k2−Zk1+1:k2−1 ∼N (0,Σ)H1 : C = Zk1+1:k2−Zk1+1:k2−1 ∼N (µ ,Σ) .where µ and Σ are the same as that in the equation (2.22). The log-likelihood function for Cunder H1 is derived in equation (2.26), and similarly, we compute the log-likelihood function21for C underH0 as:ln f (Cθ ,H0) =−12 lndet(2piΣ)−12[CTΣ−1C].(2.30)From equations (2.26) and (2.30), and according to the Neyman Pearson detection framework,we can calculate the log-likelihood ratio as [31, 215]:lnΛ= ln[f (Cθ ,H1)f (Cθ ,H0)]= µ TΣ−1C.(2.31)From equation (2.31), we know that the test statistics for NPD are:T = µ TΣ−1C (2.32)and we can then further rephrase our hypothesis testing problem as:H0 : T ∼N(0,µ TΣ−1µ)H1 : T ∼N(µ TΣ−1µ ,µ TΣ−1µ).Now, the problem of detecting a microsaccade has been transformed into the problem of deter-mining the correct distribution betweenN(0,µ TΣ−1µ)andN(µ TΣ−1µ ,µ TΣ−1µ), givenT . Then the Neyman Pearson detection theory tells us that the Neyman Pearson test can beperformed as:TH1≷H0ρ (2.33)where ρ is the threshold based on the significance value α . We can calculate ρ as:ρ =√µ TΣ−1µF−1(1−α) (2.34)where F−1(·) is the inverse cumulative distribution function (CDF) for the standard Gaussiandistribution, such that the probability of false alarm, PFA, equals α . Some common choices ofα are 0.05 and 0.01. We can also calculate the probability of detection at significant value α as:PD = 1−Fρ−µ TΣ−1µ√µ TΣ−1µ (2.35)22Algorithm 2 The NPD for microsaccades1: Get a sequence of tracked eye-gaze positions Zk0:k12: Apply Algorithm 1 to estimate σ2v and σ2w3: Pick a value for L ∈ Z+, which is the length that we want to section vector Zk0:k14: for i = k0,k0+1, ...,k1−L+1 do5: Let Z = Zi:i+L−1, which is a subsection of Zk0:k16: Calculate θˆ i by applying MLE based on data points in Z7: Calculate T using equation (2.32) using θˆ i8: Calculate ρ using equation (2.34)9: Calculate10: if T > ρ then11: Ji = 112: else13: Ji = 014: for i = k0,k0+1, ...,k1−L+1 do15: if θˆ (i)1ML < Tθ1median[∣∣∣θˆ 1ML∣∣∣] then16: Ji = 017: The NPD return the vector Jk0:k1−L+1 and matrix θˆ k0:k1−L+118: Post process Jk0:k1−L+1 and θˆ k0:k1−L+1 and yield detected saccadic eventswhere F(·) is the CDF for the standard Gaussian distribution. The optimality of NPD, accordingto the Neyman Pearson lemma, guarantees that NPD with threshold in equation (2.33) is themost powerful test given PFA.2.3.3 NPD AlgorithmIn the two previous sections, the necessary components to detect a microsaccade were derived.In this section, we will combine the proposed parameter estimation methods with NPD.The proposed NPD algorithm is shown in Algorithm 2. From Algorithm 2, we know thatwe apply a sliding window of length L on the data vector Zk0:k1 , and treat the windowed datapoints as either in state H0 or H1 to perform the Neyman Pearson hypothesis tests. Steps 14through Step 18 are needed to post process the detection results. The aim of the added steps isto rectify the problem of overfitting and we explain these steps in the following paragraphs.The MLE Algorithm is used regardless of whether the windowed data points are in stateH0 or H1. Therefore, over-fitting can occur if we arbitrarily fit data points in state H0 to themodel of stateH1 using MLE. Such over-fitting can also increase the probability of false alarm;indeed, we notice that when θ1 is very close to 0, the difference between H0 and H1 is verysmall. One extreme condition is thatH0 =H1, when θ1 = 0, reducing the proposed detectionmethod to a random guess. To avoid the over-fitting and rectify the increased probability of falsealarm, a thresholding is needed for the parameter θ1. Figures 2.6b and 2.6d show the estimated23θ1 for a real gaze signal. Since θ1 is estimated using state modelH1, we are arbitrarily fittingthe windowed data points to follow a saccadic eye movement. We can see from Figures 2.6band 2.6d that the absolute amplitude θ1 of θ1 is large when a true saccadic event has occurredand θ1 is small if there is no saccadic event. Also, based on the fact that the state H0 is morecommonly observed than the stateH1 (the saccadic events only occur a few times per second),we can apply an adaptive threshold to θ1 in order to solve the problem of over-fitting. Bycalculating the median of all the estimated |θ1|, median[∣∣∣θˆ 1ML∣∣∣], where θˆ 1ML is a vector ofall estimated θ1 of the input sequence Zk0:k1 , we can estimate the median value of θ1 whenno saccadic event has occurred. By multiplying a factor Tθ1 , we can reject the hypothesis H1regardless of T and ρ if θˆ1ML ≥ Tθ1median[∣∣∣θˆ 1ML∣∣∣]. The aforementioned reasoning explainsthe steps 13 through 15 in Algorithm 2, and θˆ (i)1ML stands for the ith element in θˆ1ML .The Step 18 post processing is necessary due to the fact that the hypothesis testing is sta-tistically accurate if the testing data either follows H0 or H1. However, given the windowingmethod as in Algorithm 2 Step 5, we cannot guarantee that the vector Zi:i+L−1 precisely followsstateH0 orH1. We usually pick the value L to around the maximum possible duration of a mi-crosaccade. Yet, we cannot choose a value too large, otherwise two or more microsaccades canbe entirely enclosed in the vector Zi:i+L−1. The five common conditions that we can encounterare listed below:(i) Zi:i+L−1 followsH0(ii) An entire microsaccade is enclosed in Zi:i+L−1 and the microsaccade start at time j = i(iii) An entire microsaccade is enclosed in Zi:i+L−1 and the microsaccade start at time j > i(iv) A section of a microsaccade is enclosed in Zi:i+L−1 and the microsaccade start at timej < i(v) A section of a microsaccade is enclosed in Zi:i+L−1 and the microsaccade end at timej > i+L−1The conditions (i) and (ii) are ideal scenarios, such that the two state model captures the be-havior of the data points flawlessly. However, for conditions (iii), (iv) and (v), the data pointsonly partially fit our model, and the resulting behavior of the detectors need to be analyzedindividually under simulation. By using both simulated data as well as actual data, we find thatunder conditions (iii) and (v), NPD tends to rejectH1 and acceptH0. For condition (iv), NPDalmost always accept H1 as a sectioned microsaccade as condition (iv) also fits the model inH1, yet the resulting estimated amplitude is reduced. Based on the behavior of NPD, we canperform the post processing of Jk0:k1−L+1 and θˆ k0:k1−L+1 as follows: 1) Apply a median filter of24strength M to Jk0:k1−L+1, such that only consecutively detected events are kept. 2) In each of theconsecutive all 1 regions, find the starting time of the microsaccade based on the position thatyields the minimal false alarm probability, which can be calculated as:PFA = 1−F(√T). (2.36)To compute the MLE of θ efficiently, the search of candidate parameter pairs (θ2,θ3) can beaccelerated if we impose more constraints on (θ2,θ3). One such constraint is that the parameterpair satisfies a maximum velocity and occurs within a specific duration range. These constraintson (θ2,θ3) significantly accelerate the computation.2.4 ResultsIn this section, results from simulated data as well as actual data are presented. We start bytesting the parameter estimation accuracy of the MLE method for θ , as well as for vmax, tv andd, using simulated data where we know the ground truth. The receiver operating characteristic(ROC) curve of the NPD with simulated data is also presented. Then, we compare the proposeddetector with the state-of-the-art CNN method, in terms of the area under the curve (AUC) ofthe ROC plot, and also in terms of the measurement sensitivity index. We then compare theNPD and the BIM methods to evaluate parameter estimation accuracy in terms of the linearityof the main sequence.2.4.1 Test with Simulated DataThe aim of this subsection is to test our MLE and NPD method with known parameters andground truth. In our models, we have 5 parameters σ2v , σ2w, θ1, θ2 and θ3. A thorough test of allcombinations between these parameters is not practical. Therefore, we exploit the measurement,signal-to-noise ratio (SNR), to capture the joint behavior of all parameters. The SNR, in mostscenarios, directly affects the performance of statistical estimation and detection.Data generation is the first step before testing our model. We set k0 = 0, k1 = 100, θ1 = 1,θ2 = 40, θ3 = 40 and σ2w = 0.0049. The SNR is controlled by σ2v . We pick three values forσ2v , which are: 0, 0.01 and 0.09. Indeed σw and σv were defined earlier, in equation (2.1) but,as a reminder, it makes sense to fix the parameter σw as it relates to an intrinsic oculomotorprocess and this value captures it well. Note the value of σw is only fixed in this subsection forsimulation, and is estimated from real data for the detection tasks. However, we can change thevalue of σ2v based on the eye tracking technology we are using. The resulting SNR with σ2v =0, 0.01 and 0.09, are 23dB, 18dB and 10dB respectively. Based on the specified parametersettings, for each σ2v , we generate 2000 independent trials. Among the 2000 trials, we randomlyset each trial to follow the stateH0 orH1 with equal probability.25Parameters/SNR 23dB 18dB 10dBθ1 3.82×10−2 5.25×10−2 1.51×10−1θ2 6.15×10−2 2.10×10−1 10.14θ3 186.89 594.62 1527.07d 1.92 13.04 172.02vmax 7.40×10−3 3.00×10−2 1.57×10−1tv 6.23×10−2 4.18×10−1 15.67Table 2.1: MSE of MLE estimated parametersWe first test our Algorithm 2 with all parameters provided to the algorithm. Hence, theresulting detection is theoretically optimum under the Neyman Pearson detection framework.Figure 2.4 shows the resulting empirical and theoretical distributions for test statistics T understateH0 andH1. We can see that in Figure 2.4a, such that SNR = 23dB, the two distributionsof T under state H0 and H1 are highly separable, by picking a threshold at the value of 25.However, with deceases of SNR, such as in Figure 2.4c, we can observe an overlap between thetwo distributions. Hence, some trade-offs are needed before we apply a threshold to differentiatethe two distributions. The ROC curves for the detection results are shown in Figure 2.5. We cansay that a detector is better if its corresponding ROC curve is further away from the diagonaldotted line (chance line) in Figure 2.5a, which represents the random guess. From equations(2.34) and (2.35), we can calculate the theoretical ROC curves; both empirical curves (solidlines) and theoretical curves (dashed lines) are plotted in Figure 2.5. We can see that NPD is veryrobust to noise, and all empirical ROC curves are very close to the theoretical values. Figure2.5b is the zoomed plot of Figure 2.5a to show the performance of the NPR when α ∈ (0,0.025].With the same simulated dataset, we then test our MLE method for estimating θ as wellas d, tv and vmax. Only trials with state H1 are used to perform the estimation of θ . Thenbased on estimated θ , we employ equations (2.9), (2.10) and (2.6) to estimate tv, vmax and drespectively. We measure the performance of the estimator via the mean squared error (MSE).Table 2.1 shows the parameter estimation results, where MSE for θ1, θ2, and θ3 are unit free,MSE for d and tv have unit of ms2, and MSE for vmax has unit (°/ms)2. We can see from Table2.1 that the estimation of θ3 have the largest inaccuracy. However, despite the large inaccuracyin the estimated θ3, the estimated d, vmax and tv are still accurate.2.4.2 Test with real dataIn this subsection, we demonstrate the outputs of our detector using real data, and we performa comparison between the NPD with the CNN method and the BIM. For the CNN method, thestate-of-the-art, we aim to compare the detector performance in terms of the AUC of the ROCplot. For the BIM, the best known statistics based approach, we aim to compare the parameter26(a) SNR = 23dB(b) SNR = 18dB(c) SNR = 10dBFigure 2.4: The distribution of statistics T with different SNRestimation accuracy (maximum velocity and amplitude), through the main sequence.Demonstration of NPD outputsThe output of the NPD given a trial of inputs (X and Y components of the eye gaze position)consists of the following components: 1) estimated θ for each windowed subsection of the inputdata points; 2) detection result (a binary sequence, where 0 indicates the absence of a saccadicevent, and 1 indicates the existence of a saccadic event); 3) the best fitted θ1, θ2 and θ3 for eachdetected event and the corresponding parameters, i.e., t1: the starting point of the event; t2: theend point of the event; d: duration of the event; vmax: maximum velocity of the saccadic eyemovement; and tv: the time index of reaching vmax.270 0.2 0.4 0.6 0.8 1PFA00.20.40.60.81P DSNR = 23dBSNR = 18dBSNR = 10dBSNR = 23dBSNR = 18dBSNR = 10dB(a) Full ROC0 0.005 0.01 0.015 0.02 0.025PFA0.9750.980.9850.990.9951P DSNR = 23dBSNR = 18dBSNR = 10dBSNR = 23dBSNR = 18dBSNR = 10dB(b) Zoomed ROC plotFigure 2.5: ROC curves for Algorithm 2 with known parametersSince most existing detection algorithms only produce the binary detection results, ourmethod is more immediately informative about the detected saccadic eye movements. Withall the estimated parameters for each of the detected saccadic events, we can plot the noise-freesaccadic eye movement on top of the noisy eye-gaze position measurements. Figure 2.6 demon-strates some of the NPD outputs. Figures 2.6a and 2.6c show the X and Y components of theraw gaze signal (purple lines); we can observe two saccadic events. The bold lines (blue andred) reproduce the saccadic eye movement (noise-free) based on the estimated θ . The circlesand crosses in Figures 2.6a and 2.6c indicate the starting and ending points of the event basedon the definition discussed in Section: Statistical Modeling of Gaze Data. The stars indicate thepositions of maximum velocity and their corresponding saccadic events. We also calculate thevalue of the maximum velocity, and we indicate vmax in Figures 2.6a and 2.6c. Figures 2.6b and2.6d show the estimated θ1. We can see that the estimated θ1 does not depend on the driftingmovement of the eye, as it reverts around the zero line when no saccadic event has occurred.28500 550 600 650 700 750 800 850 900 950Discrete time index k-0.4-0.200.20.40.6X position (degree)-22.8o/s67.4o/sSaccadic event 1Saccadic event 2start pointend pointmax velocitygaze signal (X)(a) Output X500 550 600 650 700 750 800 850 900 950Discrete time index k-0.4-0.200.20.40.60.81AmplitudeEstimated 1zero line(b) Estimated θ1 (X)500 550 600 650 700 750 800 850 900 950Discrete time index k-0.3-0.2-0.100.10.2Y position (degree)25.9o/s 28.8o/sSaccadic event 1Saccadic event 2start pointend pointmax velocitygaze signal (Y)(c) Output Y500 550 600 650 700 750 800 850 900 950Discrete time index k-0.100.10.20.30.4AmplitudeEstimated 1zero line(d) Estimated θ1 (Y)Figure 2.6: Demonstration of the NPD outputComparison with the CNN methodThe CNN method is the state-of-the-art approach in detecting saccadic events, having “human-level performance” [14]. The comparison is performed by using the public dataset in [14], andTable 2.2 summarizes the data information for testing the performance of the NPD as com-pared to the CNN. We can see that for datasets A, C and D, the EyeLink 1000 gaze tracker isused and for the dataset B, the search coil is used to track the gaze position. It is worth notingthat the dataset B is very different from the rest of the datasets, not only due to the differencein tracking method, but also due to the difference in the underlying eye movement. Catch-upsaccades (saccades during smooth pursuit) are the main saccadic event in the dataset B. Thecatch-up saccades exhibit different characteristics from the saccades at fixation, and thereforethey are outside the application scope of our proposed detection method. However, the detec-tion results are also presented with dataset B for reference purposes. All saccadic events in thedataset are manually labelled by experts. Thus, we can plot the ROC curve to demonstrate thedetection results. Among the four datasets, we exclude all trials that have missing values inthe measurements, which may be due to the temporary loss of eye features while tracking. Wealso discard trials where a saccadic event is truncated at the beginning or end of the trial. Forthe CNN method, we use the online server provided by [14] to perform the detection. Amongseveral of the already generated weights provided by [14] for the CNN, we select the pretrained“combined weights”, which results from the largest number of training data. More specifically,the “combined weights” are trained using the data collected during the same experiments that29collected dataset A, B and C. The data trials for training and testing are mutually exclusive.For our NPD, we randomly choose values for L, M, α , and Tθ1 given an entire dataset. Somecombinations of the parameters are superior to others, and the ROC curve is plotted by pickingthe upper envelope of the PD, PFA value pairs. Based on the NPD performances by using therandomly selected parameters, we acquired insights regarding the proper choices of these pa-rameters. The proper choices of parameters for all the datasets share the same properties, andwe discuss it in Section: “Discussion and Conclusion”.Figure 2.7 shows the ROC curves (full ROC curves on the left, and zoomed curves forPFA ∈ [0,0.2] and PD ∈ [0.48,1] on the right) of the detection results for each dataset, and Table2.3 summarizes the AUC for each detectors. We can see that other than the dataset B, the NPDyields larger AUC value than that of the CNN method. Therefore, we conclude that our NPDoutperforms the CNN method when the dataset does not contain catch-up saccades. BesidesAUC, we also use the measurement sensitivity index, denoted as d′, to compare the detectorperformance. d′ reflects the discriminating ability of the target signal with contamination noise[143]. A larger d′ represents a stronger discriminating power of the detector. Table 2.3 showsthe sensitivity index of each detector for all datasets. The results presented in Table 2.3 agreewith the results in Table 2.3.The reasons behind the performance differences between the two detectors can be analyzedas follows. With dataset B, based on the fact that our statistical testing is constructed withoutconsidering smooth pursuit movement, we expect to observe inferior performance. In that sense,the CNN is more versatile in detecting different kinds of saccadic events. However, for theCNN, we can notice that the AUC and the sensitivity index are relatively stable across datasetsA, B and C. But for dataset D, the performance of the CNN drops substantially, as reflectedby the much smaller AUC and sensitivity index values. The CNN exhibits poorer performanceon the dataset D because the experiments in D were not included in the training dataset. Thisis consistent with poorer performance of CNNs for unseen datasets, as expected for such data-driven approaches. The NPD, on the other hand, is relatively stable across different datasets (A,C and D), as indicated by similar AUC and sensitivity index values. The stable performanceof the NPD on different datasets A, C and D partially results from the structured algorithmsteps: the parameter estimation step and then the Neyman-Pearson decision making step. Theparameter estimation step is explicitly included in our NPD algorithm for the handling of datadifferences. That is, we can parameterize data differences in terms of σw, σv and θ beforeproducing the detection outcome.Comparison with the BIMWe now perform the comparison with BIM using the dataset in [152]. It contains 5 sequencesof tracked gaze positions from 5 participants, and Table 2.4 shows the detail. In the Data info30Dataset Dataset A Dataset BSubject Human MonkeySampling Frequency 1000 Hz 1000 HzGaze tracker Eyelink 1000 Search CoilSaccade typesMicrosaccade & memorysaccadesSaccades during smoothpursuitData size 1000 trials of 1 sec each 1000 trials of 1 sec eachDataset Dataset C Dataset DSubject Monkey HumanSampling Frequency 500 Hz 1000 HzGaze tracker Eyelink 1000 Eyelink 1000Saccade types MicrosaccadesMicrosaccade & MemorysaccadesData size 53 trials of 1.468sec each 3000 trails of 0.75sec eachTable 2.2: Summary of data information (for testing) in [14].Detector AUC (CNN method) AUC (NPD) d′ (CNN method) d′ (NPD)Dataset A 0.9341 0.9929 2.1314 3.4641Dataset B 0.9545 0.9171 2.3908 1.9597Dataset C 0.9484 0.9840 2.3045 3.0331Dataset D 0.8759 0.9917 1.6330 3.3849Table 2.3: AUC and sensitivity index for each dataset of the CNN (with pretrained “com-bined weights”) and NPD methods.column in the Table 2.4, the number indicates the participant number, and the letter “X” or“Y” indicates the horizontal or vertical component of the gaze position, respectively. Followingthe convention used in [146], such that the main sequence of detected microsaccade eventsare plotted to guarantee the validity of the detected events, we also plot the main sequence asa testing criteria for the parameter estimation accuracy for the detected microsaccades. Themain sequence is known as the log-linear relationship between the amplitude and the maximumvelocity of microsaccades events [251]. Figure 2.8 shows the main sequence of the detectedmicrosaccades using both detection methods, where R denotes the R-squared score for linearregression. A detailed detection result is summarized in Table 2.5, where N stands for thenumber of events detected under different detection methods, and Na is the adjusted numberof detected events for NPD. Na is calculated as the total number of microsaccades of both eyesminus the number of binocularly occurring microsaccades of one individual, such that it reflectsthe same quantity as calculated by N (BIM). Due to the high noise level of dataset 4H through5V , as reflected by large σˆv and σˆw compared to the rest, the NPD cannot detect any event.31Data info Sample frequency Data length σˆv σˆw1X 1000Hz 78167 0.0195 0.00321Y 1000Hz 78167 0.0241 0.00322X 1000Hz 78302 0.0139 0.00412Y 1000Hz 78302 0.0157 0.00473X 1000Hz 59366 0.0163 0.00393Y 1000Hz 59366 0.0168 0.00504X 1000Hz 78523 0.0214 0.00454Y 1000Hz 78523 0.0227 0.00475X 1000Hz 78122 0.0606 0.00665Y 1000Hz 78122 0.0997 0.0090Table 2.4: Data information in [152] and NPD parameter estimatesMeasurements Subject 1 Subject 2 Subject 3 Subject 4 Subject 5R (BIM) 0.4421 0.4425 0.4496 0.4496 0.1325R (NPD) 0.8791 0.8090 0.9276 NaN NaNN (BIM) 15 12 15 15 7N (NPD) 25 4 11 1 0Table 2.5: Summary of the detection results.Therefore, the main sequence from NPD for participants 4 and 5 cannot be plotted for thecomparison with BIM.By observing the results shown in Table 2.5 for NPD and the estimated σv and σw from eachdataset, we can notice that NPD can detect more events when noise is small. That is due to thefalse alarm probability constraint imposed within NPD once we set α . When noise is huge (σvand σw are large), even the true saccadic events may not pass the test, thus reducing the totalamount of detected events for NPD. The advantage of this mechanism is that it guarantees thatall detected events are reliable at a significance level α . However, this may lead to the decreaseof the probability of detection as noise level increases. To counteract the decrease of PD, we canincrease the value of α , at a cost of increasing PFA. BIM, on the contrary, does not encounterthis kind of problem, as it can detect more or less the same amount of events regardless of thenoise level. But, we cannot know to what extent that a detected event from BIM is a false alarm.2.5 SummaryIn this chapter, we focus on Obj 1, and develop a statistical model for gaze signals. To validatethe model, we performed a detailed study of the microsaccade detection, where a novel statis-tical model for fixational eye movement is built, and the Neyman Pearson detector is derived32based on the model. When compared to the state-of-the-art CNN method, our NPD yields alarger AUC value, implying that for certain datasets our NPD can better detect saccadic eyemovements. As for the comparison with the BIM method, we claim that our model is statis-tically more accurate, and the detected events are less likely to be false alarms in comparisonto the BIM. The results also suggest that our NPD can detect both the microsaccades and thenormal saccades.On the one hand, based on the simulation and testing with read datasets, there are still pointsof improvements in our NPD. The first is that the computational complexity is high. In the de-veloped algorithm of the NPD, the OpenMP library is used to enable parallel computing forcomputation acceleration. More advanced optimization or approximation methods are neededto further reduce the computational burden of the NPD. The second comes from the tuning ofthe parameters of the NPD. As a parametric method, the performance of the NPD relies on theproper choice of the parameters (i.e., M (the strength of the median filter), L (the section lengthfor detection), α and Tθ1). When inputting datasets acquired at different sampling frequencies,or with different measurement noise, we usually need to adjust some of the parameters in theNPD. A set of inappropriately selected parameters of the NPD may hinder the optimal perfor-mance of the detector. Based on empirical evidence, in Table 2.6, we summarize the properchoices of these parameters for the dataset with 1000Hz sampling rate. For datasets of differentsampling rates, the choices for M and L can be adjusted proportionally to the sampling ratedifference. Finally, when dealing with data with missing values, or when catch-up saccades arepresent in the dataset, the NPD is less reliable. To extend our NPD to a method that is capableof dealing with more application scenarios, a more complex statistical model needs to be devel-oped. The length of the dataset is also worth mentioning. Since we need to perform parameterestimation for σv and σw based on the data itself, a longer (more than 1000 data points) trial ofgaze signal is preferred for parameter estimation accuracy.On the other hand, compared to previous detection methods in the literature, our approachhas the following advantages. First of all, it is a statistical method and therefore, all estimationsand detection results are statistically valid and accurate. With the Neyman Pearson lemma, wecan have a confidence bound for our detection results given α . The second is the online appli-cability. Unlike the BIM or CNN method, which cannot be implemented online, our detectionmethod is strictly causal, and the detection can be made once a microsaccade has finished. Onemodification needed based on our proposed algorithm is that we need to introduce a calibrationphase at the beginning for the estimation of σv, σw, and median[|θ 1ML |]. Once the estima-tion is finished, Algorithm 2 can be used online, for the purpose of HCI or for more advancedcognitive studies and research. Last but not the least, the NPD can inform the user with amore complete profile of the detected events. The associated characteristics of a saccadic event,including duration, amplitude, and maximum velocity, are provided to the user.33Parameters SuggestedvalueRemarksL 20While L = 20 is the suggested value for our NPD, the possiblerange of L is between 15 to 35. For L values within the givenrange, the detector exhibits a stable performance (PD > 0.9 andPFA < 0.1). However, the computational speed of our NPD is pro-portional to the value L, such that the smaller the L value is, themore efficient our NPD can detect saccadic events. The drawbackof picking a small value of L is that the estimation accuracy of θ1can be affected, as we can only get a section of an entire saccadicevent for estimation. The estimation bias of θ1 is approximatelyproportional to the difference of the truncation size (L) and the ac-tual duration of a saccadic event (d), d−L, if L < d.M 16While M = 16 is the suggested value for our NPD, and the possiblerange of M is between 6 to 25. For M < 6, we can observe a highfalse alarm probability (PFA > 0.1). For M in the range between 6and 25, the detector exhibits a stable performance (PD > 0.9 andPFA < 0.1). But when M > 25, we can notice a decrease in the res-olution of our NPD (two consecutively occurring saccadic eventscannot be distinguished from each other).α 1.0×10−4 Due to data overfitting, it is not recommended to pick α > 0.01, asthis may lead to a high false alarm probability PFA > 0.1.Tθ1 4.0As the factor for the threshold of the detector, the detection re-sult is sensitive to this parameter. The detector performance re-mains approximately the same (PD > 0.9 and PFA < 0.1) if we setTθ1 ∈ [3.5,4.5]. When Tθ1 < 3.0, such that we are applying a weakthreshold, our algorithm is more sensitive to noise. Thus, we oftenget a high false alarm probability (PFA > 0.1). When Tθ1 > 5.0,such that we are applying a strong threshold, our algorithm is morerobust to noise. But, we are imposing too strong a threshold suchthat saccadic events with small amplitudes cannot be detected, re-sulting in a low detection probability (PD < 0.9).Table 2.6: Suggested parameter ranges based on empirical experience (for gaze signalsampled at 1000Hz).340 0.2 0.4 0.6 0.8 1PFA00.10.20.30.40.50.60.70.80.91P DCNN, AUC = 0.93411NPD, AUC = 0.99285chance line(a) Full ROC dataset A0 0.05 0.1 0.15 0.2PFA0.50.550.60.650.70.750.80.850.90.951P DCNN, AUC = 0.93411NPD, AUC = 0.99285chance line(b) Zoomed ROC plot dataset A0 0.2 0.4 0.6 0.8 1PFA00.10.20.30.40.50.60.70.80.91P DCNN, AUC = 0.95454NPD, AUC = 0.91708chance line(c) Full ROC dataset B0 0.05 0.1 0.15 0.2PFA0.50.550.60.650.70.750.80.850.90.951P DCNN, AUC = 0.95454NPD, AUC = 0.91708chance line(d) Zoomed ROC plot dataset B0 0.2 0.4 0.6 0.8 1PFA00.10.20.30.40.50.60.70.80.91P DCNN, AUC = 0.94840NPD, AUC = 0.98401chance line(e) Full ROC dataset C0 0.05 0.1 0.15 0.2PFA0.50.550.60.650.70.750.80.850.90.951P DCNN, AUC = 0.94840NPD, AUC = 0.98401chance line(f) Zoomed ROC plot dataset C0 0.2 0.4 0.6 0.8 1PFA00.10.20.30.40.50.60.70.80.91P DCNN, AUC = 0.87589NPD, AUC = 0.99166chance line(g) Full ROC dataset D0 0.05 0.1 0.15 0.2PFA0.50.550.60.650.70.750.80.850.90.951P DCNN, AUC = 0.87589NPD, AUC = 0.99166chance line(h) Zoomed ROC plot dataset DFigure 2.7: ROC curves of NPD and CNN for each dataset35-5 -4 -3 -2 -1log amplitude (degree)-5.5-5-4.5-4-3.5-3-2.5-2-1.5-1-0.5log max velocity (degree/msec)DataRegression Line(a) Subject 1 BIM, R = 0.4421-4 -3 -2 -1 0log amplitude (degree)-6-5-4-3-2log max velocity (degree/msec)DataRegression Line(b) Subject 2 BIM, R = 0.4425-3 -2.5 -2 -1.5 -1log amplitude (degree)-5.5-5-4.5-4-3.5-3-2.5-2-1.5log max velocity (degree/msec)DataRegression Line(c) Subject 3 BIM, R = 0.4496-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2log amplitude (degree)-3.4-3.3-3.2-3.1-3-2.9-2.8-2.7-2.6-2.5-2.4log max velocity (degree/msec)DataRegression Line(d) Subject 1 NPD, R= 0.8791-1.5 -1 -0.5log amplitude (degree)-3.6-3.5-3.4-3.3-3.2-3.1-3-2.9-2.8-2.7log max velocity (degree/msec)DataRegression Line(e) Subject 2 NPD, R = 0.8090-1.2 -1 -0.8 -0.6 -0.4 -0.2 0log amplitude (degree)-3.4-3.2-3-2.8-2.6-2.4-2.2log max velocity (degree/msec)DataRegression Line(f) Subject 3 NPD, R = 0.9276Figure 2.8: Main Sequence with NPD and BIM36Chapter 3Survey on Human-computerInteraction Methods for UltrasoundMachines3.1 IntroductionThe design and functionality of ultrasound (US) machines have experienced significant ad-vancements in the past decades. With the goal of improving user experience and system effi-ciency when performing US scans, research was conducted from many perspectives [7]. Forinstance, in [64, 69, 75], emphasis was placed on the understanding of what constitutes er-gonomic US scan protocols and operator behaviors. Machine design ergonomics were alsostudied, including US machine lighting conditions [130], machine layout design [16], controlpanel design [10], and chair selection [224]. In computer-assisted interventions, research hasalso been proposed for more intelligent and efficient human-computer interaction (HCI) withUS machines. Voice control [173], and eye-gaze contingent machine control [248, 249] wereproposed as alternatives to conventional control panel-based machine interaction. Moreover,scan automation, an emerging tool in US medical examinations, has been pervasively integratedinto US machines. It has been shown that scan automation methods, i.e., [142] by Samsung,and [184, 202] by Philips, can reduce workload and expedite US scans.Among the existing research to improve the HCI for US machines, the logical structureof HCI, which impacts the interoperability, efficiency and ease-of-use, is largely neglected.Therefore, in this chapter, we take a new approach and focus on the analysis of the controllogic employed by different cart-based US machines. With our study, we intend to answer thefollowing questions:1. What HCI control logic is currently implemented in the common functions of cart-based37US machines?2. What are its merits and drawbacks? and3. How can we improve the HCI control logic for more efficient and user-friendly machinecontrol?To answer these questions, we surveyed sonographers to understand the frequency of usingUS machine functions. For logical analysis, we propose the use of the unified modelling lan-guage (UML), a flexible and scalable graphical representation of logic and semantics [18]. Incomparison to other graphical diagrams that represent HCI control logic, such as the Buxton’sthree-state model [23] and finite-state machines [39], UML diagrams introduce super states,which allow for more compact and expressive representation of the logic or semantic structuresbeing described [58, 71]. The UML diagram representation of HCI control logic is independentof the machine layout and internal signal handling, facilitating for effective logical abstractionand visualization. With standardized UML representation, we showed that the advantages anddeficiencies of control logic on both conventional and novel/intelligent HCIs can be analyzedfor improvements.The rest of the chapter is organized as follows. In Section 3.2, we present survey details andinvestigation procedures for different US machines. Survey results are presented in Section3.3. Then, in Section 3.4, we establish a structured UML diagram representation for the controllogic in US machines, and provide the UML diagrams for the most frequently used US machineoperations. Next, in Section 3.5, we summarize the commonly adopted control logic functionsand evaluate their impact on machine efficiency and user experience. In the same section wealso demonstrate that UMLs can be used to improve the logically sub-optimal HCI methods andanalyze novel/intelligent HCI. We conclude our chapter in Section 3.6.3.2 MethodsIn this section, we present the main methods used in this study. Firstly, to understand thesonographers’ daily handling of US machines, field observations, interviews and a survey wereconducted. Secondly, for the analysis of control logic, we investigated US machine models.Finally, we used the UML diagrams to analyze and visualize the HCI control logic. Our studyfollowed protocols and consent that were approved by the Behavioural Research Ethics Boardof the University of British Columbia.3.2.1 Field observationWe observed 23 US scans in two hospitals in Vancouver, Canada, which include general, obstet-rics and gynecology (OB/GYN), cardiac, vascular, breast, abdominal, and renal US exams. The38details of the observation procedures and outcomes are presented in [77]. The scan protocol,US machine used, and personnel in different US exams vary significantly, yet it is possible todiscover and rigorously describe common practice.3.2.2 Survey and interviewWe conducted our survey among members of the British Columbia Ultrasonographers’ Society(www.bcus.org), a professional association of Diagnostic Medical Sonographers in Canada. Inthe span of a three-month period, we collected 66 survey responses, 48 (73%) of which werecomplete. We collected the following information through the survey:Demographic characteristics of survey respondentsWe collected the respondents’ number of years of experience and specialization to check if theyrepresent the general sonographer community.Usage frequency of US machine operations and functionsThere are hundreds of operations and settings available for controlling a cart-based US machine.However, only a fraction of these are typically used by sonographers during a single US scan.We intend to find out which functions are more frequently used than others. In this chapter,we do not differentiate machine settings, such as B-mode gain and US transducer transmitfrequency, from the US scan task related operations, such as distance measurement and colorDoppler. We term both the machine settings and US scan task related operations as “functions”.In the survey, sonographers were asked to classify the US machine functions they can think ofinto the following three categories:• frequently used functions (used in more than 90% of all scans)• less frequently used functions (used in 40% and 90% of all scans)• infrequently used functions (used in fewer than 40% of all scans).This categorization with reference usage percentages helps a survey respondent to qualitativelyrank the usage frequencies for each function according to their own experience.Casual discussions and formal interviews were conducted throughout our study with two se-nior sonographers, each with more than 30 years experience, for the design of survey questionsand for the confirmation of survey-based findings. The first sonographer primarily specializedin OB/GYN US scans and the second sonographer primarily specialized in general US scans.What is more, we talked with 4 additional sonographers regarding their perspective on new HCIdesigns for US machines, and their opinions about our study’s outcomes.39Pointing and Selectingtrackball operationPointing Selectingbuttom pressingstatestart endFigure 3.1: UML diagram example: Pointing and Selecting3.2.3 US machine investigationTo understand the control logic employed on different US machines, we analyzed 10 differentUS machines. The US machines investigated are listed in Table 3.1. Among the 10 machines,3 were manufactured by BK Medical; 2 were manufactured by Philips; 2 were manufacturedby GE Healthcare; 2 were manufactured by Siemens; and the last by Exact Imaging. The yearsof manufacture span from 2001 to 2018. We use the machines listed in Table 3.1 as a repre-sentation of the commonly used HCI designs for conventional mid- to high-end US machinesfrom major manufacturers, most of which are sold worldwide [66]. All other machines couldbe described in a similar manner. This analysis is a continuation of our preliminary studyperformed in [248], and we added more machine models in this study.3.2.4 UML diagramWe use the UML diagram to help visualize the HCI design and logic. A UML diagram typicallyconsists of states, which are steady conditions of a machine or a system, and transitions, whichmark the connections between states. By using super states, which contain multiple intercon-nected sub states, UML enables high-level abstraction and compact expressiveness of complexlogic and semantics [18].Figure 3.1 shows a UML diagram example. We use a rounded square to represent a state(or a super state); the name of the (super) state is noted at the top of the rounded square abovea horizontal line. A super state usually contains several functionally related sub states. Thereare three states in Figure 3.1, where the state “Pointing and Selecting” is a super state, contain-ing sub states “Pointing” and “Selecting”. Possible transitions between states are denoted byarrows, and the transition conditions are marked beside the arrow. If no transition condition isshown, then the transitions between the two states are unconditional. If no arrow is presentedbetween two states, it means that a direct transition between these states is not possible. Thefilled dark circle stands for the starting point of the diagram, and the concentric circles with theinner circle filled represents the end of the diagram. While black is used for the main body of aUML diagram, blue text and arrows are used for annotations.40ID # Brand Model Manufacture year1 GE Voluson 730 20012 BK medical SonixTOUCH 20023 BK medical SonixRP 20054 Siemens ACUSON cypress 20065 Siemens ACUSON S2000 20116 GE LOGIQ S7 20127 Philips iU22 20148 BK medical bk3500 20169 Philips Hd5 201810 Exact Imaging ExactVu 2018Table 3.1: US machine details3.3 Survey Results3.3.1 Demographic characteristics of survey respondentsDemographic statistics are presented in Figure 3.2. A majority of survey respondents are fe-male (85%). As a comparison, we also present the demographic characteristics of sonographersin British Columbia (BC), Canada, and USA. Statistics are collected from the Health SciencesAssociation of BC [161] and Data USA [212], respectively. It shows that the demographic char-acteristics of our survey participants is aligned with the statistics from the larger populations.The respondents’ experience as sonographers is summarized in Figure 3.3a. We note that amajority of respondents (56%) have more than 20 years experience.We also collected the respondents’ US scan specializations. Our survey reached out tospecialists in general (42 responses), OB/GYN (41 responses), vascular (32 responses), mus-culoskeletal (MSK) (27 responses), cardiac (17 responses), breast (3 responses), neonatal (2responses), neural (1 response) and ocular (1 response) scans. Many sonographers have mul-tiple specializations, and 75% of the survey participants have three or more specializations.The presence of multispecialty and senior survey participants helps reduce biases in the surveyresults as they are less biased towards a single type of US scan.3.3.2 Usage frequency of US machine operations and functionsWe instructed the survey respondents to consider, among the functions the survey respondentsare aware of, the functions that are actually used during a US scan.Figure 3.3b shows that 54% of the survey participants report they typically use from 30%to 60% of the available functions they are aware of during a single US scan. We present thetop 10 most commonly used functions in each category in Figure 3.4. We use the Kendall tau4110% 90%19% 81%15% 85%0 20 40 60 80 100Percentage (%)Sonographers in the USASonographers in BCSurvey participantsMale Female(a) Sex distribution20-29 30-39 40-49 50-59 60+Age group010203040Percentage (%)Survey participantsSonographers in BCSonographers in the USA(b) Age distributionFigure 3.2: Demographic characteristics of survey respondents as compared to that fromsonographers in British Columbia (BC), Canada, and in the USA10%13%8%13% 33%23% Less than 2 years2 - 5 years6 - 10 years11 - 20 years20 - 30 yearsMore than 30 years(a) Years of experiences27%54%19%10% - 30%30% - 60%More than 60%(b) Function usage frequencyFigure 3.3: Survey result statisticscorrelation measurement with statistical tests to check the correlation between usage frequencyrankings in different categories [115]. Table 3.2 shows the pairwise results, where p-values arecalculated based on the null hypothesis that correlation is absent. The result shows sonogra-phers were consistent in naming the frequently and less frequently used US machine functions,whereas different sonographers usually report different infrequently used functions, as reflectedby their large p-values (accepting null hypothesis if p-value> 0.05) in Table 3.2. The disagree-ment on some of the frequently used functions, such as focus, time gain compensation (TGC)and color Doppler is likely due to the different areas of specialization of different sonographers.42Function category 1 Function category 2 Correlation p-valueFrequently used Less frequently used 0.286 0.004Frequently used Infrequently used −0.091 0.369Less frequently used Infrequently used 0.041 0.688Table 3.2: Kendall tau correlation and statistical test results between pairs of function us-age frequency categories.For instance, a cardiac sonographer tends to classify color Doppler as a frequently used functionwhereas an obstetric sonographer would use color Doppler less frequently.To quantify function usage frequency based on categorized feedback, a scoring system isapplied. Based on the fact that functions that have been mentioned more frequently in thefeedback are more likely to be used than those that have been mentioned less frequently, weadd a score of 3, 2, or 1 to the corresponding function when it is classified as “frequently usedfunction”, “less frequently used function” or “infrequently used function”, respectively. Thetop 10 highest scored functions are: B-mode Gain (98), Focus (97), Depth (92), Freeze (89),Colour Doppler (88), TGC (80), Measurements (78), Annotation (78), Capture (74), Power(PW) Doppler (63). The scoring results coincide with the functions presented in Figure 3.4a.Complete function scores are presented in Appendix B. Based on the general usage frequencyacross multiple specializations and scan types covered in this study, our analysis will cover the10 most frequently used functions for all scan specialists.For complete survey questions and responses, please refer to [77].3.4 UML control logic representationThrough observation, we discovered that, for each of the top 10 most frequently used functions,the control logic across all observed US machines are identical, as they can all be expressed viathe same UML diagram. Note that the logical analysis is independent of the technical charac-teristics of a US machine, i.e., responsiveness, image quality, feature set, etc., which may alsoimpact the HCI user experiences. In this section, we define the three-level classifications forUS machine functions and present the UML control logic associated with the 10 most frequentlyused functions.3.4.1 Function classificationTo better categorize the functions and analyze the results, we define the following three levelsof classification according to function complexity:• Binary functions: the functions that can only produce two possible outcomes. The freezeoperation is an example, as the US image on the main screen is either frozen at a specific43142024242426293030320 10 20 30countFrequencyColour DopplerCaptureAnnotationTGCMeasurementsFreezeDepthFocusB-mode Gain(a) Frequently used functions3444555513200 10 20countFocusCalculation PackagesPanoramic ViewCine ToolsAngle SteerDynamic RangeM-ModeTransducer OptionsColour DopplerPW Doppler(b) Less frequently used functions22333334560 2 4 6countColour DopplerTGCGray MapsFrame Speed AverageColor MapsM-ModePatient ToolsDynamic RangePanoramic View3D View(c) Infrequently used functionsFigure 3.4: Function usage frequency (top 10 in each category)frame, or streaming continuously.• One-dimensional (1D) functions: the functions that can produce a discrete set (cardinalitygreater than 2) of 1D outcomes. The B-mode gain operation is an example: we have onedegree of freedom in choosing the gain value for the US image.• Multi-dimensional (MD) functions: the functions that can produce a discrete set of ndimensional outcomes (n > 1). The TGC is an example: we need a vector of severalelements to represent the TGC state.Complex functions usually consist of simpler functions. We will analyze the logical and opera-tional connection and difference between each class in the following sections.Table 3.3 summarizes the complexity classification for the 10 most frequently used func-tions. Although most functions are MD functions, frequently used functions are mostly binaryand 1D functions. Based on the complexity classification, we will present the control logic ofeach complexity class separately.44Function Complexity Minimum step(s)B mode gain Binary 1Focus 1D 1Depth 1D 1Freeze Binary 1Colour Doppler MD 4TGC MD 2Measurement MD 7Annotation MD 4Capture Binary 1PW Doppler MD 4Table 3.3: Function complexity classification and minimum number of step(s) requiredfor the task completionFreezeUS Image StreamingUS Image Frozenfreeze buttonfreeze button(a) FreezeCaptureIdleSave Imagecapture button(b) CaptureFigure 3.5: UML diagram for binary functions3.4.2 UML diagramsUML for binary functionsFigure 3.5 shows the UML diagrams for the freeze and capture functions. We use super statesto abstract the functions, and use sub states to present the possible outcomes of the function.The hierarchy of states in the UML diagram captures the fact that binary functions can result intwo possible states (outcomes). For all the machines listed in Table 3.1, a designated button isassigned for the corresponding binary function.UML for 1D functionsA typical 1D function, e.g., setting B-mode gain, depth, frequency or focus, is illustrated by theUML diagram of Figure 3.6. Note that the frequency function is the 11th most frequently usedfunction. As it can be expressed using the same UML diagram, it is also included. As before,we denote a 1D function as a super state, and use sub states to present the possible functionoperation outcomes. In Figure 3.6, Gi, i = 1,2, ...,N, denote the states for the corresponding45B Mode Gain / Depth / Frequency / Focusdesignated knob / button...G2G1 Gd ... GNb1b2b1b2b1b2b1b2b1b2b1b2b3 b3 b3any operation other than b1 b2 b3b3Figure 3.6: UML diagram: 1D functionsoperation, such as gain values, frequency values and so on. b1 and b2 stand for operations ofpressing the corresponding buttons. For all machines other than # 10 in Table 3.1, knobs areused for 1D functions. Hence, b1 stands for the clockwise rotation of the designated knob, andb2 stands for the counter-clockwise rotation of the knob. For machine # 10, pairs of buttons areused for 1D functions. Figure 3.6 also shows dotted transition lines, b3, which represents thereset operation, i.e. returning the machine state to the default state Gd directly, regardless of thecurrent state. Some of the machines from Table 3.1 have a designated reset button, while othersdo not, which is why we showed the reset operation as dotted lines. For machines that do nothave the reset button, a reset equivalent is performing a machine operation other than b1, b2,and b3.UML for Multi-dimensional functionsFor MD functions, the UML diagrams can get much more complicated than for binary or 1Dfunctions. However, we can always partition MD functions into several base operations, manyof which are binary and 1D operations, and use the nested super states to represent the controllogic.Pointing and selecting One commonly used base operation is the pointing operation, whichis to determine a coordinate on the image or on the screen. MD functions, i.e., measurement,annotation, and color Doppler, require this pointing operation for task completion. We depictthe UML diagram for the pointing operation in Figure 3.7. Each state is denoted by a pair ofintegers (a coordinate), i.e. (X i,Y j). A trackball is always used for the pointing operationamong all the US machines that we observed. The pointing super state consists of N×M substates, whose cardinality is normally of the order of 105. The user is only allowed to transitbetween adjacent states through actions b4 (right), b5 (down), b6 (left) and b7 (up). A selectionoperation usually follows the pointing operation to confirm the point selection. Figure 3.1 showsthe UML diagram for the pointing and selecting operation.46b4 b5 b6 b6Pointing(X1,Y1)(X1,Y2) ... (Xi,Yj-1) ......... (Xi,Yj+1) ... (XN,YM-1)(X1,YM) ... (XN-1,YM) (XN,YM)(Xi-1,Yj) (Xi,Yj) (Xi+1,Yj) ...(X2,Y1) ... (XN,Y1)b4 b6 b5b7 b5b7b4 b6 b6 b5b7 b7 b4 b5 b7b4 b6 b5b4 b6 b5 b7 b4 b6 b5 b7 b7b4 b6 b5 b7b4 b6 b5 b7b7b7b4b7 b5 b5b4b4b4 b6Figure 3.7: UML diagram: pointingMeasurementType selectionEllipse FittingPolygon FittingContineous ContouringDistanceArea / CircumferenceFigure 3.8: UML diagram: MeasurementMeasurement With the pointing and selecting super state, we can better explain the controllogic for measurement. For all US machines that we observed, the touch screen is alwaysinvolved for the measurement operation. There are two kinds of basic measurement tasks: dis-tance measurement, and area/circumference measurement. More advanced US machines with3D US imaging also allow for volume measurement, which we do not discuss in this chapter.Figure 3.8 shows the UML diagram for measurement. Because different measurement types areneeded, a layered menu is employed on the touch screen. For distance measurement, it con-sists of two consecutive pointing and selecting operations. For ellipse fitting and polygon fittingmeasurements, they consist of three or more consecutive pointing and selecting operations. Forcontinuous tracking measurement, a sequence of pointing operations is needed, with a final se-lection operation signalling the task completion. Hence, the measurement operation, regardlessof the measurement types, is to perform repeated pointing and selecting.47Slider bar i...Si,2Si,1 Si,d ... Si,Lai,1ai,2ai,1ai,2ai,1ai,2ai,1ai,2ai,1ai,2(a) Slider barTGCDefault TGCSlider BarsSlider Bar 1Slider Bar 2...Slider Bar i...Slider Bar Na1,: a1,:a1,: ai,: aN,:a2,:a2,: ai,:ai,: aN,:a:,:change presetschange presets(b) TGCFigure 3.9: UML diagram: TGCTGC Slider bars (physical or virtual) are used for TGC adjustments. In Figure 3.9a, we showthe UML diagram for the control logic in the ith slider bar. Si, j is the jth ( j = 1,2, ...,L) statefor the ith slider bar. The transition from Si, j to adjacent states is activated via operations ai,1(right sliding) and ai,2 (left sliding). It is shown that the UML diagram for the operation on oneslider bar is very similar to the UML diagram for 1D functions in Figure 3.6. In Figure 3.9b,we present the super state for TGC, where N slider bars are involved for setting the TGC. ai,:denotes any operation on the ith slider bar, and a:,: denotes any operations on any slider bar.Annotation The control logic for the annotation function is shown in Figure 3.10a. The com-pletion of the annotation task requires two steps: the pointing and selecting step, for the posi-tioning of the intended annotation area, and the user input step, for informing the machine withthe annotation phrases. For some machines, these can be carried out in arbitrary order, whilefor others, these two states have to be visited repeatedly for adjustment. Thus, dashed lines areused in Figure 3.10a for state transitions.Color Doppler & PW Doppler The operation for color/PW Doppler function requires theregion of interest (ROI) selection operation first. As shown in Figure 3.10b the ROI selectionoperation contains steps for selecting two points on the screen. Once the region is specified, theuser may need to adjust several 1D functions, i.e., angle, filters, pulse repetition frequency. Thecontrol logic for 1D functions follows the UML diagram as shown in Figure 3.6.Please note that due to the expressiveness of the UML diagram with nested super states,48AnnotationUser inputPointing and Selectingtrackballkeyboard(a) AnnotationROI SelectionPointing and SelectingPointing and Selectingreselect(b) ROIFigure 3.10: UML diagrams for annotation and ROI selectionwith the diagrams shown in Figures 3.1, 3.9b, 3.10a, and 3.10b being straightforward, in spiteof their ability to capture thousands of possible outcomes for each operation.Based on the UML diagrams, we can find the minimum number of steps (minimum numberof states to visit) to complete a specific function. We summarize these in Table 3.3.3.5 Control logic evaluationThe analysis from the previous section demonstrates that a consistent control logic is adoptedacross a wide range of cart-based US machines. In this section, we evaluate what logical consis-tency brings to the sonographers, in terms of efficiency and user experience. The control logicis sub-optimal if the UML diagram can be simplified, and vice versa. Our analysis integratedfindings from discussions and interviews with sonographers as presented in Section 3.2.3.5.1 Advantages of using a consistent control logicInteroperabilityFrom the examinations of our machines in Table 3.1, we note that for frequently used binaryfunctions, designated buttons are always involved; and for frequently used 1D functions, knobsor a pair of buttons are always used. This paradigm is not hard to explain. A button hastwo possible states, which matches the number of states of binary functions; a knob with tworotational states is easy for the user to navigate through a chain of 1D states (shown in Figure3.6).In fact, this paradigm encourages the interoperability of US machines, as the status of theHCI elements by itself, i.e., the toggle status of buttons, or the rotational direction of knobs,are direct indicators of the machine settings. By the analysis of the UML diagrams, we findthat no other method can further simplify the control logic under this paradigm. Therefore, weconclude that this paradigm for frequently used binary and 1D functions is readily efficient, andthus no improvement is required.49HCI element reuseThe identical UML diagrams for 1D functions enables the reuse of the same HCI elements forother 1D functions. Some observed US machines have already implemented the HCI elementsreuse for 1D functions. For example, knobs originally designed for the frequency adjustmentsare reused for sweep adjustment under the color Doppler mode. According to the UML dia-grams, we know that, if no ambiguity is involved, both the knobs and slider bars can be reusedfor all 1D functions. HCI element reuse in US machines enables a simpler and less expensiveuser interface.3.5.2 Potential improvements to the consistent control logicPointing and selecting efficiencyThis operation is a basic step for more complicated functions, i.e., color Doppler, annotation,zoom and measurement. While it is a common practice to use a trackball and a selection buttonfor the pointing and selecting operation on cart-based systems, attempts have been made to alterthe common practice. In portable US systems, such as tablet or smartphone based handheld USmachines, pointing and selecting can be made on a touch screen, directly on top of the USimage. Thus, a single touch on the screen has replaced the two-step operations of rolling thetrackball and then pressing the selection button. We can see from Figure 3.7 that the transitionbetween states using the trackball can only be achieved via going through adjacent states. Nodirect transition from one arbitrary state (Xi,Yj) to another non-adjacent state is allowed. Thepointing and selecting operation on portable US machines, however, allows arbitrary transitionbetween states with the use of the touch screen, making the operation more efficient. Therefore,logically, the pointing and selecting operation can be further simplified by allowing arbitrarytransitions between non-adjacent states for improved efficiency.Furthermore, fine tuning of the pointing operation with the 2D grid based UML as shownin Figure 3.7 can be difficult as it requires articulated movement of the trackball to differentiatethe adjacent coordinates. To make position fine tuning easier, the uniformly formatted 2D gridfor the UML of pointing should be replaced by a non-uniformly formatted 2D grid, where theuser should be provided with more controllability over a relatively small but important region.Logical independence on related functionsBased on the UML diagrams from Section 3.4, we know that each of the frequently used func-tions are independently operated, and the adjustment of one function does not affect another.That is a sub-optimal control logic, as different operations do not affect US images in an in-dependent manner. For example, the adjustment of depth expands or narrows the view on the50US image, and with the view change, settings such as TGC, gain and frequency may also needadjustment. The current control logic requires the sonographers to perform the re-adjustment,which can prolong the scan duration. Some machines provide built-in auto-optimization func-tions, which jointly optimize multiple machine settings at once. This function partially solvesthe problem of logical independence on related functions, but the sonographer’s preferred/cus-tomized settings are usually overwritten after auto-optimization. To solve this problem, a moredirect method would be the logical connection of related functions, such as linking of the TGCfunctions with depth and focus, and preserving the sonographer’s settings while adjusting oneof the related functions.Measurement operation simplificationTable 3.3 shows that the measurement function requires at least 7 steps to complete, which is themost complicated function in the top 10 most frequently used functions. In practice, the sono-graphers usually need far more than 7 steps for US image measurement. The logical analysisreveals that the measurement operation is highly repetitive, as it requires repeated pointing andselecting. Repetitive operations may lead to fatigue and/or machine operator injuries. Meth-ods have been developed to simplify the measurement procedure via automatic measurementfor specific tasks. With automation, 2 or 3 steps are usually needed for task completion. Forgeneral measurement tasks, we can still simplify the measurement operation via simplifyingthe control logic. In Figure 3.8, 3 steps are needed just to select the type of measurement (dis-tance or area/circumference measurement), which typically involves a combination of buttonsand touch screen for interaction. Logically, the type selection steps can be curtailed, as thenumber of selected points on the screen by itself can indicate the measurement type. Distancemeasures take precisely 2 points; ellipse fitting measures require 3 points (2 points for the edgesof the long axis, and an additional point for the length of the short axis); and polygon fittingmeasures also need more than 3 points. Thus, the control logic for the measurement can besimplified by using the number of pointing and selecting operations to inform the machine ofthe measurement types, reducing the operations on type selection. Figure 3.11 shows the UMLdiagram for optimized measurement operations. Compared to the UML in Figure 3.8, stepsrequired to select measurement types are no longer needed, reducing the time and operationsfor measurement.3.5.3 Applicability to intelligent HCIScan automation methods such as the GE scan assistant [88], are emerging intelligent HCIsfor more efficient and effortless interaction on US machines, such as auto-optimization andauto-measurements. Discussions with sonographers have confirmed their claimed effects ofreducing repetitiveness and expedite US scans. Logically speaking, scan automation methods512st point selectionselectcancel3nd point selectionselectcancellast point selection end1st point selectionselectcancelselectmouse tracingbutton holdendselectcancelMeasurement (optimized)polygonfittingcontinoustrackingenddistancemeasureendellipsefitting...Figure 3.11: UML diagram: Measurement (optimized), where “select”, “cancel”, and“end” are abbreviated button operations to select a point, cancel a point selection,and finish the measurement, respectively.increase scan efficiency by reducing the number of states needed for a task by auto-completionor structured guidance. Usually, we deem the control logic for a UML diagram to be optimalif it cannot be simplified by reducing the total number of states in the UML diagram, or byreducing the number of state transitions for task completion. While this is true for most cases,for artificial intelligence (AI) enabled HCIs, arbitrary reduction of controllable states duringa US scan also means the sonographer is losing control of the machine outcome. Take theAI empowered auto-measurement for example, as little control is provided to the sonographerfor the measurement outcome, sonographers complain that they can either accept the result asa whole, or re-do the measurement by themselves. States presented in a UML diagram alsopresent controllable operations. While it is convenient to compress the states to their minimumor use AI to handle all states automatically, we should be cautious when removing importantstates through intelligent HCI. Operator controllable states are important for accurate diagnosisand efficient machine control.3.6 SummaryThe investigation of the HCI control logic used on common cart-based US systems brings an-swers to the three questions raised in Section 3.1. Observations and a survey of sonographersinformed us of the usage frequency for functions on US machines. With the proposal of theUML diagram, we visualized the consistent control logic for the 10 mostly used functions.UML-based analysis together with interviews and the survey identified the benefits of hav-ing consistent control logic for interoperability and HCI element reuse, and provides improve-ments to pointing and selecting, measurement and joint optimization. We also applied our UMLmethod to devise ways of reducing controllable operations states in intelligent HCI designs.52The study presented in this chapter has some limitations. This chapter does not represent asnap-shot in time of cart-based machines, as we were limited in access to a limited number ofcenters where our study participants worked. Therefore, there is a significant difference in theHCI from the point of view of responsiveness and image quality. Nevertheless, the observedinteraction logic does not seem to have changed in the last two decades, as far as cart-based ma-chines are concerned. Additionally, we only focused on the 10 most frequently used functionsacross several different US scan tasks and scenarios. Functions classified as less frequently orinfrequently used may also be of importance for certain scans. We believe that the analyticalframework proposed in this chapter can also be of help in the evaluation and improvement ofother functions. While the UML formalism was applied to cart-based US systems, it is a generalapproach that can be applied to any HCI setting.53Chapter 4Human-Ultrasound Machine InterfaceAssisted by Gaze Tracking4.1 IntroductionSince the introduction of ultrasound (US) machines, the human computer interaction (HCI)methods employed in US machines are constantly evolving, and combine a variety of modalities[7, 227]. The current application scenarios for US scans range widely, and are principally butnot limited to medical or diagnostic purposes [43]. Regardless of where ultrasound machinesare used, user productivity is always of primary concern in the design of the user interface [7].Traditionally, a physical control panel (PCP) on the US machine provides the main HCI formachine control [226, 227]. A mix of physical buttons, keys, sliders, trackballs and knobs arecommonly used on the PCP of modern US machines. More recent designs have added touchscreens [172] or remote controllers [13] to the PCP. However, according to [77], having a PCPseparated from the main screen can cause interference with the workflow, as the operator needsto switch his or her attention frequently between the main screen and the PCP. Furthermore, itwas also found that work related injuries are closely related to the repetitiveness of adjusting theparameters on the PCP [77]. Moreover, some ultraportable machines lack some or all physicalcontrols. A supplement or alternative to using the PCP on US machines is the use of voicecontrol [157, 225, 239]. However, the use of voice control during a US scan is not preferable,as it can disturb the natural conversation between the patient and the sonographer [77]. To solvethe usage difficulties observed in current HCIs on US machines, we propose to integrate eyegaze tracking technology into the interface.One’s eye gaze naturally reflects his or her location of attention, and, usually, people haveto fixate on or near a target in order to perform a task [240]. Therefore, it is natural to employeye gaze tracking as an HCI method. With the advancement of gaze-tracking technology, it54has been applied to many fields, including social science, psychology, computer science andneuroscience [183]. Attempts to employ gaze tracking to the control of medical devices havealso been made to facilitate effortless control of systems, such as in [79, 102, 179]. The firstattempt to integrate gaze tracking to control US machines was performed in [78], where theoperation of image zooming and panning are integrated with gaze tracking. However, the “Mi-das touch” problem [101] is not properly handled, and thus task completion time when usinggaze-supported HCI is not reduced compared to the conventional HCI [78]. Despite that, [78]provides guidelines to future research on gaze-supported HCI designs for US machines.In our proposed HCI, to increase examination efficiency, we intend to merge some aspectsof the PCP to the main screen, such that the operator no longer needs to switch attention to thePCP to set certain parameters. This is meant as a test of concept study. To achieve this, withouthaving to touch the screen, we use a simple handheld controller as the replacement of the buttonsor keys on the PCP for the US machine. As a result of the work from [78], hybrid-control modeis introduced to better handle the “Midas touch” problem. To avoid switching attention fromthe image to the controls, the pop-up menu appears on top of the US image based on the gazeposition in order to alter the machine settings easily.A detailed demonstration of the proposed HCI is presented in Section 4.2. A user studywas conducted to test the validity of our design ideas. The methods and results of the userstudy are presented in Section 4.3. We conclude this chapter in Section 4.4 by discussing all theadvantages and disadvantages in our HCI, as well as our ideas for future work.4.2 System DesignIn this section, we provide the rationale for and the design details of the proposed gaze-supportedHCI. We start by presenting the common features of HCI designs among several brands of fre-quently used US machines. Next, we summarize the difficulties with some of these features,and use them to motivate our design goals. We then proceed to explain our specific HCI design.4.2.1 Common HCI design features of US machinesWe conducted a survey to observe the HCI designs on different models of US machines. Atotal of 7 different models, listed in Table 4.1, were studied. Despite the difference in brandsand in the year of manufacture, three common characteristics can be observed among these USmachines. First, as shown in Figure 4.1b, the HCIs are composed of two components: a mainscreen, for the display of US images and machine status; and a PCP, for all control inputs to themachine. Second, even though there are differences in layout and shapes, all PCPs consist ofbuttons, knobs, one trackball and one touch screen. Third, the methods for adjusting frequentlyused parameters are very similar or even identical among the observed machines. Among the55Machine ID Brand Model Manufacture year1 GE Voluson 730 20012 BK medical SonixTOUCH 20023 BK medical SonixRP 20054 BK medical bk3500 20165 Philips iU22 20146 Philips Hd5 20187 Exact Imaging Exact VU 2018Table 4.1: US machine detailshundreds of functions that can be used on US machines, only approximately 10 of them areused frequently (in more than 90% of US scans) [77]. The three most frequently used functionsare image freezing, low-resolution zoom, and measurement. These are discussed next.Image freezing is always achieved by pressing the designated button near the trackball. Forlow resolution zoom of the image, 2 machine models use a pair of buttons that control zoomin and out. The others use the rotation of a knob to control zooming. For measurement, thefollowing procedures are followed on all HCIs:1. Press the“measurement” button on the PCP2. Select the measurement type (distance, area) on the touch screen3. Perform the measurement by using the trackball and the selection buttonBased on these observations, we conclude that the general HCI methods for frequently usedcontrol inputs on US machines have remained relatively unchanged during recent years, andthese HCI designs are adopted broadly. As a consequence, sonographers report little difficultyswitching from one machine to another to perform US scans.4.2.2 Usage difficulties and design ideasAccording to [77], the PCP based HCI suffers from several usage difficulties, which are: I)the requirement of an excessive amount of operations on the PCP for the adjustment of certainsettings, which prolongs the duration of the US scan; II) the repetitiveness of body movementwhen adjusting some parameters on the PCP, which not only introduces unnecessary workloadto the operators, but also facilitates work related injuries; and III) the frequent switch of attentionbetween the main screen and the PCP, which can hinder the normal operation flow.Therefore, the goal of our HCI system is to solve the problems I) to III), and facilitateeffortless interaction on US machines.56The reason for the employment of the gaze tracker comes from the information conveyedin gaze signals. As the positional information about one’s eye gaze naturally reflects the regionof interest when performing certain tasks, the excessive steps needed on the PCP to inform themachine about the location of the operation can be reduced. Besides, if we use gaze signals asa source of control input, the requirement of body movement can also decrease. Some of therequired machine control can be achieved directly with eye gaze. Therefore, the employmentof the gaze tracker provides the solutions to problems I) and II).However, due to the “Midas touch” problem, (there is “passive” gaze at the ultrasound imageand “active” gaze at a control button, with switching between the two), gaze tracking may notbe a reliable source of control input. Therefore, we use a multimodal HCI, by complementingeye gaze with a simple handheld controller. As opposed to the PCP, a handheld controller iscompact and flexible to use, and very little body movement is required for its operation. Thus,problem II) can be addressed.With the gaze tracker and the handheld controller, we can accomplish our design idea ofsupplementing or replacing the PCP. We propose to merge the PCP into the main screen, anduse the gaze signal and the handheld controller for control inputs. With a merged PCP on themain screen, parameter adjustment can be achieved by looking at the main screen only, andvirtual buttons and menus on the main screen replace the physical ones on the PCP. Hence,the switch of attention is no longer a problem to the operators, and problem III) is solved.However, we also do not want the virtual buttons and menus on the main screen to obstructthe US images. Therefore, a gaze-supported pop-up menu is designed that can be invoked orrevoked by pressing the designated button on the controller. The pop-up menu will show up atthe location of gaze for efficient setting of parameters at the location of attention.4.2.3 HCI arrangementTo test our design ideas, we build the system whose component diagram is shown in Figure 4.1a,and the system setup is shown in Figure 4.1c. The personal computer (PC) is the processingcenter that transmits the US images and control inputs between our HCI and the US machine.According to the study in [77], the following frequently used control inputs are applied in ourHCI: freeze, low resolution zoom and measurement. With these controls of the US machine, wecan test our design ideas as our HCI can cope with a variety of US scans for routine US exams.We can see from Figure 4.2a that the controller has a trackball and several buttons. There-fore, it is easy to move the cursor with the trackball, and the buttons on the controller can beused in addition to the gaze signal to form a multimodal control input to avoid the “Midas touch”problem. The symmetrical design of the controller makes it easy for both the left-handed andthe right-handed person to use. The key is that the operator does not need to see the controller,only feel it during use.57(a) Component diagram for gaze-supported multimodal HCI(b) Conventional US machine (c) Gaze-supported HCIFigure 4.1: System demonstrationTo be able to adjust the above-mentioned parameters, alongside with the gaze-supportedpop-up menu, two more key functionalities are introduced in our HCI: gaze-supported zoomingand hybrid-control mode.The gaze-centered zoom can be achieved by scrolling the trackball up or down after thepop-up menu has been invoked. The scrolling up will zoom in the image at the center of thepop-up menu, and vice versa. The following procedures are followed for gaze-centered zoom:1. Activate the pop-up menu at the point of gaze.2. Scroll the trackball up/down to zoom in/out the image.We can compare this with the low resolution zoom of conventional HCIs, which follow thefollowing procedure:1. Reach for the knob/button for image zooming.2. Rotate/press the knob/button to zoom in or out the image3. Pan the image if necessary.58Based on this comparison, our design can simplify the operation of image zooming, as nopanning is needed since the HCI knows the proper center for the zooming operation.The hybrid-control mode is designed also for pointing and selecting with our HCI. Becausethe human eye movement has a random component in it, most commercial gaze trackers have atracking error of approximately 1° of visual angle [62]. Therefore, fine pointing and selectingbased on gaze position alone is not practical. For that reason, we introduce a hybrid-controlmode. By pressing and holding the trigger button on the controller, the hybrid-control modeis activated. On activation of the hybrid-control mode, the cursor is set to the current gazelocation. While holding the trigger button, the operator can move the trackball to fine-tune thelocation of the cursor. On releasing the trigger button, a selection is made. Such hybrid-controlmode allows the operator to quickly find the cursor and position it to the point of gaze.Measurement of area or length can also be performed with the pop-up menu. By clickingthe trigger button while the pop-up menu is invoked, the measurement mode is entered. Onentering the measurement mode, the cursor will change colour (to pink), as shown on Figure4.2b. Selecting points in measurement mode can be completed via the hybrid-control mode orvia using the trackball and clicking the trigger button for selection confirmation.To freeze or unfreeze the image, the left button on the controller needs to be pressed. Sincethe freezing of the US image is a time-sensitive operation, a dedicated button on the controlleris assigned to the image freezing operation.Figure 4.2b shows the main screen of our HCI. The pop-up menu is invoked. The letter “M”symbol in the pop-up menu indicates the measurement mode. The “+” and “-” symbols indicatethe zoom in and zoom out operations, respectively. By selecting one of the operations in thepop-up menu, the corresponding symbol will be updated to reflect the operator’s selection. Inour case, we select the zoom in operation, and the “+” symbol changes to “×5.39”for example,indicating the image is zoomed to be around 5 times larger. The two co-centered circles anda cross mark the location of the gaze position. The status bar on the side shows the operationstatus of our HCI.4.3 User StudyIn this section, we discuss the user study performed to compare our proposed HCI with a con-ventional US machine HCI. First, we specify our testing goals and hypotheses. Then, the userstudy is presented. Lastly, we perform the analysis of collected data from the user study, to testour hypotheses.59(a) Controller (b) Main screenFigure 4.2: Gaze supported HCI demonstration4.3.1 Goals and hypothesesThe goal for the user study is to test the design ideas of our HCI. Therefore, a comparativeuser study is performed for first time users of our system. In our HCI prototype, we enabledonly a limited set of menu selections. However, the US scanning scenarios achievable withthese selections occur frequently. For evaluation, we select a typical tumor size measurementtask. Similar measurement tasks are frequently practiced during cardiac and fetal US scans.The efficiency and workload are the evaluation criteria for HCIs. Metrics for the assessmentof HCI efficiency and workload are the task completion time and the NASA task load index(TLX), respectively. The NASA TLX is a commonly used subjective survey for evaluating theworkload of a given task [83, 84], and more detailed description can be found in Appendix C.The following hypotheses are made:H1: There is no difference in the task completion time when using our HCI comparingto the conventional HCI.H2: There is no difference in workload when using our HCI comparing to the conven-tional HCI.4.3.2 ExperimentIn the experiment setup, the SonixTOUCH US machine (BK Medical, Richmond, Canada) isused, both as a reference standard, and to test our new multimodal, gaze-enabled prototype.As discussed in Section 4.2.1, the HCI method on the SonixTOUCH US machine for our USexam task is typical of commonly employed HCI designs. To achieve the connectivity andfunctionality of our HCI system as depicted in Figure 4.1a, the Ulterius SDK and the Qt560framework are used in our software. We employ the GP3 Eye Tracker (Gazepoint Research,Vancouver, Canada), which has an eye tracking accuracy of 1° in visual angle and a samplingrate at 60Hz. A phantom (CAE, Montreal, Canada) with 5 ellipse-shaped tumors inside ischosen for the participants to perform the tumor size measurement task.A total of 8 participants were involved in our experiment. They were first-time users ofan US machine of any kind, have not used our HCI, and have not had prior experience withgaze tracking. The Behavioural Research Ethics Board at the University of British Columbiaapproved our study (study number H15-02969-A004). The following protocol is followed withthe participants.First, they are informed of the purpose and procedure of the experiment. A basic shootinggame is designed for each participant to become familiarized with the hybrid-control mode.Once they are familiarized with the gaze tracker and the hybrid-control mode, demonstrationson how to use both our proposed HCI and the conventional HCI for tumor size measurementare presented. The tumor-size-measurement task is composed of three steps.1. Maneuver the US transducer on the phantom to look for tumors.2. Once a tumor is shown on the image, freeze the image, then zoom the tumor to the centerof the screen.3. Measure the area of the tumor.Each participant is asked to perform 5 sets of measurements with each HCI. They are given5 min to practice the tumor size measurement on each of the HCIs before their measurementsare recorded for analysis. Once the participant freezes the image, the tumor measurement taskcompletion time is recorded. After participants have completed all the measurement tasks, theyare asked to play the shooting game again using each of the HCIs, and their performance isrecorded to test the selection accuracy. Immediately after the game, the participants are askedto complete the NASA TLX form, and user feedback is collected.Figure 4.3 shows the interface for the simple shooting game. We can see there are 6 targetson the screen. The user can shoot at the target by clicking near the center of the target. To playthe shooting game, users are instructed to shoot all 6 targets in an orderly manner. A new frameof randomly generated targets will appear if the user completed shooting at the current targets.The aim of this game is to familiarize the user with using a hybrid-control mode. Unlike theconventional method, where the cursor can only be moved in a continuous manner, the hybrid-control mode enables the user to position the cursor to any arbitrary location on the screen in aninstant (i.e., from one target center to another) by gazing at the desired location. This shootinggame can also be played without the hybrid-control mode, where the cursor needs to be operatedthrough the trackball.61Figure 4.3: Demonstration of the shooting gameTraditiona HCI Gaze-supported HCI10152025303540Duration (sec)p = 0.0003Figure 4.4: One-way ANOVA for task duration4.3.3 Data analysis and resultsFrom the user study as described above, the following data are collected: 1) the selection ac-curacy at the beginning of the experiment with our HCI, 2) the selection accuracy at the endof the experiment with our HCI, 3) the selection accuracy with the conventional HCI, 4) thetask completion time with our HCI, 5) the task completion time with the conventional HCI, 6)the NASA TLX for our HCI, and 7) the NASA TLX for the conventional HCI. Based on theuser data, we can evaluate H1 and H2 accordingly. We choose a significance value α = 0.05for the hypothesis testing using one-way ANOVA. The Matlab function “anova1” was used forthe generation of the test results where outliers are detected automatically and removed whenperforming the test.Analysis of HCI efficiencyThe one-way ANOVA test is performed on the task completion time with each HCI to evaluateH1. Figure 4.4 shows the result. At p = 3.00× 10−4, the one-way ANOVA test rejects thehypothesis H1. Thus, there is a difference in the task completion time when using differentHCIs for the given task. Compared to the average task completion time of 19.2 seconds withconventional HCI, using our HCI can reduce the average task completion time by 20.6% to 15.3seconds.62In addition to the task completion time, the measurement accuracy is another factor to eval-uate the efficiency of the HCI. Therefore, the following hypothesis needs to be evaluated:H3: There is no difference in measurement accuracy with different HCIs during the userstudy.Considering image noise and that the area of the tumor on the image is different when scannedfrom different angles, we cannot use the actual size of the tumor as the evaluation standard forthe measurement accuracy. Instead, we use recorded images during the experiment after theuser has frozen the image. The edge of the tumor on the recorded image is manually traced byan expert. We use the area of the traced shape as the ground truth to evaluate the accuracy ofthe measurement. Since the tumors on the images are of different sizes and zoomed to differentscales, rather than using the absolute differences between the measured size and the groundtruth, we employ the relative measurement error, calculated as:relative measurement error =|measured area−ground truth|ground truth. (4.1)We use the relative measurement error for the one-way ANOVA test of hypothesisH3. At p =0.12, the one-way ANOVA test does not rejectH3, suggesting that the measurement accuracyis not affected by the differences in HCIs during the experiment. Figure 4.5a shows the testresult.Furthermore, we evaluate the accuracy of pointing and selecting using the hybrid-controlmode, comparing to the conventional method. The selection accuracy in the shooting game canhelp us to perform the comparison. Therefore, the hypothesis below is proposed:H4: There is no difference in the accuracy of pointing and selecting with differentHCIs during the user study.To test the hypothesis H4, the one-way ANOVA test is carried out on the three data sets: theselection accuracy at the beginning of the experiment with our HCI, the selection accuracy at theend of the experiment with our HCI, and the selection accuracy with conventional HCI. Figure4.5b shows the results. With p = 0.27, the one-way ANOVA test does not reject the hypothesisH4, indicating that making selections with our HCI is as accurate as using the conventional HCI.Hence, we can verify that doing measurements with each of the HCIs are equally accurate.Analysis of HCI workloadTo evaluate the hypothesisH2, the data of NASA TLX for each of the HCIs needs to be evalu-ated. The NASA TLX divides the workload required for a given task into 6 sections, which are:mental demand, physical demand, temporal demand, performance, effort, frustration. NASA63Traditional HCI Gaze-supported HCI00.050.10.150.20.250.3Relative measurement error p = 0.1196(a) Measurement errorTraditional HCI GameGame-supported HCI Game at beginingGame-supported HCI Game after training00.511.522.5Error (cm)p = 0.2710(b) Selection errorFigure 4.5: One-way ANOVA for measurement and selection errorsMental DemandPhysical DemandTemporal DemandPerformanceEffortFrustration0204060Workload ScoreTraditional HCIMental DemandPhysical DemandTemporal DemandPerformanceEffortFrustration0204060Workload ScoreGaze-supported HCIFigure 4.6: NASA TLX score for each workload componentsTLX evaluations assign the score on each section of the workload, and the overall workload ofthe task is calculated as the sum of each component scores. Figure 4.6 shows the bar plot of theaveraged score for each workload component. The large error bar indicates the large individualdifferences when evaluating the workload. Nevertheless, Figure 4.6 reveals that the workloadrequired when using each of the HCIs are different. The conventional HCI requires more physi-cal activity whereas our gaze-based HCI demands more mental activity, and is more frustratingto the users. This result is in accordance with the user feedback at the end of the user study.When passing the overall NASA TLX score to the one-way ANOVA test, it does not reject thehypothesisH2 with p = 0.75, suggesting that our HCI does not impose excessive workload onthe operator compared to the conventional HCI. The testing of the hypothesisH2 also indicatesthat our design of the gaze-supported HCI is intuitive, causing no extra effort for the operatorto interact with the US machine. Figure 4.7 shows the ANOVA test results of hypothesisH2.Based on the user feedback, there is still room for improvement on our gaze-supported HCI.The most commonly reported problem is with the use of the gaze tracker. Out of the 8 partic-ipants in our user study, 3 of them need to re-calibrate the gaze tracker during the experiment64Traditional HCI Gaze-supported HCI100150200250300Workload Scorep = 0.7521Figure 4.7: One-way ANOVA on NASA TLX overall scoredue to observed deterioration in tracking accuracy. As for the calibration of the gaze tackerat the beginning of the experiment, 7 participants repeated the calibration and one needed tocalibrate three times to yield a desirable tracking accuracy. Repeated calibrations take time, andmay hinder the work flow for US machine operators. Another reported issue when using ourHCI is from the natural blinking of the eye. Since the gaze location is lost when the operatorblinks, and the gaze tracker takes some time to resume the tracking after the operator reopenshis or her eyes, some large disturbances of the tracking accuracy can occur.4.4 SummaryIn this chapter, we present a novel approach to the user-machine interaction on US machines(Obj 3). Statistical results imply that our HCI can reduce the task completion time while main-taining the overall workload to the operators. Three novel functionalities were implemented inour HCI: gaze-centered zooming, gaze-supported pop-up menu, and hybrid-control mode.However, for the testing of design ideas, only a limited number of control inputs wereenabled in our HCI. Our future work will consider a larger set of control inputs in our HCI.The reported problems of repeated calibrations and loss of gaze positions during eye blinkingneed to be solved. A delay is also observed on refreshing the US images on our HCI. Sinceall participants are asked to freeze the image before the measurement, the delay does not affectthe performance of the given task. Yet, for clinical practices, the delay in our HCI needs to beminimized.65Chapter 5Disruption-Free Auto-Recalibrationfor Gaze Trackers5.1 IntroductionGaze tracking technology is proving its merits in many academic and industrial applications,because of its ability to measure one’s point of gaze as a time series in real time [34]. In thebiomedical engineering field, according to [34], gaze tracking is used to 1) provide assessmenttools of expertise level for medical practitioners [19], 2) gain knowledge on the perception andlearning process during medical practice, which can help train novices and explore visual per-ception strategies [29, 122], 3) develop novel human-computer interaction (HCI) methodologiesand assist in usability studies of medical devices [8, 134, 248], and 4) aid automated biomedicalimage processing algorithms and interpretations [25, 54].A typical commercial gaze tracker has a tracking accuracy around 1° of visual angle. Thisis achieved immediately after a calibration step, which usually involves asking the user to lookat a number (5-9) of designated targets, and fitting a model to the data. After calibration, eyegaze tracking performance starts to deteriorate. Various causes can lead to the deterioration,including relative position changes between the tracker camera and the user, and changes inlighting conditions. The problem of tracking accuracy deterioration (TAD) during normal useof gaze trackers has been reported in prior studies [94, 106, 248]. TAD is gradual, and maybe hard to notice by the users after only a few minutes of use [109]. However cumulatively,TAD becomes significant and is clearly noticeable by the user [34]. TAD-caused user interfacedifficulties have been reported in [248], where the gaze tracker is used to assist with the humancomputer interface (HCI) of an ultrasound (US) machine. According to [248], users need tointentionally re-calibrate the tracker for its continued use. Re-calibration is time-consumingand it interferes with the normal workflow. In this chapter, we propose a novel, implicit gaze66tracker re-calibration method, which can maintain eye-gaze tracking accuracy implicitly byacquiring additional calibration assessment points without any explicit user intervention.At the beginning of gaze tracking, an initial calibration is required for every user. WhenTAD occurs, some task-dependent methods may be useful for implicit re-calibration. Gaze posi-tion prediction [106], and smooth pursuit pattern matching [74, 211] are used for re-calibration,provided that the gaze location exhibits predictable movement patterns in specific applications.However, the methods studied in [74, 106, 211] are not generalizable to most other gaze trackingapplications. In [194], the hand-eye coordination during interaction in a virtual reality systemis closely examined, and the authors hypothesize that hand-eye coordination patterns can beused for gaze tracker re-calibration, but the hypothesis is not yet validated through real-worldimplementation. In this chapter, we have studied the hand-eye coordination patterns duringultrasonography tasks and validated the hypothesis by developing a statistical model for hand-eye coordination patterns. With our model, we apply the Gaussian process regression (GPR)framework for re-calibration. We evaluate our re-calibration method on two datasets, one fromthe user study under the framework of the gaze-tracking contingent US machines, the otherfrom a benchmark dataset. To the best of our knowledge, no implicit re-calibration method thatexploits the hand-eye coordination has been developed.The rest of the chapter is organized as follows. In Section 5.2, we present our statisticalmodel and propose our implicit re-calibration method. In Section 5.3, we demonstrate theperformance of our re-calibration method. We present conclusions and suggestions for futurework in Section 5.4.5.2 Statistical Modelling and re-calibrationIn this section, we describe our novel approach of implicit re-calibration. To start, we developstatistical models for tracked gaze coordinates and the hand-eye coordination patterns. Then,we briefly review the GPR framework. Lastly, we present our method of applying the GPRframework to our statistical model for implicit re-calibration.5.2.1 Statistical ModelingModeling gaze trackingFor a gaze tracker to measure the location of one’s eye gaze, several characteristics of the eye,denoted as mk, which includes pupil centroid, pupil diameter and so on, are usually measuredat discrete time k, to estimate the gaze coordinates. Let zk = [zxk,zyk]T ∈ R2 denote the measuredgaze coordinate at time k, where T denotes the transpose operation. Then we have:zk = f (mk) (5.1)67where f (·) : Rp −→ R2 is a deterministic function which we can estimates gaze location zkbased on a vector of measured eye characteristics mk. Both the function f (·) and the parametervector mk are dependent on the specific gaze trackers, and users usually have limited access totheir values. Based on [246], from a user’s perspective, we can rewrite Equation (5.1) as:zk = gk +qk(mk)+ vk (5.2)where gk = [gxk,gyk]T is the actual gaze coordinate, qk(·) is a deterministic function that measuresthe tracking bias given mk; and vk ∈ R2, according to [246], can be modelled as additive whiteGaussian noise. When the tracker is used first, to remove the tracking bias qk(mk), an initialcalibration is performed. However, qk(mk) is a time varying function, and its influence on zk isgradually escalating with the increase of k.Modeling hand-eye coordinationThe hand-eye coordination, as studied in [137], shows that the gaze usually arrives at or nearthe intended selection position around 0.25 s ahead of the actual selection time. Let si = [sxi ,syi ]Tdenote the selection coordinate at time i. The hand-eye coordination informs us of the distancebetween the selection position and the actual gaze positions, d i,k = si−gk, ∀|i− k| < l, whichshould exhibit some uniform pattern across different individuals, where l denotes the time rangefor which the coordination holds [131, 137].By using our user study dataset (recorded during the use of the gaze tracking contingentUS machine), and the bench-mark dataset in [168] (recorded during daily use of personal com-puters), we produce similar hand-eye coordination results, shown in Figure 5.1. In each of thedataset, the user’s HCI operation and gaze coordinates are recorded. Detailed descriptions oneach of the dataset are presented in Section 5.3. To produce Figure 5.1, we compute the dis-tance of each selection position with respect to gaze coordinates within a 1.5 s time window(1 s before the selection and 0.5 s after the selection). For each selection that the user made, wecompute ei,k = zk− si = [exi,k,eyi,k]T . We use ei,k as an approximation of d i,k. As TAD affectsthe values of zk, the results are contaminated by a systematic tracking bias, yet the trend is pre-served. We present this result by showing the 25%, 50% and 75% percentiles for all values ofexi,k that are of the same index difference, k− i, in Figure 5.1a and 5.1b for each dataset. Sim-ilarly, we represent the result for eyi,k and∥∥ei,k∥∥ in Figures 5.1c, 5.1e, 5.1d and 5.1f, where ‖·‖calculates the length of a vector. Note that for all the x-axes of Figure 5.1, we have convertedthe discrete time difference k− i to the corresponding continuous time. We note that, for eachdataset, the time instance for which the gaze location is nearest to the selection location occursapproximately 0.3 s before the selection, and the left and right eyes exhibits minor differencesin hand-eye coordination patterns. We can also notice that some of the 50% percentiles have68-1 -0.5 0 0.5Time (sec)-1.5-1-0.500.51Distance (cm)25% Left eye50% Left eye75% Left eye25% Right eye50% Right eye75% Right eye(a) exi,k in our study-1 -0.5 0 0.5Time (sec)-2024Distance (cm)25% Left eye50% Left eye75% Left eye25% Right eye50% Right eye75% Right eye(b) exi,k in data [168]-1 -0.5 0 0.5Time (sec)00.511.522.5Distance (cm)25% Left eye50% Left eye75% Left eye25% Right eye50% Right eye75% Right eye(c) eyi,k in our study-1 -0.5 0 0.5Time (sec)01234Distance (cm)25% Left eye50% Left eye75% Left eye25% Right eye50% Right eye75% Right eye(d) eyi,k in data [168]-1 -0.5 0 0.5Time (sec)11.522.5Distance (cm)25% Left eye50% Left eye75% Left eye25% Right eye50% Right eye75% Right eye(e)∥∥ei,k∥∥ in our study-1 -0.5 0 0.5Time (sec)468101214Distance (cm)25% Left eye50% Left eye75% Left eye25% Right eye50% Right eye75% Right eye(f)∥∥ei,k∥∥ in data [168]Figure 5.1: Hand-eye coordination from two datasets (best viewed in color)69drifted away from zero. By analyzing the TAD patterns at the end of each of the trials, webelieve the drifting is mainly caused by TAD.Based on [137], and the analysis from dataset in [168], we model the hand-eye coordinationas:d i,k = si−gk ∼N (0,Σk−i) (5.3)for ∀|i− k| < l, where N (0,Σk−i) stands for joint Gaussian distribution with zero mean andcovariance Σk−i. With Equation (5.3), we model the hand-eye coordination as a wide sensestationary Gaussian process.Combining equations (5.2) and (5.3), we have:qk(mk) = zk− si−d i,k− vk (5.4)and define:gk(zk) = E[qk(mk)]= E [zk− si] . (5.5)where E(·) stands for the statistical expectation. The essence of Equation (5.5) is to average outthe noise terms in Equation (5.4), which are d i,k and vk. With Equation (5.5), we can calculatethe averaged measurement bias when having a measurement zk. The terms gk(zk) and qk(mk)represent the measurement bias at time k. The advantage of Equation (5.5) is that all quantitiesare readily available to the user. By finding the gk(zk) for all possible values of zk, we cancomplete the re-calibration task.Several parametric models have been proposed for gk(zk), such as polynomial models[38]. Thus, intuitively, by estimating the parameters in the model, we can complete the re-calibration. However, this intuitive approach does not work well when we use Equation (5.5)for re-calibration. There are two major reasons for this:The first is due to the incomplete and uneven coverage of selection positions on the screen.Figure 5.2 shows the heat maps of selection positions based on our user study and based on thedata from [168]. All coordinates are normalized with respect to the height of the digital screen,and the color encodes the probability of the selection positions. We can see from Figures 5.2aand 5.2c that selection positions are mainly concentrated in certain regions on the screen, whichreflect the normal usage of ultrasound machines such that important regions of the image arepurposefully positioned at the center of the screen. In Figures 5.2a and 5.2c, the selection posi-tion allocation is very different, as it reflects the daily browsing behavior on personal computers.In both dataset, a large portion of the screen, especially the edges, are seldom visited for selec-tion. Therefore, the re-calibration algorithm that tries to estimate gk(zk) must be generalizableto positions where no additional calibration points are provided. Polynomial models usuallyperform poorly under such conditions, and are therefore unsuitable for our problem.70(a) User study (all) (b) User study (last 30)(c) Data [168] (all) (d) Data [168] (last 30)Figure 5.2: Heatmap of selection positions on the screenThe second reason comes from the terms d i,k and vk that we need to average out. Directlyfrom Figure 5.1, we know that the noise introduced by vk and d i,k is high, and usually at a scalethat at least doubles the initial tracking error. To handle re-calibration points with such a highlevel of noise, proper modelling of the noise is usually required. From the two reasons as statedabove, it is natural to turn to a statistical learning approach for the solution.5.2.2 Gaussian Process Regression (GPR) FrameworkThe GPR framework, derived from Bayesian statistics (detailed description in [230]), is a statis-tical learning method that attempts to estimate a function h : f = h(x), f ∈R, x ∈Rm, by n train-ing data pairs: {y ∈ Rn,X ∈ Rn×m}, where y = [y1, ...,yn]T is a vector of measurements (withadditive white Gaussian noise) of h(X ) = [ f1, ..., fn]T . By learning from the training dataset, theGPR attempts to estimate the function h(·) and generates it for unseen inputs X ? ∈ Rl×m, forpredicting f ? = h(X ?) ∈ Rl , where n, m and l are positive integers. The core assumption madeby the GPR is the following joint Gaussian distribution: yf ?∼N0,Λ+σ2n I ΛT?Λ? Λ?? (5.6)where I is the identity matrix; σn ≥ 0 is the standard deviation of the noise in y; Λ+σ2n I =cov(y) is the covariance matrix for y; Λ?? = cov( f ?); and Λ? = cov(y, f ?). The (p,q) element71in the covariance matrix is calculated as:cov(yp,yq) = λ (xp,xq)+σ2n δp,q (5.7)cov(yp, f?q) = λ (xp,x?q) (5.8)cov( f?p, f?q) = λ (x?p,x?q) (5.9)where yi denotes the ith element in y; xi denotes the ith row in X ; λ (·) is the kernel function thatmodels the correlation between different row vectors in X ; and δp,q is the Dirac delta function,which equals to 1 when p = q and 0 otherwise. The kernel functions we choose are from theMatern covariance kernel function family with parameters σ f > 0 and ρ > 0 [230]:λ1(d|σ f ,ρ)= σ2f exp(− dρ)(5.10)λ2(d|σ f ,ρ)= σ2f(1+√3dρ)exp(−√3dρ)(5.11)λ3(d|σ f ,ρ)= σ2f(1+√5dρ +5d23ρ2)exp(−√5dρ)(5.12)λ4(d|σ f ,ρ)= σ2f exp(− d22ρ2)(5.13)where d =∥∥xp− xq∥∥ is the Euclidean distance of vectors xp and xq. Under the GPR framework,unseen data points { f ?,X ?} and the training dataset {y,X} are jointly Gaussian as specifiedby Equation (5.6), where the kernel function captures the correlation. Therefore, with the GPRframework, we can estimate f ? as:f˜ ? = E [ f ?X ,y,X ?] = Λ?[Λ+σ2n I]−1y. (5.14)The learning process of the GPR is to find the hyper-parameters θ , including σn in Equation(5.6) and the parameters σ f and ρ in the kernel function λ (·) that maximizes the marginallog-likelihood function for the training dataset:ln p(yX ) =−12yT[Λ+σ2n I]−1y− 12ln∣∣∣Λ+σ2n I∣∣∣− n2 ln(2pi). (5.15)The GPR framework fits our application scenario for the following two reasons: The first isthat the tracked gaze positions can be modelled as a Gaussian process, which matches the jointGaussian assumption in the GPR model [211, 246]. Second, the GPR framework is a parametricmethod with kernel function λ (·), and thus it can be adapted to individual differences as well tothe wide-sense stationarity of the noise in the measurement.725.2.3 Implicit Recalibration with GPRThe first step is to acquire re-calibration points. With the hand-eye coordination, wheneverthe user makes a selection on the screen, we record the selection coordinate, together with allthe gaze coordinates within the 0.33 s time window before the selection. The value of 0.33 sis used by considering the results in [137] and the analysis made in Section 5.2.1, as gazepositions within that time window exhibits similar adjacency to the corresponding selectionposition. Therefore, at selection time i, we can collect data points {si,Z (i−l):i}, where Z (i−l):i ={Z i−l,Z i−l+1, ...,Z i}, l is the index range for the gaze coordinates. Since gk(zk) is a time varyingfunction, we only keep the data points from the latest 30 selection positions for re-calibration.Figures 5.2b and 5.2d shows the heat maps of the position on the last 30 selection positions ineach dataset.Once the data points are collected, we can perform the re-calibration. In order to estimatethe expectation operation in Equation (5.5), regional averaging of the data points is needed.This is another reason why the GPR framework is a natural fit for our problem. Accordingto [86], the GPR framework can be considered as a linear smoother. From Equation (5.14),we know that the estimation of f ? is a result of linear combination of y with weighting matrixΛ?[Λ+σ2n I]−1. However, different from the ordinary kernel smoothing method, the errorfunction we intend to minimize in the GPR framework is the negative log-likelihood.We use 4 kinds of different kernel functions in order to test on the best suited kernel thatmatches the dataset. Before we can feed our data points to the GPR framework for learning,some additional processing is needed. First, for each {si,z(i−l):i}, we calculate ei,k = zk− si,∀k ∈ {(i− l) : i}. Note that we need to apply the GPR framework to the X and Y componentsin ei,k separately as de-calibrations for the X and Y components are usually different. To meetwith the zero mean condition as required by the GPR (in Equation (5.6)), we need to normalizeei,k. We use e¯i,k to represent the normalized ei,k. Then, we can form our training datasets{{e¯i,i− j,zk} : i ∈ c, l ≤ j ≤ 0} for estimating gk(zk), where c is a set of discrete time indices ofthe latest 30 selections that the user made.The training process of the GPR framework is to find the hyper-parameters, θ = [σn,σ f ,ρ]T ,that minimize the marginal log-likelihood of Equation (5.15). As modelled in Equations (5.3),(5.4), and (5.5), we know that ei,k contains time-varying additive Gaussian noise d i,k, whichdoes not strictly follow the GPR modeling assumptions. To mitigate the effects from d i,k, wefix the hyperparameter ρ = 0.9 during the training process. The hyperparameter ρ controls thesmoothing strength when making predictions. A larger value of ρ generally encourages theGPR framework to average more training points for calculating the predictions. Thus, by fixingρ = 0.9, we have fixed the smoothing strength of the GPR. By doing so, we are also making theGPR framework more robust to outliers.In the implementation for our re-calibration algorithm with GPR, we attempt to find the73local maximum using a gradient based method on Equation (5.15) for σ f and σn, with initialvalues of σ f = 1 and σn = 1. Generally, the σ f and σn that achieve a local maximum forEquation (5.15), can fit the training dataset well [230].To complete re-calibration, we need to estimate gxk(zk) and gyk(zk) for each of the eyes.Thus we need to estimate 4 separate functions. During the training process, as specified inEquation (5.15), inverting the matrix Λ+σ2n I is computationally intensive, especially when thetraining dataset is large. As an optimization operation, we can only use one set of values ofσ f and σn, and jointly maximize the sum of the 4 marginal log-likelihood functions from the 4functions that we attempt to estimate. This optimization approach reduces the times that neededto compute the matrix inversion. Also, it stops the GPR model from over fitting.5.3 ResultsTo validate our implicit re-calibration method, we have tested it on two different datasets. Wealso evaluated the robustness of our method and the validity of the hand-eye coordination model.5.3.1 Re-calibration performance evaluation standardThe averaged tracking bias (ATB) across the screen is used in this chapter for evaluating there-calibration outcome. For the calculation of ATB, the user is asked to fixate consecutivelyon a sequence of 9 designated positions across the screen, at each position for a duration ofapproximately 2 s. To reduce the effects of the additive noise in the calculation of ATB, wefind the median coordinate of the tracked gaze position during each of the fixation periods, anddenote it as z¯ j for fixation position w j. With z j and w j, we can have the tracking bias vectorε = [ε1, ...,εn]T , where ε j =∥∥z¯ j−w j∥∥, 1 ≤ j ≤ n, n is the number of fixation positions. TheATB is calculated as the mean value of the elements in ε . We denote the tracking bias vector ofinitial bias, end bias and re-calibration bias as ε I , εE and εR, respectively, which are needed forassessing our method.Based on the values of ε I and εE , we can determine if TAD is statistically significant, viathe ANOVA test on ε I versus εE . If the ANOVA test returns a p-value less than 0.05, andavg(ε I) < avg(εE), we declare that TAD is significant, where avg(·) calculates the average ofelements in a vector. For cases where TAD is statistically significant, we define 5 classes ofre-calibration performance:• Class A, avg(εR)s< avg(ε I)• Class B, avg(εR)s= avg(ε I)• Class C, avg(εR)s> avg(ε I) and avg(εR)s< avg(εE)• Class D, avg(εR)s= avg(εE)74• Class E, avg(εR)s> avg(εE)wheres< , s= ands> denote the less than, equal to and greater than comparison with statisticalsignificance determined via ANOVA test, respectively. It is not hard to see that Class A is thebest and Class E is the worst, and all the statistical significance in this chapter is determinedby the ANOVA test with significance value 0.05. Hence, the performance of the re-calibrationcan be compared via the percentage of re-calibration results falling into each of the classes.Additionally, we assign scores from 5 through 1 to Class A through E, respectively, and thus wecan have the averaged score for the re-calibration method with a given dataset. A higher scoregenerally represents a better performance of the re-calibration. For cases where no statisticallysignificant TAD is observed, we only check if the performance falls into Class E, such that there-calibration method worsens the tracking bias.5.3.2 User study and Data AcquisitionA user study was designed to validate our re-calibration method. Our study was approved by theBehavioural Research Ethics Board at the University of British Columbia (UBC). The frame-work proposed in [248] is used, where a gaze tracker and a hand-held controller are employedto replace the control panel for the US machine. Figure 5.3 shows the system setup. In [248],the operator’s gaze is continuously tracked, which can be applied for controlling the machinetogether with the use of the controller. In our user study setting, to ensure that the re-calibrationprocess is indeed implicit, the gaze-based controls are disabled, and all testing subjects are notinformed of the purpose of the user study until the completion of the study tasks. By doingso, the subjects have no information on how accurate their tracked gaze positions are, nor canthey change their normal coordination of hand-eye movement for re-calibrating the gaze trackerpurposefully.The user study protocol works as follows. Upon arrival, the subject is informed of thestudy procedures and the proper use of our system. A subject study consent form is signedby the participant. Immediately after the initial calibration, ε I is measured with 9 fixationpositions. Since TAD occurs slowly, we intentionally re-position the gaze tracker randomly andslightly after the initial calibration to introduce additional tracking bias. After that, the taskof measuring the circumference of an elliptical structure in the US images are assigned to theparticipant. Control inputs are collected via the hand-held controller with the participant usingthe non-dominant hand. Once 10 measurements are completed, we collect the εE data, andcomplete the study. For the user study, we enrolled 8 students from UBC, with basic experiencewith gaze tracking and US imaging. The GP3 gaze tracker (by Gazepoint, Vancouver, Canada),sampling at 60 Hz, was used.Additionally, the dataset from [168] is also used to validate our re-calibration method. Al-though the dataset was not intentionally established for gaze tracking re-calibration, we can still75(a) Main screen(b) Controller (c) Gaze tracker (d) System schemeFigure 5.3: System setup (reproduced from [248])utilize it to verify our implicit re-calibration method as ε I and εE can be computed from thedataset. The dataset consists of gaze coordinates as time series from 51 subjects, as well asall the HCI operations performed during the process of reading, web searching and typing onthe computer. The Tobii Pro X3-120 eye tracker (by Tobii AB, Danderyd, Sweden), samplingat 120 Hz, was used. Similar to our user study, the participants whose data were collected in[168] were unaware of the gaze tracking accuracy. They had no intention to change their normalhand-eye coordination for re-calibration.For the evaluation on the assumption that the hand-eye coordination model is always fol-lowed during the usage of the system, an auxiliary user study was performed. The procedureof the auxiliary study is identical to the user study described above, except that the users areprovided with an intrusive task. Whenever a selection is made on the screen, the user is askedto fixate at the selection position where a small red circle will appear. To make sure that theuser maintains his or her fixation on the circle, they are asked to press the confirmation buttonas soon as the color of the circle changes to green at a random time. With the intrusive tasks,we are certain about the user’s fixation positions, and we can use the gaze positions during theintrusive tasks for re-calibration. By comparing the re-calibration results with data from intru-sive tasks and data from normal selections, we can evaluate the applicability of our hand-eyecoordination model. 6 participants took part in the auxiliary user study.For all datasets, re-calibration was performed on the Z¯ j values collected for calculating εE .Then we used the re-calibrated coordinates to compute εR.76mean (left) std (left) mean (right) std (right) left right difference?Initial bias 1.60 cm 0.43 cm 1.73 cm 0.83 cm No (p = 0.37)End bias 3.06 cm 1.21 cm 3.14 cm 1.44 cm No (p = 0.78)Table 5.1: Statistics on ATBs5.3.3 Performance EvaluationBefore presenting the re-calibration results, we demonstrate the statistics that reflect the trendand severity of TAD, with the dataset in [168]. Among the 51 trials, we select 43 trials whoseinitial bias and end bias can be properly computed (no missing values). Table 5.1 shows themean and standard deviation (denoted as std) of all initial bias and end bias for the left andright eyes respectively. The last column in the Table 5.1 shows whether there exists a differencein mean value in initial bias and end bias between the left and right eyes, and the statisticalsignificance is determined by the ANOVA test with significance value 0.05. We can see that themean value of the end bias roughly doubles the mean of initial bias for both of the eyes. Figure5.4a helps us visualize the TAD better by considering time as an additional factor that influencesTAD. Median TAD line is plotted as a linear function of time, which shows that TAD intensifiesat a rate of approximately 0.55 mm/min. Please note that we only use the linear trend linesto coarsely depict the averaged TAD within a short period of time (less than an hour), and nolinearity on the TAD with respect to time is assumed.As discussed in Section 5.3.1, we use categorized classes and scores to represent the re-calibration outcomes on two different datasets. We do not differentiate the re-calibration for theleft or right eyes when presenting the results. For each trial in both datasets, we use 8 differentvariations when applying the GPR framework for re-calibration, where we use 4 different kindsof kernel functions, and for each of the kernel functions, we also compare the influence of theproposed optimization method discussed in Section 5.2.3.For the dataset collected by our gaze-contingent US user study, re-calibration is not per-formed for the right eye of one subject, as the tracker failed to capture the eye feature duringthe ATB measurement phase. The re-calibration and the evaluations were performed on the restof the dataset. For each eye of every subject, the significance in TAD is detected in 53.33% ofthe trials. For the selected dataset in [168], 60.47% of the tracking processes exhibit significantTAD.For trials with significant TAD, Figure 5.4b shows the re-calibration results, where λi∗stands for the case using optimization with kernel λi(·), and λi stands for the method withoutoptimization, i ∈ {1,2,3,4}. Due to the relatively small size of the user study dataset, nodifference is exhibited among different variations of our GPR framework. According to Figure5.4b, we conclude that 75% of the TAD reduction is statistically significant for the user study77dataset, and more than 80% of the TAD reduction for dataset in [168] is not only significant butalso comparable to the initial calibration TAD. We can see that for both datasets, all variationsof methods yield similar results. Please note that the higher frequency of the gaze tracker andbetter tracking accuracy contribute to the fact that the scores for the dataset in [168] are greaterthan that for the user study dataset. For all datasets, the methods with optimization perform atleast as well as those without optimization.We also perform the re-calibration on the rest of the dataset where no significant TAD isobserved, and re-calibration performance at Class E is never observed, suggesting that our re-calibration method does not worsen the calibration. Hence, we conclude that our method iscapable of reliably redressing TAD. The result is statistically significant.To evaluate the robustness of our method, synthetic random data {ys,X s} is appended tothe original training dataset {y,X} for training the GPR model. X s is generated as uniformlydistributed random gaze coordinates on the screen, and ys is the horizontal or vertical componentof uniformly distributed selection position on the screen (independent of X s). Figure 5.5 showsthe score value difference between datasets with added noise and original data. A plateau canbe observed when the added noise is relatively small (fewer than 20 points for the data in [168],and fewer than 7 points for user study dataset), which suggests that our method have somecapability in handling a corrupted dataset with random noise.For the validation on the applicability of the hand-eye coordination model, ANOVA testis performed on the re-calibration results, εR, from the auxiliary user study. No statisticaldifference is observed on the avg(εR) when using data points collected during the intrusive taskas compared to the data points collected during normal selection. This shows that the hand-eyecoordination model is valid and our implicit re-calibration method is as accurate as the explicitre-calibration method given the same additional calibration points.5.4 SummaryIn this chapter, we presented a new eye gaze re-calibration method that uses hand-eye coordi-nation, a continued effort for Obj 3 following Chapter 4. This approach should be helpful forcommon use of computers, whenever a selection process is involved. Based on the coordinatedhand-eye movements, we derive an implicit re-calibration method, and confirm its effectivenessand generality by evaluating it on two different cases. With a growing number of gaze trackingcontingent medical devices, not confined to ultrasonography, we believe that our method canfacilitate a more widespread use of eye gaze tracking.Improvements to our re-calibration method can still be made. The first would be the auto-mated detection of the TAD, that can activate or deactivate the re-calibration intelligently. Also,methods that can optimize the parameter ρ based on different noise levels can be developed.The setting of fixing ρ during the training process may have curtailed the adaptability of our78(a) TAD trend with respect to time3.43.53.63.73.8Score3.383.67 3.69 3.653.693.65 3.65 3.643.71all 1 1* 2 2* 3 3* 4 4*Kernel type050100Percentage(b) Re-calibration resultsFigure 5.4: Data visualization (best viewed in color). For (a), the light blue dots on theleft show the initial bias for each trial, and the corresponding end bias is plotted atthe time instance that reflects the duration of the gaze tracking period. The thickblue line marks the median trend of all selected trials, whose slope is 0.55 mm/min,and the intercept is the median initial bias (1.53 cm). For (b), we use dual y-axes topresent the score (right y-axis) and percentage values (left y-axis) separately.method to a broader range of noise conditions and individual differences. Furthermore, forreal-time implementation, computational efficiency for the GPR model needs to be improved.793 7 20 55 148 403Number of added noise points-1-0.50Score difference11*22*3*344*(a) Data in [168]3 7 20 55 148 403Number of added noise points-2-1.5-1-0.5Score difference 11*22*3*344*(b) User study dataFigure 5.5: Demonstration of robustnesss to noise (best viewed in color)80Chapter 6Human Attention for AttentionNetworks in Deep Learning forMedical ImagesUsing human attention for machine learning methods is a novel trend in recent studies. In thischapter, we investigate two approaches for integration. The first approach is more straightfor-ward, where we exploit and develop new multi-task learning methods to boost network perfor-mance when trained with human attention. In the second approach, we propose a novel methodfor the direct integration of network attention with human attention. Details of the approachesare presented next.6.1 Multi-task UNet: Jointly Boosting Saliency Prediction andDisease Classification on Chest X-ray Images6.1.1 IntroductionRecent work in machine learning and computer vision has demonstrated the advantages of in-tegrating human attention with artificial neural network models, as studies show that manymachine vision tasks, i.e., image segmentation, image captioning, object recognition, and etc.,can benefit from adding human visual attention [140].Visual attention is the ability inherited in biological visual systems to selectively recognizeregions or features on scenes relevant to a specific task [21]. Two major components shapethe visual attention: the “top-down” component based on the task and the visual system, andthe “bottom-up” component where certain features on scenes pertinent to the task distinguishthemselves from the rest [21, 44]. Early research on saliency prediction focuses more on the81uncovering of the “bottom-up” component [21], while more recent attempts, empowered bymachine learning, start to consider both components, and thus the terms, saliency predictionand visual attention prediction, are used interchangeably [203]. In this section, we use theterm saliency prediction as the prediction of human visual attention. With the establishmentof several benchmark datasets, data driven approaches demonstrated major advancements insaliency prediction (review in Borji [20] and Wang et al. [222]). However, saliency predictionfor natural scenes is the primary focus, and little work has been done in the medical domain.Hence, we intend to fill the gap and study the saliency prediction for examining chest X-ray(CXR) images, one of the most common radiology tasks worldwide.CXR imaging is commonly used for the diagnosis of cardio and/or respiratory abnormal-ities; it is capable of identifying multiple conditions through a single shot, i.e., COVID-19,pneumonia, heart enlargement, etc. [27]. There exists multiple public CXR datasets [100, 223].However, the creation of large comprehensive medical datasets is labour intensive, and requiressignificant medical resources which are usually scarce [33]. Consequently, medical datasets arerarely as abundant as that for non-medical fields. Thus, machine learning approaches appliedon medical datasets need to address the problem of data scarcity. In this section, we exploit themulti-task learning for a solution.Multi-task learning is known for its inductive transfer characteristics that can drive strongrepresentation learning and generalization of each component task [30]. Therefore, multi-tasklearning methods partially alleviate some of the major shortcomings in deep learning, i.e., highdemands for data sufficiency and heavy computation loads [46]. However, to apply multi-task learning methods successfully, challenges emerge, which can be the proper selection ofcomponent tasks, the architecture of the network, the optimization of the training schemes andmany others [46, 243]. This section investigates the proper configuration of a multi-task learningmodel that can tackle visual saliency prediction and image classification simultaneously.The main contributions of this section are: 1) development of a new deep convolutional neu-ral network (DCNN) architecture for CXR image saliency prediction and classification basedon UNet [182], and 2) proposal of an optimized multi-task learning scheme that handles overfit-ting. Our method aims to outperform the state-of-the-art networks dedicated either for saliencyprediction or image classification.6.1.2 BackgroundSaliency prediction with deep learningDCNN is the leading machine learning method applied to saliency prediction [105, 124, 128,167]. Besides, transfer learning with pre-trained networks was observed to boost the perfor-mance of saliency prediction [128, 164, 165].82A majority of DCNN approaches are for natural scene saliency prediction, and so far, onlya few studied the saliency prediction for medical images. By Cai et al. [25], the generativeadversarial network is used to predict expert sonographer’s saliency when performing standardfetal head plane detection on ultrasound (US) images. However, the saliency prediction is usedas a secondary task to assist the primary detection task, and thus, the saliency prediction per-formance failed to outperform benchmark prediction methods in several key metrics. Similarly,by Karargyris et al. [112], as a proof-of-concept study, the gaze data is used as an auxiliary taskfor CXR image classification, and the performance of saliency prediction is not reported in thestudy.CXR image classification with deep learningPublic datasets for CXR images enabled data driven approaches for automatic image analysisand diagnosis [133, 191]. Advancements in standardized image classification networks, i.e.,ResNet [87], DenseNet [99], and EfficientNet [206], facilitate CXR image classification. Yet,CXR image classification remains challenging, as CXR images are noisy, and may containsubtle features that are difficult to recognize even by experts [27, 117].6.1.3 Multi-task Learning MethodAs stated in Section 6.1.1, component task selection, network architecture design, and trainingscheme are key factors for multi-task learning. We select the classification task together with thesaliency prediction based on the fact that attention patterns are task specific [113]. Radiologistsare likely to exhibit distinguishable visual behaviors when different patient conditions are shownon CXR images. This section introduces our multi-task UNet (MT-UNet) architecture, andderives a better multi-task training scheme for saliency prediction and image classification.Multi-task UNetFigure 6.1 shows the architecture of the proposed MT-UNet. The network takes CXR images,x ∈R1×H×W , where H and W are image dimensions, as the input and produces two outputs, pre-dicted saliency ys ∈R1×H×W , and classification distributions yc ∈RC, where C is the number ofclasses. Both ys and yc represent distributions, and thus, elements in ys and yc are non-negativeand sum to 1, respectively. For the simplicity of notation, batch dimensions are neglected.The standard UNet architecture is visible from Figure 6.1. As a well-known image-to-imagedeep learning model, the UNet structure has been adopted for various tasks. For example,the UNet is appended with additional structures for visual scene understanding [104], and thefeatures from the middle of the UNet are extracted for image classification tasks [112].The differences of the proposed MT-UNet from earlier attempts lie in the use of classifica-83............1x1 Convolution + Softmax2x2 Max poolingConcatenationUpsamplingGlobal average pooling3x3 Convolution + Batch Normalization + ReLULinear transform + ReLU + dropoutLinear transform + Softmax641282565121024x, inputy s, saliency predictiony c, classificationFigure 6.1: MT-UNet architecture. The solid blocks represent 3D tensors, RF×H×W ,where F , H, and W denote feature (channel), height and width dimensions, respec-tively. The solid circles represent 1D tensors. Arrows denote operations to the ten-sors. Numbers above some of the solid blocks stand for the number features intensors.tion heads (shaded in light green in Figure 6.1), which are added not only to the middle but alsothe ending part of the standard UNet architecture. This additional classification specific struc-ture aggregates middle and higher-level features for classification, exploiting features learnt atdifferent depths. The attention heads perform global average pooling operations to the 4D ten-sors, followed by concatenation, and two linear transforms (dense layers) with dropout in themiddle to produce classification outcomes. The MT-UNet belongs to the hard parameter shar-ing structure in multi-task learning, where different tasks share the same trainable parametersbefore branched out to each tasks’ specific parameters [216]. In Figure 6.1, we use yellow andgreen shades to denote network structures dedicated for saliency prediction and classification,respectively. Usually, multi-task learning using hard parameter sharing schemes has drawbacksof introducing too many additional parameters to the network [216]. Therefore, in our design,we wish to avoid this drawback by minimizing task specific structures.Multi-task Training SchemeBalancing the losses between tasks in a multi-task training process has a direct impact on thetraining outcome [216]. There exist multi-task training schemes [36, 76, 114, 190], and amongwhich, we adopt the uncertainty based balancing scheme [114] with the modification proposedin [136]. Hence, the loss function is:L =1σ2sLs+1σ2cLc+ ln(σs+1)+ ln(σc+1) (6.1)84where Ls and Lc are loss values for ys and yc, respectively; σs > 0 and σc > 0 are trainablescalars estimating the uncertainty of Ls and Lc, respectively; σs and σc are initialized to 1;ln(σs + 1) and ln(σc + 1) are regularizing terms to avoid arbitrary decrease of σs and σc. Asstated in Section 6.1.3, ys and yc are normalized with Softmax to represent distributions, so theloss function for ys and yc are Kullback-Leibler divergence (KLD) loss and cross-entropy loss,respectively. With Equation (6.1), we know that a loss with low uncertainty (small σ value) isprioritized in the training process, and L > 0.Although the training scheme in Equation (6.1) yields many successful applications, it canstill be further optimized. During a training process, a learning rate, r > 0, and effective learningrate, re > 0, are key elements for the training outcome, and they are controlled by several factors.The first factor is the use of the optimizer. Many optimizers, i.e., Adam [119] and RMSProp[209], deploy the momentum mechanism or its variants, which can adaptively adjust re duringtraining. As a learning rate scheduler is often used for more efficient training, it is the secondfactor to influence r. The influence of r from a learning rate scheduler can be adaptive, i.e.,reduce learning rate on plateau, or more arbitrary, i.e., cosine annealing with warm restarts[141]. By observing Equation (6.1), we know that an uncertainty estimator σ for a loss L alsoserves as a learning rate adaptor for L, which is the third factor. More specifically, given a lossvalue L with learning rate r, the effective learning rate for parameters with a scaled loss value Lσ2is rσ2 . The combined effects from several adaptive factors on the learning rate can quickly drivea DCNN model to overfitting the data. As medical datasets are usually small in size, overfittingcan occur more easily. Figure 6.2 shows the training and validation losses as well as the σ valuesduring a multi-task training process specified in Equation (6.1), with the Adam optimizer and areduce-learning-rate-on-plateau learning rate scheduler. We can see from Figure 6.2a, at epoch40, after an initial decrease in both the training and validation losses, the training loss starts todecrease more rapidly, while the validation loss begins to amplify, indicating model overfitting.After continued increase of validation loss, the learning rate scheduler resets the model to the40th epoch, when the validation loss is at its minimal, and reduces the learning rate to 10%.Yet overfitting still occurs with the reduced learning rate. The behavior of σ values as shownin Figure 6.2b is also sub-optimal. When overfitting occurs, training loss decreases, reducingits variance at the same time. Thus, σ decreases accordingly, which increases the effectivelearning rate. The increased effective learning rate further deepens overfitting and decreasestraining loss, creating a vicious circle for overfitting.Mathematically, we can illustrate the vicious circle for overfitting as follows. Let L ≥ 0 bethe loss for a task,T , and σ > 0 be the variance estimator for L used in Equation 6.1. Therefore,the loss for T following Equation 6.1 can be expressed as:L =Lσ2+ ln(σ +1). (6.2)85The partial derivative ofL with respect to σ is:∂L∂σ=−2Lσ3+1σ +1. (6.3)During a gradient based optimization process, to minimize L , σ converges to the equilibriumvalue (σ remains unchanged after gradient descend) which is achieved when ∂L∂σ = 0. There-fore, the following equation holds when σ is at its equilibrium value, denoted as σ˜ :L =σ˜32σ˜ +2(6.4)which is calculated by letting ∂L∂σ = 0. Let f (σ˜) = L, σ˜ > 0, we can calculate that:d f (σ˜)dσ˜=σ˜2(2σ˜ +3)2(σ˜ +1)2> 0, ∀σ˜ > 0. (6.5)Therefore, we know that f (σ˜) is strictly monotonically increasing with respect to σ˜ , and hencethe inverse function of f (σ˜), f−1(·), exists. More specifically, we have:σ˜ = f−1(L). (6.6)As a pair of inverse functions share the same monotonicity, we know that σ˜ = f−1(L) is alsostrictly monotonically increasing. Thus, when L decreases due to overfitting, we know that σ˜will decrease accordingly, forcing σ to decrease. The decreased σ leads to an increase in theeffective learning rate for T , forming a vicious circle of overfitting.20 40 60 80 100 120 140 160 180Epoch22.533.54Losstraining lossvalidation loss(a) Losses20 40 60 80 100 120 140 160 180Epoch11.11.21.31.4 cs(b) σ valuesFigure 6.2: Training process visualization with Equation (6.1)To alleviate overfitting, we propose the use of the following equations to replace Equation(6.1):L =1σ2sLs+Lc+ ln(σs+1) (6.7)86or:L = Ls+1σ2cLc+ ln(σc+1). (6.8)The essence of Equations (6.7) and (6.8) is to fix the uncertainty term for one loss in Equation(6.1) to 1, so that the flexibility in changing effective learning rate is reduced. With the uncer-tainty term fixed for one component loss, Equations (6.7) and (6.8) demonstrate the ability toalleviate overfitting and stabilize the training process. It is worth noting that Equations (6.7)and (6.8) cannot be used interchangeably. Experimental results suggest that the fixing of theloss with greater uncertainty has more positive impacts to the training outcome. In this study,the training process with Equation (6.8) achieves the best performance. Ablation study of thismethod is presented in Section 6.1.5.6.1.4 Training and Evaluation MethodsCXR image saliency datasetWe use the “chest X-ray dataset with eye-tracking and report dictation” [112] shared via Phy-sioNet [154] in this study. The dataset was derived from the MIMIC-CXR dataset [110, 111]with additional gaze tracking and dictation from an expert radiologist. 1083 CXR images areincluded in the dataset, and accompanying each image, there are tracked gaze data; a diagnos-tic label (either normal, pneumonia, or enlarged heart); segmentation of lungs, mediastinum,and aortic knob; and radiologist’s audio with dictation. The CXR images in the dataset are inresolutions of various sizes, i.e., 3056× 2044, and we downsample and/or pad each image to640×416. A GP3 gaze tracker by Gazepoint (Vancouver, Canada) was used for the collectionof gaze data. The tracker has an accuracy of around 1° of visual angle, and has a 60 Hz samplingrate [248].Evaluation MetricsSeveral metrics have been used for the evaluation of saliency prediction performances, andthey can be classified into location-based metrics and distribution-based metrics [24]. Due tothe tracking inaccuracy of the GP3 gaze tracker, location-based metrics are not suited for thisstudy. Therefore, in this section, we follow suggestions in [24] and use KLD for performanceevaluation. We also include histogram similarity (HS), and Pearson’s correlation coefficient(PCC) for reference purposes.For the evaluation of classification performances, we use the area under curve (AUC) met-rics for multi-class classifications [82], and the classification accuracy metrics.87Training SettingsWe use the Adam optimizer [119] and the reduce-learning-rate-on-plateau learning rate sched-uler with a factor of 0.1 for all the training processes. All training, testing, and evaluation areperformed with the PyTorch framework [170]. Hyper-parameters for optimizations are learningrate, patience in reducing learning rate on plateau. The dataset is randomly partitioned into70%, 10% and 20% subsections for training, validation and testing, respectively. The randomsampling process preserves the balanced dataset characteristic, and each of the three classesoccupy 33.3% in all sub-datasets. All the results presented in this section are based on at least5 independent training with the same hyper-parameters. NVIDIA V100 GPUs (Santa Clara,USA) were used.6.1.5 Experiments and ResultTo validate our MT-UNet architecture and the optimized training scheme, comparisons are per-formed with multiple benchmark architectures for CXR image classification and saliency pre-diction. An ablation study is also performed to confirm the improvement of our optimizedtraining scheme.ResNet50 IRNetv2 EffNetv2-S MT-UNet0.80.810.820.830.840.85AUC(a) AUC metricsResNet50 IRNetv2 EffNetv2-S MT-UNet0.60.650.7Accuracy(b) Accuracy metricsFigure 6.3: Comparison with benchmark classification networks. Box plots are used todemonstrate the statistics of the metrics values. The red horizontal line indicatesthe median values; the top and bottom lines in blue boxes denote the 75th and 25thpercentile, respectively; and the shorter black bars at the far ends of each box plotsshow the maximum and minimum values [231].Benchmark comparisonIn this subsection, we compare the performance of our MT-UNet, with benchmark networks forCXR image classification and saliency prediction.For CXR image classification, the benchmark networks were chosen from the top perform-ing networks for CXR image classification examined in [59], which are ResNet50 [87] andInception-ResNet v2 (abbreviated as IRNetV2 in this section) [205]. Following Karargyris et al.88[112], we also include a state-of-the-art general purpose classification network: EfficientNetV2-S (abbreviated as EffNetV2-S) [207] for comparison. Results are presented in Figure 6.3, andWe can see that MT-UNet generally outperforms the other classification networks.For CXR image saliency prediction, comparison was conducted with 3 state-of-the-artsaliency prediction models, which are SimpleNet [178], MSINet [124] and VGGSSM [28].Table 6.1 shows the result, where MT-UNet outperforms the rest.Metrics SimpleNet MSINet VGGSSM MT-UNetKL ↓ 0.756±0.009 0.748±0.003 0.744±0.007 0.727±0.004CC ↑ 0.549±0.008 0.557±0.002 0.561±0.005 0.567±0.004HI ↑ 0.541±0.002 0.544±0.001 0.546±0.003 0.548±0.001Table 6.1: Performance comparison between saliency prediction models. Values are pre-sented as median statistics followed by standard deviations behind the± sign. Metricswith up-pointing arrow ↑ indicate greater values reflect better performances, and viceversa. KL, CC, and HI stand for Kullback–Leibler divergence, Pearson’s correlationcoefficient, and histogram intersection, respectively.Ablation studyTo validate the modified multi-task learning scheme, ablation study is performed. The multi-task learning schemes following Equation (6.1), (6.7), and (6.8) are compared, and they aredenoted as MTLS1, MTLS2, and MTLS3, respectively. Please note that the best-performingMTLS3 is used for benchmark comparison in Section 6.1.5. Figure 6.4 shows the trainingprocess for MTLS2 and MTLS3. With Figures 6.2 and 6.4, we can see that overfitting occursboth for MTLS1 and MTLS2, but the overfitting is reduced in MTLS3. The training processesshown in Figures 6.2 and 6.4 are with optimized hyper-parameters. The resulting performancesare compared in Table 6.2. We can see that MTLS3 outperforms the rest learning schemes bothin classification and in saliency prediction.Metrics MTLS1 MTLS2 MTLS3AUC ↑ 0.830±0.019 0.830±0.010 0.843±0.012KL ↓ 0.727±0.007 0.738±0.006 0.727±0.004Table 6.2: Performance comparison between multi-task learning schemes. The metricsare presented as median statistics followed by standard deviations behind the ± sign.The up-pointing arrow ↑ indicates larger value stands for better performance, and viceversa.8950 100 150Epoch234Losstraining lossvalidation loss(a) MTLS2 losses50 100 150Epoch11.051.11.15cs(b) MTLS2 σ20 40 60Epoch2.22.42.62.83Losstraining lossvalidation loss(c) MTLS3 losses20 40 60Epoch11.11.21.31.4cs(d) MTLS3 σFigure 6.4: Multi-task learning schemes comparison6.1.6 DiscussionIn this section, we build the MT-UNet model and propose a further optimized multi-taskinglearning scheme for saliency prediction and disease classification with CXR images. While amulti-task learning model has the potential of enhancing the performances for all componenttasks, a proper training scheme is one of the key factors to fully unveil its potentiality. As shownin Table 6.2, MT-UNet with the standard multi-task learning scheme may barely outperformexisting models for saliency prediction or image classification.Several future work could be done to improve this study. The expansion of the gaze trackingdataset can be one of the future works. So far, only 1083 CXR images are publicly availablewith the radiologist’s gaze behavior. Also, more dedicated studies on multi-task learning meth-ods, especially for small datasets, can be helpful for machine learning assisted medical tasks.Overfitting and data deficiency are the two main challenges encountered in this and many med-ical machine learning studies. A better multi-task method may handle these challenges moreeasily.906.2 Gaze-Guided Class Activation Mapping: Leveraging HumanAttention for Network Attention in Chest X-rays Classification6.2.1 IntroductionAnalogies and comparisons are frequently made between the artificial neural network (ANN)with its biological counterparts. Multiple explorations and innovations in ANN have been en-lightened by concepts from biological neural networks, among which, the attention mechanism,popularized by [217], is an increasingly prominent research direction [80]. The essence of atten-tion, whether artificial or biological, is “the flexible control of limited computational resources”;its precise mechanism is still pending discovery [93, 138].Though conceptually intertwined, the attention mechanism applied to computer vision tasksusing convolutional neural networks (CNN) does not generally mirror human attention. One ofthe most frequently implemented attention models in CNNs, the multiplicative attention (orthe dot-product attention), enables spatial and/or feature-wise attention via explicitly designednetwork architectures, i.e., the Squeeze-and-Excitation Network [98] and the Attention GatedNetwork [186]. These explicitly crafted architectures can regulate the information flow and au-tomatically form attentions during the training process, but these attentions usually have limitedinterpretability. Another branch of research, the class activation mapping (CAM) [244] and itsvariants [35, 188, 220], aims to unveil the specific regions and/or features in the input imagesthat the network is attending to for specific computer vision tasks. The CAM attention methodscan be applied to most generic CNN architectures, i.e., ResNet [87] and EfficientNet [206], andthe resulting attentions are presented as (ideally human interpretable) 2D attention heat maps.In this section, unlike previous research, we focus on the central question: can human at-tention be directly exploited to help the formulation of network attention and enhance networkperformance?To find the answer to this question, we must first quantitatively measure human attention.One candidate of such measurement is gaze tracking, which measures the position of one’s eye-gaze as a time series [34, 248]. Due to tracking inaccuracies [249] and the stochastic nature ofeye movement [247], the raw tracking data are often too noisy to faithfully reflect human atten-tion. Therefore, visual heat maps, generated by 2D clustering and smoothing of the tracked gazepositions, are more commonly employed to represent the distribution of one’s visual attention[92].With the visual heat maps, we propose the method of gaze-guided class activation map-ping (GG-CAM) that uses the visual attention heat map to supervise the formation of the CAMattention in generic CNN architectures for image classification. When applied to the diseaseclassification tasks for chest X-ray (CXR) images in a public dataset [112], GG-CAM accom-plishes three objectives:911. Be a lightweight (only 3 additional trainable parameters) and generic method that canbe easily added to most classification CNNs for the integration of human attention tonetwork attention.2. Achieve higher performance of CNN classifications and allow the fully trained networksto no longer require visual attention as reference.3. Improve interpretability of the CNNs by spatially relating pathology to anatomy.The rest of the section is organized as follows. We first present the necessary background inSection 6.2.2, where we review recent advancements of disease classification on CXR imageswith CNN, and deep learning methods using visual attention. We also summarize the CAMmethod in Section 6.2.2. Next, in Section 6.2.3, we propose the GG-CAM method. To validateour method, experiments are conducted, and results are discussed in Section 6.2.4. Lastly, weconclude our section in Section 6.2.5.6.2.2 BackgroundDeep learning with Chest X-Ray (CXR) imagesCXR imaging is one of the most frequently used medical diagnostic tools (e.g., , for emergencytriage), and is capable of identifying multiple pathologies, e.g., , pneumonia, pneumothorax, tu-berculosis, COVID-19, and cardiomegaly (enlarged heart) [27]. There exist several public CXRdatasets, such as ChestX-ray14 [223], CheXpert [100], MIMIC-CXR [110] and Ped-Pneumonia[116], that focus on single or multiple pathologies.For datasets focusing on a single pathology, existing CNNs for classification (ResNet [87],MobileNet [97], etc.) can yield outstanding classification performance [117]. For example, inthe pediatric pneumonia CXR dataset (Ped-Pneumonia [116]), [85] reports a binary classifica-tion accuracy of 98.4%. However, datasets targeting multiple pathologies are more difficult forthe CNNs to classify [27, 117]. One example is the MIMIC-CXR dataset [110], where thereare at least 70 different kinds of pathologies and abnormalities, multiple of which may coexistin a single CXR image [235]. The state-of-the-art CNN classifier for the MIMIC-CXR dataset[110] is achieved by [235] and [232], which share the same CNN architecture but focus on twodifferent classification tasks. In [232], the focus is on the binary classification task (normalversus pathological/abnormal CXR images), and the reported area under curve (AUC) metricsis 0.824. In [235], 70 pathologies and abnormalities are considered for classification, and theAUC for each class ranged from 0.628 to 0.985.Other than the challenges arising from the complexity of CXR images, the fact that mostCXR datasets are imbalanced adds an additional level of difficulty for network training andfair benchmarking [117]. To solve the problem, in [112], a balanced dataset derived from the92MIMIC-CXR dataset [110] was proposed. Images in the balanced dataset can be classified intothree mutually exclusive categories: normal, pneumonia and cardiomegaly.Visual heat maps assisted convolutional deep learningVisual heat maps, as a measurement of one’s visual attention, are employed to assist with CNNtraining and optimization through two primary approaches. The most straightforward approachis to feed visual heat maps into the CNN together with the images, as in [25, 26, 139, 192],because gaze patterns are task specific [113]. By doing so, the network can further processthe information conveyed in the visual heat maps to enhance performance; however, this addsan input to the network, making it harder to deploy in real-world tasks. The other approachavoids the dependency on the visual heat maps through representation learning, by using thevisual heat maps to facilitate the learning of representative and robust features through transferlearning [54, 55, 241] or multi-task learning [17]. Still, transfer learning methods may sufferfrom negative transfer or overfitting, and multi-task learning methods commonly introduce alarge number of additional parameters to the network.Although existing CNN methods with visual heat maps yield improved performance, thenetworks are designed to process the visual heat maps in the same manner as other informationsources, leaving the embedded attention characteristics underexplored.Class Activation Mapping (CAM)Most classification CNNs are built with the same consecutive computational blocks (shown inFigure 6.5a): a feature extractor (or called backbone network), a global average pooling layer,a linear layer (or called dense layer or fully connection layer), and a softmax layer (or othernormalization operation). To demystify the internal mechanisms of the CNN “black boxes”, theCAM method [244] was proposed. In [244], the authors focused on the global average poolinglayer and the linear layer in the CNN, from which they extracted 2D attention maps, depictingregions on the input that the network is attending for the outputs. The detailed CAM method isexplained next.Let A ∈ RG×H×W be the output tensor from the feature extractor, where G stands for thenumber of features, and H and W represent the spatial dimensions; p ∈RG be the output tensorfrom the global average pooling layer; and y ∈ RC be the output tensor from the linear layer,where C is the number of classes that the network is trying to categorize. Without loss ofgenerality, we neglect the batch dimension for tensors in this section. Therefore, we have:y = Λp+λ (6.9)where Λ ∈ RC×G and λ ∈ RC are trainable weights and biases, respectively, in the linear layer.93More specifically, the cth element yc in vector y is related to A through the following equation:yc =G∑k=1[Λc,k · pk]+λ c=G∑k=1Λc,k · 1HWH∑i=1W∑j=1Ak,i, j+λ c (6.10)where Ak,i, j denotes the element positioned at (k, i, j) in A. In this section, we use subscriptsas in Equation (6.10) to denote an element in a tensor. By altering the order of summation inEquation (6.10), we have:yc =1HWH∑i=1W∑j=1[G∑k=1Λc,k ·Ak,i, j]+λ c. (6.11)The CAM for the network, Ω ∈ RC×H×W , is defined as:Ωc,i, j =G∑k=1Λc,k ·Ak,i, j (6.12)which is the innermost summation in Equation (6.11).By definition (Equation (6.12)), Ω has three dimensions. We further define the cth 2D sliceof Ω in the first dimension as Ωc ∈ RH×W ; Ωc is a 2D attention heat map that explains whichregions of the input contribute to the decision of predicting class c as the CNN output. This canbe better explained by combining Equations (6.11) and (6.12):yc = Ω¯c+λ c (6.13)whereΩ¯c =1HWH∑i=1W∑j=1Ωci, j (6.14)is the mean of all elements in Ωci, j. From (6.13) we can see that the value of yc can only be in-creased if elements in Ωc are increased. Ω has been frequently applied for network explanationand weakly unsupervised localization [9, 135].6.2.3 GG-CAM: Proposed ApproachOur objective is to improve network performance and interpretability by integrating humanattention with network attention. The proposed GG-CAM method has two major components.One component is the loss function designed for the supervision of the network attention by94Featureextractor(backbonenetwork) Output Input: Image...... ...Global average poolingSoftmaxLinear layer(a) Computational blocks for generic classification CNNsCAM Featureextractor(backbonenetwork) Output Input: Image... ...CAM layerSoftmaxVisualHeatmap Selective MSE lossCross entropy loss(b) Computational blocks for CAM-CNNFigure 6.5: Comparison between generic classification CNN and CAM-CNN.human attention [221]. The other is the revised multi-task learning method that better regulatesthe training focus between the classification task and attention supervision task.Attention SupervisionUsually, to train a classification CNN, the cross entropy loss,Lce, is used:Lce =− ln yˆY (6.15)95where Y ∈ {1,2, ...,C} is the true class label, and yˆ = Softmax(y). Substituting (6.13) in (6.15)yields:Lce =−Ω¯Y −λ Y + ln[C∑i=1exp(Ω¯i+λ i)]. (6.16)From (6.16), we see that the cross entropy loss only focuses on the mean values of each Ωi,such that different Ω can possibly result in the sameLce as long as the mean value for each Ωi,i ∈ {1,2, ...,C}, is unchanged. This opens the way for us to regulate Ω without jeopardizing thenetwork’s performance.To regulate network attention with human attention, we propose the selective mean squareerror (MSE) loss,Lsm, which supervises Ω with visual heat map Ψ during training [221]:Lsm =1HWH∑i=1W∑j=1(Ω˜Yi, j−Ψi, j)2(6.17)where Y ∈ {1,2, ...,C} is the true class label for input I . Ω˜c is related to Ωc via the followingequation:Ω˜c = Sigmoid(αΩc) (6.18)where c= 1,2, ...,C and α > 0 is a trainable scalar. In Equation (6.17), we can see that for inputI with label Y , only ΩY is supervised, and other Ωi, i 6= Y , are neglected. That is based on thefact that one’s visual pattern is task specific as described in Section 6.2.2. We expect that thebiological and artificial attention for a heart condition is different from that for a lung infection.Therefore, visual attention for input I with label Y should only be used to supervise ΩY .Equation (6.18) is applied to elements inΩ for two main reasons. The first reason lies in theboundlessness of Ω: elements in Ω can have any values. Figure 6.6 shows the distribution ofelements in Ω for two fully-trained CNNs, ResNet and EfficientNetV2, for the CXR classifica-tion task. The distribution functions are shown in log-scale. We can see that both distributionsare heavy-tailed with median, mode and mean values around 0.0. However, elements in thevisual heat maps are bounded between 0.0 and 1.0. Hence, to scale Ω, the Sigmoid functionis used. Additionally, we introduce an additional trainable parameter α in Equation (6.18) toallow for more flexibility in the supervision. α is initialized to 1.0 before training.Training schemeIn the previous section, other than the classification loss Lce, we introduced Lsm for attentionsupervision. Therefore, the GG-CAM method is a multi-task learning method. However, differ-ent from most multi-task learning methods, our approach requires minimal alternations of thenetwork architecture and only 3 additional regulatory trainable parameters.From Equations (6.9) through (6.12), we know thatΩ is embedded in the classification CNN96-50 0 50Value10-610-410-2Density (log scale)(a) EfficientNetv2-10 -5 0 5 10Value10-610-410-2Density (log scale)(b) ResNetFigure 6.6: Distribution of CAM elements.though it is not explicitly computed. To better utilize Ω, we rearrange the order of computationas described in Section 6.2.2 so that the network can output Ω and y simultaneously. We namethe rearranged computation as the CAM layer in this section. Figure 6.5b depicts the generalstructure of GG-CAM modified CNNs.In this section, we adopt the following equation to perform multi-task learning [136]:L =12σ2smLsm+1σ2ceLce+ ln(σsm+1)+ ln(σce+1) (6.19)where σsm > 0, σce > 0 are trainable parameters that can dynamically weigh the two lossesLsm andLsm; and ln(σsm+1) and ln(σce+1) are used to penalize large values of σsm and σce,respectively. Equation (6.19) was derived from [114], where σsm and σce are used to measure theuncertainties of the corresponding loss terms. The advantage of Equation (6.19) as comparedto the loss used in [114] is that the loss function is non-negative and can not diverge to minusinfinity during training.Methods in [114, 136] initialize σsm and σce to 1.0 for multi-task learning. While this initial-ization works for many applications, adjustment can be made for improved training outcomes inour setting, where attention supervision is an auxiliary task for disease classification. Therefore,we do not require the supervised network attention to strictly follow human attention as long asthe classification performance is boosted. Following this rationale, the training focus needs tobe on the classification task (reducing Lce) rather than on the attention supervision (reducingLsm), especially towards the ending phase of the training process. To better regulate the trainingprocess, we propose the use of different initialization values for σsm and σce in Equation (6.19),with σsm and σce no longer being initialized to 1.0, but rather to different initial values. Figure6.7 shows the training process with the proposed initialization method for GG-CAM-modified9750 100 150 200 250 300Epoch123(a) Validation lossLce50 100 150 200 250 300Epoch0.511.5(b) σce50 100 150 200 250 300Epoch0.0150.020.025(c) Validation lossLsm50 100 150 200 250 300Epoch10-5(d) σsmFigure 6.7: Training process visualization for GG-CAM modified EfficientNetV2. Logscale for the y-axis is used for all plots. Color blue is used for plots related to classi-fication. Color orange is used for plots related to attention supervision.EfficientNetV2. σce and σsm are initialized to 1.0 and 1.0×10−10, respectively. We can see thatthe σce remains stable for approximately 200 epochs, when it starts to decrease, suggesting thatthe training focus is shifted to the classification task. σsm continues to increase until approxi-mately 200 epochs, gradually giving away training priority. The validation lossLce achieves itsminimum towards the end of the training process, at epoch 277, andLsm achieves its minimumtowards the beginning of the training process, at epoch 28. This training process follows thegoal of the learning scheme and uses attention supervision as an auxiliary task for the priorclassification task. More details are presented through the ablation study in Section 6.2.4.6.2.4 Experiments and ResultsIn this section, experiments to validate the proposed GG-CAM method are discussed. We applyGG-CAM to two standard classification CNNs: ResNet50 [87] and EfficientNetv2 (S) [207].We start by presenting the dataset used in this study and settings in the training process. Then,ablation studies are used to validate our proposed methods. Next, we discuss the benchmarkcomparison results, where the GG-CAM-modified ResNet50 and EfficientNetV2 (S) are com-pared against several benchmark CNNs for disease classification with CXR images. Lastly,limitations and future improvements are discussed.98DatasetWe use the multi-modal CXR dataset from [112] (available at PhysioNet [73] under the Phy-sioNet credentialed health data license). It contains 1083 CXR images originating from theCXR-MIMIC dataset [110]. Accompanying each image is a label (either normal, cardiomegalyor pneumonia), segmentation (mediastinum, aortic knob, left and right lungs), radiology report(text and audio), and tracked eye-gaze coordinates from an expert radiologist examining theimage. The GP3 Gaze Tracker (Gazepoint, Vancouver, Canada) with 60 Hz sampling frequencywas used, and the tracked gaze positions have an accuracy of approximately 1° of visual an-gle (approximately 100-400 pixels on the image depending on the screen size, eye-to-screendistance and calibration accuracy). It is a balanced dataset with an even share for each label.For the training, validation, and testing, we use 70%, 10%, and 20% of the dataset, respec-tively. Each sub-dataset is randomly generated and preserves the label balancing. The raw CXRimages in the dataset follow the digital imaging and communications in medicine (DICOM)standard and have very high resolution, i.e., 2544×3056 pixels. Before feeding the images toCNNs, we downsample the images so that the heights and weights are halved. To reduce thehigh dynamic range of DICOM images, we normalize all images to the range of [0.0,1.0].To generate the visual heat map Ψ, Algorithm 3 is used. Let E 1:M = {E 1, ...,E M} be asequence of 2D coordinates of tracked eye-gaze position at time indices 1 through M whenviewing image I ∈ RH0×W0 . Let H (·) be the 2D histogram function that takes a sequence ofcoordinates as input and outputs a 2D heat map.Algorithm 3 Generate visual heat map Ψ1: ConstructH with horizontal and vertical bins corresponding to the positions of each pixelin I .2: Ψ←H (E 1:M), where Ψ ∈ RH0×W0 .3: Ψ← G (Ψ;B), where G (·;B) is the Gaussian blur operation with standard deviation param-eter B for the Gaussian kernel [67].4: Ψ← Resample(Ψ), such that the new Ψ has dimension H×W .5: Ψ← Normalize(Ψ), such that elements in the new Ψ lie between [0,1]6: Return ΨTrainingWe use the PyTorch framework for the implementation, training, and testing of CNNs. Duringthe training process, we implement the “reduce learning rate on plateau” mechanism, suchthat the learning rate will be reduced to 10% once the validation loss has not decreased for Pconsecutive epochs. For all networks, we optimize the following hyper-parameters: learning99rate, optimizer, and P. The optimizers we use are: Adam, Adamax, stochastic gradient descent(SGD), and SGD+momentum, with their default settings. For GG-CAM-modified networks,we also optimize for the initialization of σce, σsm; and the Gaussian blur parameter B.We use “EffNet” and “ResNet” to abbreviate EfficientNetV2 (S) and ResNet50, respec-tively. We append “GG-CAM” to the GG-CAM-modified network name (connected by a plussign) to differentiate standard CNNs from GG-CAM-modified CNNs, respectively, e.g., , EffNetvs. ResNet+GG-CAM. Due to the randomness in the network’s performance even with identicalhyper-parameters, results for CNNs shown in this section are based on 5 independent trainingruns with the same hyper-parameters, unless otherwise stated.Ablation StudyPerformance comparison The proposed GG-CAM method is ablated into two base versionsfor evaluation. One base version is the pure backbone network without GG-CAM modification.Another base version is the GG-CAM-modified network which uses the standard multi-tasklearning method rather than our proposed learning scheme.In Table 6.4, the area under the curve (AUC) and accuracy metrics for classification withResNet, EffNet, ResNet+GG-CAM, and EffNet+GG-CAM are shown. ANOVA tests [91] sug-gest that the AUC and accuracy metrics for ResNet and EffNet are statistically inferior to thatfor ResNet+GG-CAM and EffNet+GG-CAM, respectively.To validate the effects from our proposed learning scheme, Figure 6.8 shows the effectsfrom different initialization for σce and σsm. We can see that different initial values of σce andσsm result in very different training outcomes. When both σce and σsm are initialized to 1.0,it is the standard multi-task learning scheme. From Figure 6.8, we can see that the standardinitialization method is clearly sub-optimal, as many other initialization combinations outper-form the standard method. With Figures 6.8a and 6.8b, we can observe the following patterns:1) if σsm is not too small (> 2.0× 10−10), metrics at the top-left corner are usually better thanthose in the bottom-right corner; and 2) very small σsm values, e.g., , 9.4× 10−14, sharply un-dermine the performance. Therefore, for the optimal multi-tasking training scheme, we needσce > σsm, and a wide search space for σce and σsm. To generate Figure 6.8, hyper-parametersfor the corresponding networks, i.e., optimizer, learning rate, P and B, are fixed to the optimizedvalues.Attention visualization and comparison To visualize and compare network attention and vi-sual attention, in Table 6.3, we present ΩY and Ψ for testing images with ResNet, EffNet,ResNet+GG-CAM, and EffNet+GG-CAM. By observing the figures, we can see that networkattention, as compared to human visual attention, is different. Concentrated red areas of the vi-100ΩY Y = 1 (normal) Y = 2 (cardiomegaly) Y = 3 (pneumonia)ΨΨ ΨEffNet+GG-CAMResNet+GG-CAMEffNetResNetTable 6.3: Visualization of visual heat map Ψ and network attention ΩY , (Y = 1,2,3), foreach CNN. The heat maps are plotted on top of the CXR image for better visualiza-tion. The heat maps are color coded where red/orange indicates high concentration ofattention and blue/green indicates low attention distribution.sual heat map indicate that the expert’s attention is specific to the targeted regions, which usuallyare the abnormal areas. Whereas the network attention is much less concentrated and covers abroader extension of spaces. Despite the differences, patterns of attention from GG-CAM mod-ified CNNs can be observed. For Y = 1 (normal CXR images), attention for EffNet+GG-CAMand ResNet+GG-CAM is allocated to the entire heart and lung regions in the image, whichcan be interpreted as the network checking if any abnormality is present in these regions. ForY = 2 (cardiomegaly CXR images), heart and neighbouring regions are attended. For Y = 3(pneumonia CXR images), the network attention approximately follows the visual attention,highlighting potentially infected lung sections. However, the attention for EffNet and ResNet ismore random and less interpretable.Hyper-parameter tuning As two additional hyper-parameters, the initial values for σce andσsm, are introduced to the training process, we also investigated what is the proper method tofine-tune all the hyper-parameters. Through experimentation, we found an efficient two-step1019.4e-141.7e-103.1e-075.5e-041.0e+001.8e+033.3e+061.7e-103.1e-075.5e-041.0e+001.8e+033.3e+065.9e+091.1e+13(a) AUC for ResNet+GG-CAM9.4e-141.7e-103.1e-075.5e-041.0e+001.8e+033.3e+061.7e-103.1e-075.5e-041.0e+001.8e+033.3e+06(b) AUC for EffNet+GG-CAMFigure 6.8: Visualization of network performance for different initial values for σce andσsm. The AUC values are color coded through color brightness, where bright (whiteor yellow) colors correspond to large values and dark (red or black) colors indicatesmall values. The values shown in this figure are based on a single training. Thebest metrics are emphasized with bold, underlined fonts. Metrics from the standardinitialization are underlined and shown in blue. Log scale is used in both figures dueto the high dynamic range for σ values.method. The first step is to fix σce and σsm to 1.0, and only tune for the other hyper-parameters.Once the other hyper-parameters are tuned, we fix them, and only optimize initial values forσce and σsm. The resulting hyper-parameters are usually nearly optimal. We recommend to testdifferent initialization values over a large dynamic range, where examples of ranges can be seenin Figure 6.4.The network’s performance is sensitive to the learning rate and the initial values for σce andσsm, but less sensitive to P. The results also suggest that performance is not sensitive to B whenB is in the range [200,1000]. For B outside [200,1000], the performance deteriorates quickly.Adam is the best optimizer for ResNet+GG-CAM and EffNet+GG-CAM.Comparison with State-of-the-art MethodsBenchmark networks Deep learning models are frequently applied for disease classificationwith CXR images, and according to reviews [27, 117], standardized classification CNNs are themain workforce for this task. Therefore, we choose the top performing CNNs: MobileNet [185]and Inception-ResNet V2 [205] (abbreviated as IRNet), as studied in [59] for comparison. Ad-ditionally, we also include in our comparison the customized network, VRNet, from [232, 235][195], because VRNet was shown to outperform experts’ diagnosis. Besides, CNNs exploiting102Network AUC Accuracy Cardio. Interp. Pne. Interp.ResNet 0.721±0.017 0.525±0.024 0.329±0.095 0.611±0.106EffNet 0.734±0.017 0.553±0.033 0.877±0.107 0.583±0.168IRNet 0.741±0.010 0.558±0.026 0.479±0.112 0.458±0.225MobileNet 0.740±0.014 0.548±0.027 0.082±0.109 0.486±0.097VRNet 0.722±0.013 0.562±0.030 0.288±0.130 0.486±0.211ResNet+BAM 0.724±0.026 0.544±0.033 0.863±0.166 0.681±0.138ResNet+CBAM 0.710±0.023 0.484±0.024 0.288±0.343 0.458±0.221ResNet+GG-CAM 0.782±0.039 0.576±0.046 0.945±0.234 0.819±0.111EffNet+GG-CAM 0.802±0.015 0.590±0.020 0.836±0.229 0.944±0.028Table 6.4: Performance comparison between CNNs. Values are presented as medianstatistics followed by the standard deviation after the ± sign. The best metrics arehighlighted in bold font.attention mechanisms can sometimes outperform standard CNNs. Though they have not beenapplied to CXR image classification so far, we include ResNet with the BAM [169] and CBAM[233] attention mechanisms for completeness.Classification performance To evaluate the classification performance, the AUC metrics formulti-class classification as well as the accuracy metrics are used. Table 6.4 shows the results.We can see that our methods, ResNet+GG-CAM and EffNet+GG-CAM, outperform existingmethods (benchmark networks and attention networks) with statistical significance throughANOVA test.Interpretability We know that the pathologies that we try to classify are organ specific: car-diomegaly should only occur at a patient’s heart and pneumonia is a lung infection. Therefore,an interpretable classifier with an abnormal CXR image would put attention to the correspond-ing organ/anatomical areas on the image. Based on this rationale, we can apply the evaluationmethod in [188], where the authors calculate the percentage of CAM attention heatmaps whosepeak lies within the region of the corresponding classification objectives. A larger percentageindicates that the network is more likely to put its attention to the legitimate areas on the inputimage, and thus more interpretable. Mathematically, in our dataset, for an input I with true labelY (cardiomegaly or pneumonia), we declare the network’s attention is interpretable if:argmaxi, j(ΩYi, j)∈ SY (6.20)103where S2 is the set of image coordinates for heart segmentation, and S3 is the set of imagecoordinates for lung segmentation. Note that the dimension of ΩY is different from that ofthe input I . Therefore, coordinates in SY are scaled to match the dimension of ΩY . Table6.4 shows the network attentions’ interpretability for cardiomegaly (Y = 1) and pneumonia(Y = 2), denoted as “Cardio. Interp.” and “Pne. Interp.”, respectively. We can see fromTable 6.4 that the Cardio. Interp. usually exhibit a large variance (standard deviation greaterthan 0.1) for all CNNs. Similar large variances are also observed for Pne. Interp., exceptfor ResNet+GG-CAM and EffNet+GG-CAM. The best interpretability is achieved either byResNet+GG-CAM or EffNet+GG-CAM. Therefore, GG-CAM modified networks can achieveenhanced interpretability.LimitationsAlthough the GG-CAM modified CNNs produce better classification results, disadvantages andlimitations awaiting future improvements still exist. The first is the extended training time.Experimental results show that a GG-CAM modified CNN often requires more epochs (200to 300 epochs) to converge as compared to the standard CNN (usually less than 150 epochs).Secondly, we introduce two additional hyper-parameters to optimize in the training process,which requires additional effort and time. Thirdly, the GG-CAM is only applicable to networkswith architectures shown in Figure 6.5b. As human attention is not limited to the applicationof classification, it may be worthwhile to extend the GG-CAM method to a broader range ofnetworks and/or tasks. Finally, the lack of eye gaze data in the medical domain has limitedfurther validation of our method in different tasks. While diagnosis with CXR images usuallyfollow a set of well-structured protocols, many other medical tasks have limited or no protocol.As human attention exhibits diversified patterns across individuals, it is worth exploring howour method works when the task changes.6.2.5 DiscussionIn this section, we showed that human attention can be used to directly regulate the networkattention that not only boosts the network’s performance but also enhances the interpretability.With GG-CAM, most classification CNNs can be modified to gain the capability of augmentingtheir own attention with human attention. Therefore, by recording the gaze behavior, we candevelop better methods to alleviate the workload for medical practitioners and specialists toannotate CXR datasets, which can be time-consuming and labour-intensive.While we are still at an early stage of understanding artificial and biological neural networks,we believe our results can provide insights for future interdisciplinary research and applicationsin attention and computer vision.104Chapter 7ConclusionsIn this thesis, we investigated the integration of gaze tracking technology to human-ultrasound-machine interaction and medical image understanding. In achieving the objectives as listed inChapter 1, multi-themed research has been conducted. An integrated discussion on the maincontributions and limitations with future work of this thesis is presented in this chapter.7.1 ContributionsRevisiting the motivation for this thesis, we intend to reduce the workload for medical prac-titioners while increasing the throughput of medical procedures with the integration of gazetracking technology. Through the studies presented in this thesis, we demonstrate that gazetracking can serve for the intended purposes.More precisely, in Chapter 2, we modelled the stochasticity of tracked gaze positions dueto eye movement randomness and tracking inaccuracies. We validated our statistical modelthrough developing the Neyman-Pearson detector for the microsaccade events during eye move-ments. In Chapter 3, we deepened our understanding on the image acquisition procedures andusage difficulties on existing US machines. Through observation, questionnaire, and interview,we captured sonographer’s common practises and usage difficulties in their daily handling ofUS machines. Via UML modelling and visualization, we standardized the logical representationfor machine interaction procedures, and identified logically sub-optimal operations for improve-ments. Based on the analysis in Chapter 2 and Chapter 3, we implemented and improved thegaze tracking contingent HCI for US machines in Chapter 4 and Chapter 5. The general ideaof our novel HCI is to reduce the repetitiveness in machine operations by avoiding the switchof attention between the control panel and the main screen during US scans. To achieve thisgoal, we merge the control panel onto the main screen and use the gaze positions to performmachine controls with the pop-up menu on top of the US images. This design can shorten thetask completion time by 21% without introducing additional workload to the operator. As a105gaze tracker’s accuracy slowly deteriorates overtime, we designed an automatic recalibrationmethod based on hand-eye coordination and the Gaussian process regression model to rectifythe deterioration. With Chapter 4 and Chapter 5, we demonstrated that gaze signals are a validand useful information source for US machine control if configured properly.As image interpretation and diagnosis are also time consuming and error prone, focus inChapter 6 is on exploiting the benefit of gaze tracking for AI based intelligent medical imageunderstanding. We designed a novel deep learning architecture with multi-task learning thatcan simultaneously boost the performance of X-ray image diagnosis and saliency prediction.Furthermore, we proposed an innovative method that uses one’s visual attention to supervise theformation of network attention to enhance X-ray image diagnosis (classification) and networkinterpretability. Hence, we showed that the attentional information carried in gaze signals candirectly or indirectly boost the performance for DCNN models on medical imaging relatedtasks.7.2 Limitations and Future Work7.2.1 Qualitative versus quantitative analysisIn this thesis, qualitative analysis on sonographer’s daily use of US machines based on fieldobservation, survey and interview was conducted. While this is a valid method of investigation,the results are mostly presented through comparison rather than quantified values and/or statis-tics. To better represent user experience through quantitative analysis, future work can be donethrough direct recording of user behavior during US scans, i.e., cursor movement, keystrokes,and gaze movement. With the recorded behavioral data, time series analysis and more thoroughdata mining can be conducted.While it is relatively easy to record machine interactions, the recording of gaze behaviorduring a US scan may impose several potential restrictions to both the sonographer and thepatient. The first is the head position constraints. As the user’s eyes need to be constantlyfilmed by the gaze tracker for gaze position estimation, that requires the sonographer to be ata relatively constant position for machine operation. However, during an actual US scan, forthe acquisition of desired US images, a sonographer may need to constantly change positions,hindering the proper tracking of gaze behaviors. The second restriction comes from the trackingaccuracy deterioration as studied in our research. Constant recalibartion or post processing areneeded before the proper use of the recorded gaze data. Thirdly, for privacy concerns, a patientor sonographer may refuse to have the exam process recorded, which is common for medicalrelated research.1067.2.2 Static versus dynamic gaze data representationFor gaze tracking assisted machine methods developed in this thesis, static gaze data (2D visualheatmap) is always used. Though the use of static data yields promising results, the temporalinformation of the tracked gaze data is lost in the static gaze data representation. While itis still a challenging task in machine learning to combine inputs of different modalities, i.e.,2D image, time series, and categorical labels, future work can be done to develop multimodalmachine learning methods that can take a hybrid of input data for joint processing. What’smore, the integration of human attention with the network studied in this thesis is limited tothe CAM attention. As other forms of attention are also presented in the literature, it is worthexploring how human attention can be connected to network attentions of various forms.7.2.3 Medical behavioral data availabilityLack of medical data for various kinds of analysis is a persistent phenomenon due to lackof medical resources and privacy concerns. We need to thank the public dataset on X-rayimages with gaze data that makes our research possible. While efforts have been made tocollect and publish more medical data, behavioral data for medical procedures, to the best of ourknowledge, is largely missing for public use. While recent advancement in HCI and machinelearning are focusing on the automation methods that can be used without any human operation,a majority of medical practices, cannot be automated so far. Therefore, medical behavioraldata is of urgent need not only to the HCI community that tries to facilitate more efficientand ergonomic medical procedures, but also to the reinforcement learning community that canprovide guidance (possibility semi or full automation) on the medical processes.7.2.4 Active gaze assumptionIn this thesis, we assume that during medical diagnostics processes, the user is constantly pay-ing attention to the regions where his or her pupil is pointing towards. Therefore, the termsgaze tracking and eye tracking are used interchangeably throughout the thesis. However, thisassumption may not always be true. Even though most medical diagnostic procedures requirehigh concentration of attention, it cannot stop the medical examiner from occasional attentiondrifting, in which case, the measured gaze position cannot reflect their true visual attention lo-cation. In our study we do not handle this situation specifically, but in future work, better resultscould be acquired if we can take the attention drifting into consideration.107Bibliography[1] F. H. Adler and M. Fliegelman. Influence of fixation on the visual acuity. Archives ofOphthalmology, 12(4):475–483, 1934. → page 11[2] N. Afsham, M. Najafi, P. Abolmaesumi, and R. Rohling. 3d ocular ultrasound usinggaze tracking on the contralateral eye: a feasibility study. In International Conferenceon Medical Image Computing and Computer-Assisted Intervention, pages 65–72.Springer, 2011. → page 6[3] M. Ahmed and J. A. Noble. An eye-tracking inspired method for standardised planeextraction from fetal abdominal ultrasound volumes. In 2016 IEEE 13th InternationalSymposium on Biomedical Imaging (ISBI), pages 1084–1087. IEEE, 2016. → page 7[4] M. Ahmed and J. A. Noble. Fetal ultrasound image classification using a bag-of-wordsmodel trained on sonographers’ eye movements. In MIUA, pages 157–162, 2016. →page 7[5] R. G. Alexander, S. L. Macknik, and S. Martinez-Conde. Microsaccades in appliedenvironments: Real-world applications of fixational eye movement measurements.Journal of Eye Movement Research, 12(6), 2019. → page 9[6] C. H. Anderson, D. C. Van Essen, and B. A. Olshausen. Directed visual attention andthe dynamic control of information flow. In Neurobiology of attention, pages 11–17.Elsevier, 2005. → page 1[7] G. Andreoni, M. Mazzola, S. Matteoli, S. D’Onofrio, and L. Forzoni. Ultrasoundsystem typologies, user interfaces and probes design: a review. ProcediaManufacturing, 3:112–119, 2015. → pages 7, 37, 54[8] M. S. Atkins, A. Moise, and R. Rohling. An application of eyegaze tracking fordesigning radiologists’ workstations: Insights for comparative visual search tasks. ACMTransactions on Applied Perception (TAP), 3(2):136–151, 2006. → pages 6, 66[9] W. Bae, J. Noh, and G. Kim. Rethinking class activation mapping for weaklysupervised object localization. In European Conference on Computer Vision, pages618–634. Springer, 2020. → page 94[10] J. P. Baker and C. T. Coffin. The importance of an ergonomic workstation to practicingsonographers. Journal of Ultrasound in Medicine, 32(8):1363–1375, 2013. → pages8, 37108[11] R. W. Baloh, A. W. Sills, W. E. Kumley, and V. Honrubia. Quantitative measurement ofsaccade amplitude, duration, and velocity. Neurology, 25(11):1065–1065, 1975. →page 1[12] D. E. Barkana and A. Ac¸ık. Improvement of design of a surgical interface using an eyetracking device. Theoretical Biology and Medical Modelling, 11(S1):S4, 2014. → page6[13] K. Bartol, S. Graziano, and W. Kelton. Remote wireless control device for anultrasound machine and method, Mar. 22 2007. Patent App. 10/550,046. → page 54[14] M. E. Bellet, J. Bellet, H. Nienborg, Z. M. Hafed, and P. Berens. Human-level saccadedetection performance using deep neural networks. Journal of neurophysiology, 121(2):646–661, 2018. → pages xi, 12, 29, 31[15] H. Bengi and J. Thomas. Three electronic methods for recording ocular tremor.Medical and biological engineering, 6(2):171–179, 1968. → page 11[16] G. B. Benitez, F. S. Fogliatto, R. B. Cardoso, F. S. Torres, C. S. Faccin, and J. M. Dora.Systematic layout planning of a radiology reporting area to optimize radiologists’performance. Journal of Digital Imaging, 31(2):193–200, 2018. → pages 8, 37[17] R. Bera, V. G. Goecks, G. M. Gremillion, V. J. Lawhern, J. Valasek, and N. R.Waytowich. Gaze-informed multi-objective imitation learning from humandemonstrations. arXiv preprint arXiv:2102.13008, 2021. → page 93[18] G. Booch, J. Rumbaugh, and I. Jacobson. Unified modeling language user guide 2ndedition, 2005. → pages 38, 40[19] L. K. Borg, T. K. Harrison, A. Kou, E. R. Mariano, A. D. Udani, T. E. Kim, C. Shum,and S. K. Howard. Preliminary experience using eye-tracking technology todifferentiate novice and expert image interpretation for ultrasound-guided regionalanesthesia. Journal of Ultrasound in Medicine, 37(2):329–336, 2018. → page 66[20] A. Borji. Saliency prediction in the deep learning era: Successes and limitations. IEEEtransactions on pattern analysis and machine intelligence, 2019. → page 82[21] A. Borji, D. N. Sihite, and L. Itti. Quantitative analysis of human-model agreement invisual saliency modeling: A comparative study. IEEE Transactions on ImageProcessing, 22(1):55–69, 2012. → pages 81, 82[22] M. Brown, M. Marmor, E. Zrenner, M. Brigell, M. Bach, et al. Iscev standard forclinical electro-oculography (eog) 2006. Documenta ophthalmologica, 113(3):205–212,2006. → page 2[23] W. Buxton. A three-state model of graphical input. In Human-ComputerInteraction-INTERACT, volume 90, pages 449–456, 1990. → page 38[24] Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand. What do different evaluationmetrics tell us about saliency models? IEEE transactions on pattern analysis andmachine intelligence, 41(3):740–757, 2018. → page 87109[25] Y. Cai, H. Sharma, P. Chatelain, and J. A. Noble. Multi-task sonoeyenet: detection offetal standardized planes assisted by generated sonographer attention maps. InInternational conference on medical image computing and computer-assistedintervention, pages 871–879. Springer, 2018. → pages 7, 66, 83, 93[26] Y. Cai, H. Sharma, P. Chatelain, and J. A. Noble. Sonoeyenet: Standardized fetalultrasound plane detection informed by eye tracking. In 2018 IEEE 15th InternationalSymposium on Biomedical Imaging (ISBI 2018), pages 1475–1478. IEEE, 2018. →pages 7, 93[27] E. C¸allı, E. Sogancioglu, B. van Ginneken, K. G. van Leeuwen, and K. Murphy. Deeplearning for chest x-ray analysis: A survey. Medical Image Analysis, page 102125,2021. → pages 82, 83, 92, 102[28] G. Cao, Q. Tang, and K.-h. Jo. Aggregated deep saliency prediction by self-attentionnetwork. In International Conference on Intelligent Computing, pages 87–97. Springer,2020. → page 89[29] A. J. Carrigan, P. C. Brennan, M. Pietrzyk, J. Clarke, and E. Chekaluk. A ‘snapshot’ofthe visual search behaviours of medical sonographers. Australasian journal ofultrasound in medicine, 18(2):70–77, 2015. → page 66[30] R. Caruana. Multitask learning. Machine learning, 28(1):41–75, 1997. → page 82[31] G. Casella and R. L. Berger. Statistical inference, volume 2. Duxbury Pacific Grove,CA, 2002. → page 22[32] B. Cassin, S. Solomon, and M. L. Rubin. Dictionary of eye terminology. TriadPublishing Company Gainesville, 1990. → page 1[33] D. C. Castro, I. Walker, and B. Glocker. Causality matters in medical imaging. NatureCommunications, 11(1):1–10, 2020. → page 82[34] P. Chatelain, H. Sharma, L. Drukker, A. T. Papageorghiou, and J. A. Noble. Evaluationof gaze tracking calibration for longitudinal biomedical imaging studies. IEEEtransactions on cybernetics, 50(1):153–163, 2018. → pages 9, 66, 91[35] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian. Grad-cam++:Generalized gradient-based visual explanations for deep convolutional networks. In2018 IEEE winter conference on applications of computer vision (WACV), pages839–847. IEEE, 2018. → page 91[36] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradientnormalization for adaptive loss balancing in deep multitask networks. In InternationalConference on Machine Learning, pages 794–803. PMLR, 2018. → page 84[37] H. Chennamma and X. Yuan. A survey on eye-gaze tracking techniques. arXiv preprintarXiv:1312.6410, 2013. → page 11110[38] Z. R. Cherif, A. Nait-Ali, J. Motsch, and M. Krebs. An adaptive calibration of aninfrared light device used for gaze tracking. In IMTC/2002. Proceedings of the 19thIEEE Instrumentation and Measurement Technology Conference (IEEE Cat. No.00CH37276), volume 2, pages 1029–1033. IEEE, 2002. → page 70[39] T. S. Chow. Testing software design modeled by finite-state machines. IEEETransactions on Software Engineering, (3):178–187, 1978. → page 38[40] A. J. Chung, F. Deligianni, X.-P. Hu, and G.-Z. Yang. Extraction of visual features witheye tracking for saliency driven 2d/3d registration. Image and Vision Computing, 23(11):999–1008, 2005. → page 7[41] K. J. Ciuffreda and B. Tannen. Eye movement basics for the clinician. MosbyIncorporated, 1995. → page 11[42] N. T. Clancy, G. P. Mylonas, G.-Z. Yang, and D. S. Elson. Gaze-contingent autofocussystem for robotic-assisted minimally invasive surgery. In 2011 Annual InternationalConference of the IEEE Engineering in Medicine and Biology Society, pages5396–5399. IEEE, 2011. → page 6[43] R. S. Cobbold. Foundations of biomedical ultrasound. Oxford University Press, 2006.→ page 54[44] R. Cong, J. Lei, H. Fu, M.-M. Cheng, W. Lin, and Q. Huang. Review of visual saliencydetection with comprehensive information. IEEE Transactions on circuits and Systemsfor Video Technology, 29(10):2941–2959, 2018. → page 81[45] N. J. Cowan. 2 1 mind/brain institute, johns hopkins university, baltimore, maryland21218, usa; email:[email protected] 2 department of mechanical engineering, johnshopkins university, baltimore, maryland 21218. USA Annual Review of Control,Robotics, and Autonomous Systems, 3:243–267. → page 2[46] M. Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprintarXiv:2009.09796, 2020. → page 82[47] P. M. Daye and L. M. Optican. Saccade detection using a particle filter. Journal ofneuroscience methods, 235:157–168, 2014. → page 12[48] S. De Brouwer, M. Missal, G. Barnes, and P. Lefe`vre. Quantitative analysis of catch-upsaccades during sustained pursuit. Journal of neurophysiology, 87(4):1772–1780, 2002.→ page 2[49] M. de Rooij, B. Israe¨l, M. Tummers, H. U. Ahmed, T. Barrett, F. Giganti, B. Hamm,V. Løgager, A. Padhani, V. Panebianco, et al. Esur/esui consensus statements onmulti-parametric mri for the detection of clinically significant prostate cancer: qualityrequirements for image acquisition, interpretation and radiologists’ training. Europeanradiology, 30(10):5404–5416, 2020. → page 6111[50] K. S. Deepak, A. Chakravarty, J. Sivaswamy, et al. Visual saliency based bright lesiondetection and discrimination in retinal images. In 2013 IEEE 10th InternationalSymposium on Biomedical Imaging, pages 1436–1439. IEEE, 2013. → page 7[51] A. J. Degnan, E. H. Ghobadi, P. Hardy, E. Krupinski, E. P. Scali, L. Stratchko, A. Ulano,E. Walker, A. P. Wasnik, and W. F. Auffermann. Perceptual and interpretive error indiagnostic radiology—causes and potential solutions. Academic radiology, 26(6):833–845, 2019. → page 6[52] H. Deubel and B. Bridgeman. Fourth purkinje image signals reveal eye-lens deviationsand retinal image distortions during saccades. Vision research, 35(4):529–538, 1995. →page 11[53] J. E. Dowling and J. L. Dowling. Vision: How it Works and what Can Go Wrong. MITPress, 2016. → page 11[54] R. Droste, Y. Cai, H. Sharma, P. Chatelain, L. Drukker, A. T. Papageorghiou, and J. A.Noble. Ultrasound image representation learning by modeling sonographer visualattention. In International conference on information processing in medical imaging,pages 592–604. Springer, 2019. → pages 7, 66, 93[55] R. Droste, P. Chatelain, L. Drukker, H. Sharma, A. T. Papageorghiou, and J. A. Noble.Discovering salient anatomical landmarks by predicting human gaze. In 2020 IEEE17th International Symposium on Biomedical Imaging (ISBI), pages 1711–1714. IEEE,2020. → pages 7, 93[56] L. Drukker, R. Droste, P. Chatelain, J. A. Noble, and A. T. Papageorghiou.Expected-value bias in routine third-trimester growth scans. Ultrasound in Obstetrics &Gynecology, 55(3):375–382, 2020. → page 6[57] A. T. Duchowski. A breadth-first survey of eye-tracking applications. BehaviorResearch Methods, Instruments, & Computers, 34(4):455–470, 2002. → page 9[58] M. Dumas and A. H. Ter Hofstede. Uml activity diagrams as a workflow specificationlanguage. In International Conference on the Unified Modeling Language, pages76–90. Springer, 2001. → page 38[59] K. El Asnaoui, Y. Chawki, and A. Idri. Automated methods for detection andclassification pneumonia based on x-ray images using deep learning. In ArtificialIntelligence and Blockchain for Future Cybersecurity Applications, pages 257–284.Springer, 2021. → pages 88, 102[60] R. Engbert. Microsaccades: A microcosm for research on oculomotor control, attention,and visual perception. Progress in brain research, 154:177–192, 2006. → pages 11, 12[61] R. Engbert and R. Kliegl. Microsaccades keep the eyes’ balance during fixation.Psychological science, 15(6):431–431, 2004. → page 13112[62] A. M. Feit, S. Williams, A. Toledo, A. Paradiso, H. Kulkarni, S. Kane, and M. R.Morris. Toward everyday gaze input: Accuracy and precision of eye tracking andimplications for design. In Proceedings of the 2017 CHI Conference on Human Factorsin Computing Systems, pages 1118–1130. ACM, 2017. → page 59[63] B. Fischer and E. Ramsperger. Human express saccades: extremely short reaction timesof goal directed eye movements. Experimental brain research, 57(1):191–195, 1984. →page 1[64] D. Forsberg, B. Rosipko, and J. L. Sunshine. Factors affecting radiologist’s PACSusage. Journal of Digital Imaging, 29(6):670–676, 2016. → pages 8, 37[65] J. Forsman, N. Anani, A. Eghdam, M. Falkenhav, and S. Koch. Integrated informationvisualization to support decision making for use of antibiotics in intensive care: designand usability evaluation. Informatics for Health and Social Care, 38(4):330–353, 2013.→ page 6[66] L. Forzoni, C. Buffagni, and S. Guercini. Strategic marketing: From product roadmapto needs roadmap focus, a cultural-practical shift. Journal of Industrial and IntelligentInformation Vol, 7(1), 2019. → page 40[67] M. W. Frederick and J. W. Millerb. An efficient algorithm for gaussian blur usingfinite-state machines. In E Conf. on Machine Vision Systems for Inspection andMetrology, 1998. → page 99[68] A. Fuchs. Saccadic and smooth pursuit eye movements in the monkey. The Journal ofPhysiology, 191(3):609–631, 1967. → page 1[69] A. Garcia-Lallana, G. Viteri-Ramirez, R. Saiz-Mendiguren, J. Broncano, and J. D.Aquerreta. Ergonomics of the workplace in radiology. Radiologı´a (English Edition), 53(6):507–515, 2011. → pages 8, 37[70] M. Gertner, M. Arnoldussen, E. Chell, S. D. Hansen, M. Herron, I. Koruga, andJ. Liang. Methods and devices for detecting, controlling, and predicting radiationdelivery, Sept. 7 2010. US Patent 7,792,249. → page 6[71] M. Gogolla and F. P. Presicce. State diagrams in uml: A formal semantics using graphtransformations-or diagrams are nice, but graphs are worth their price. In University ofMunich. Citeseer, 1998. → page 38[72] J. A. Gold, L. E. Stephenson, A. Gorsuch, K. Parthasarathy, and V. Mohan. Feasibilityof utilizing a commercial eye tracker to assess electronic health record use duringpatient simulation. Health informatics journal, 22(3):744–757, 2016. → page 6[73] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark,J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. Physiobank, physiotoolkit,and physionet: components of a new research resource for complex physiologic signals.circulation, 101(23):e215–e220, 2000. → page 99113[74] A. R. Gomez and H. Gellersen. Smooth-i: smart re-calibration using smooth pursuit eyemovements. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research &Applications, page 10. ACM, 2018. → page 67[75] N. Goyal, N. Jain, and V. Rachapalli. Ergonomics in radiology. Clinical Radiology, 64(2):119–126, 2009. → pages 8, 37[76] M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei. Dynamic task prioritizationfor multitask learning. In Proceedings of the European Conference on Computer Vision(ECCV), pages 270–287, 2018. → page 84[77] Y. Halwani. An investigation of multi-modal gaze-supported zoom and pan interactionsin ultrasound machines. Master’s thesis, University of British Columbia, 2017. →pages 8, 9, 39, 43, 54, 56, 57[78] Y. Halwani, S. E. Salcudean, V. A. Lessoway, and S. S. Fels. Enhancing zoom and panin ultrasound machines with a multimodal gaze-based interface. In Proceedings of the2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems,pages 1648–1654, 2017. → pages 5, 55[79] J.-H. Han and E.-h. Yang. Method of moving the displays of an ultrasound diagnosticdevice and ultrasound diagnostic device, 2015. US Patent App. 14/314,313. → pages6, 55[80] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu,et al. A survey on visual transformer. arXiv preprint arXiv:2012.12556, 2020. → page91[81] P. Han, D. R. Saunders, R. L. Woods, and G. Luo. Trajectory prediction of saccadic eyemovements using a compressed exponential model. Journal of vision, 13(8):27–27,2013. → page 14[82] D. J. Hand and R. J. Till. A simple generalisation of the area under the roc curve formultiple class classification problems. Machine learning, 45(2):171–186, 2001. → page87[83] S. G. Hart. Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the humanfactors and ergonomics society annual meeting, volume 50, pages 904–908. SagePublications Sage CA: Los Angeles, CA, 2006. → pages 60, 131[84] S. G. Hart and L. E. Staveland. Development of nasa-tlx (task load index): Results ofempirical and theoretical research. In Advances in psychology, volume 52, pages139–183. Elsevier, 1988. → pages 60, 131[85] M. F. Hashmi, S. Katiyar, A. G. Keskar, N. D. Bokde, and Z. W. Geem. Efficientpneumonia detection in chest xray images using deep transfer learning. Diagnostics, 10(6):417, 2020. → page 92[86] T. Hastie and R. Tibshirani. Exploring the nature of covariate effects in the proportionalhazards model. Biometrics, pages 1005–1016, 1990. → page 73114[87] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages770–778, 2016. → pages 83, 88, 91, 92, 98[88] G. Healthcare. Logiq e9: Scan assistant, 2020. URLhttps://www.gehealthcare.com/products/video-scan-assistant. → pages 8, 51[89] C. W. Helstrom. Quantum detection theory. 1971. → page 16[90] C. Hennessey, B. Noureddin, and P. Lawrence. A single camera eye-gaze trackingsystem with free head motion. In Proceedings of the 2006 symposium on Eye trackingresearch & applications, pages 87–94, 2006. → page 3[91] R. V. Hogg and J. Ledolter. Engineering statistics. Macmillan Publishing Company,1987. → page 100[92] K. Holmqvist, M. Nystro¨m, R. Andersson, R. Dewhurst, H. Jarodzka, and J. Van deWeijer. Eye tracking: A comprehensive guide to methods and measures. OUP Oxford,2011. → page 91[93] B. Hommel, C. S. Chapman, P. Cisek, H. F. Neyedli, J.-H. Song, and T. N. Welsh. Noone knows what attention is. Attention, Perception, & Psychophysics, 81(7):2288–2303,2019. → page 91[94] A. J. Hornof and T. Halverson. Cleaning up systematic error in eye-tracking data byusing required fixation locations. Behavior Research Methods, Instruments, &Computers, 34(4):592–604, 2002. → page 66[95] P. R. Hoskins, K. Martin, and A. Thrush. Diagnostic ultrasound: physics andequipment. Cambridge University Press, 2010. → page 7[96] M. M. Houben, J. Goumans, and J. van der Steen. Recording three-dimensional eyemovements: scleral search coils versus video oculography. Investigative ophthalmology& visual science, 47(1):179–187, 2006. → page 3[97] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto,and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile visionapplications. arXiv preprint arXiv:1704.04861, 2017. → page 92[98] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In Proceedings of theIEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.→ page 91[99] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connectedconvolutional networks. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 4700–4708, 2017. → page 83[100] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund,B. Haghgoo, R. Ball, K. Shpanskaya, et al. Chexpert: A large chest radiograph datasetwith uncertainty labels and expert comparison. In Proceedings of the AAAI conferenceon artificial intelligence, volume 33, pages 590–597, 2019. → pages 82, 92115[101] H. Istance, R. Bates, A. Hyrskykari, and S. Vickers. Snap clutch, a moded approach tosolving the midas touch problem. In Proceedings of the 2008 Symposium on EyeTracking Research & Applications, pages 221–228. ACM, 2008. → page 55[102] A. M. Jarc. Medical devices, systems, and methods using eye gaze tracking for stereoviewer, June 22 2017. US Patent App. 15/126,151. → pages 5, 55[103] A. M. Jarc, H. C. Lin, and J. M. Sorger. Medical devices, systems, and methods usingeye gaze tracking, June 22 2017. US Patent App. 15/126,074. → page 5[104] A. Jha, A. Kumar, S. Pande, B. Banerjee, and S. Chaudhuri. Mt-unet: A novel u-netbased multi-task architecture for visual scene understanding. In 2020 IEEEInternational Conference on Image Processing (ICIP), pages 2191–2195. IEEE, 2020.→ page 83[105] S. Jia and N. D. Bruce. Eml-net: An expandable multi-layer network for saliencyprediction. Image and Vision Computing, 95:103887, 2020. → page 82[106] S. Jia, D. H. Koh, and M. Pomplun. Gaze tracking accuracy maintenance using trafficsign detection. In Adjunct Proceedings of the 10th International Conference onAutomotive User Interfaces and Interactive Vehicular Applications, pages 87–91. ACM,2018. → pages 66, 67[107] J. Jiang, X. Zhou, S. Chan, and S. Chen. Appearance-based gaze tracking: A briefreview. In International Conference on Intelligent Robotics and Applications, pages629–640. Springer, 2019. → page 3[108] C. Jin. Medical endoscope control system based on sight tracking control, July 8 2014.China Patent 104055478A. → page 5[109] S. A. Johansen, J. San Agustin, H. Skovsgaard, J. P. Hansen, and M. Tall. Low cost vs.high-end eye tracking for usability testing. In CHI’11 Extended Abstracts on HumanFactors in Computing Systems, pages 1177–1182. ACM, 2011. → page 66[110] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y.Deng, R. G. Mark, and S. Horng. Mimic-cxr, a de-identified publicly available databaseof chest radiographs with free-text reports. Scientific data, 6(1):1–8, 2019. → pages87, 92, 93, 99[111] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng,Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng. Mimic-cxr-jpg, a large publiclyavailable database of labeled chest radiographs. arXiv preprint arXiv:1901.07042, 2019.→ page 87[112] A. Karargyris, S. Kashyap, I. Lourentzou, J. T. Wu, A. Sharma, M. Tong, S. Abedin,D. Beymer, V. Mukherjee, E. A. Krupinski, et al. Creation and validation of a chestx-ray dataset with eye-tracking and report dictation for ai development. Scientific data,8(1):1–18, 2021. → pages 83, 87, 89, 91, 92, 99116[113] N. Karessli, Z. Akata, B. Schiele, and A. Bulling. Gaze embeddings for zero-shot imageclassification. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 4525–4534, 2017. → pages 83, 93[114] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weighlosses for scene geometry and semantics. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 7482–7491, 2018. → pages 84, 97[115] M. G. Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239–251,1945. → page 42[116] D. Kermany, K. Zhang, and M. Goldbaum. Large dataset of labeled optical coherencetomography (oct) and chest x-ray images. Mendeley Data, 3:10–17632, 2018. → page92[117] W. Khan, N. Zaki, and L. Ali. Intelligent pneumonia identification from chest x-rays: Asystematic literature review. IEEE Access, 2021. → pages 83, 92, 102[118] A. J. King, H. Hochheiser, S. Visweswaran, G. Clermont, and G. F. Cooper.Eye-tracking for clinical decision support: a method to capture automatically whatphysicians are viewing in the emr. AMIA Summits on Translational ScienceProceedings, 2017:512, 2017. → page 6[119] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014. → pages 85, 88[120] A. F. Klaib, N. O. Alsrehin, W. Y. Melhem, H. O. Bashtawi, and A. A. Magableh. Eyetracking algorithms, techniques, tools, and applications with an emphasis on machinelearning and internet of things technologies. Expert Systems with Applications, 166:114037, 2021. → page 4[121] H.-k. Ko, D. M. Snodderly, and M. Poletti. Eye movements between saccades:Measuring ocular drift and tremor. Vision research, 122:93–104, 2016. → page 13[122] A. Kosevoi-Tichie, F. Berghea, V. Vlad, M. Abobului, M. Trandafir, T. Gudu, A. Peltea,M. Duna, L. Groseanu, C. Patrascu, and R. Ionescu. Does eye gaze tracking have theability to assess how rheumatologists evaluate musculoskeletal ultrasound images?,2015. → page 66[123] R. J. Krauzlis and S. G. Lisberger. Temporal properties of visual motion signals for theinitiation of smooth pursuit eye movements in monkeys. Journal of Neurophysiology,72(1):150–162, 1994. → page 2[124] A. Kroner, M. Senden, K. Driessens, and R. Goebel. Contextual encoder–decodernetwork for visual saliency prediction. Neural Networks, 129:261–270, 2020. → pages82, 89[125] E. A. Krupinski. Current perspectives in medical image perception. Attention,Perception, & Psychophysics, 72(5):1205–1217, 2010. → page 6117[126] E. A. Krupinski. Image reading and interpretation. Textbook of Catheter-BasedCardiovascular Interventions, pages 113–124, 2018. → page 6[127] A. Kumar and G. Krol. Binocular infrared oculography. The Laryngoscope, 102(4):367–378, 1992. → page 3[128] M. Ku¨mmerer, T. S. Wallis, and M. Bethge. Deepgaze ii: Reading fixations from deepfeatures trained on object recognition. arXiv preprint arXiv:1610.01563, 2016. → page82[129] J. C. Lagarias, J. A. Reeds, M. H. Wright, and P. E. Wright. Convergence properties ofthe nelder–mead simplex method in low dimensions. SIAM Journal on optimization, 9(1):112–147, 1998. → page 21[130] F. Leccese, G. Salvadori, C. Montagnani, A. Ciconi, and M. Rocca. Lightingassessment of ergonomic workstation for radio diagnostic reporting. InternationalJournal of Industrial Ergonomics, 57:42–54, 2017. → pages 8, 37[131] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eyecoordination for robotic grasping with deep learning and large-scale data collection.The International Journal of Robotics Research, 37(4-5):421–436, 2018. → page 68[132] H. Lewis. The global ultrasound equipment market in 2018, 2018. → page 8[133] Y. Li, Z. Zhang, C. Dai, Q. Dong, and S. Badrigilan. Accuracy of deep learning forautomated detection of pneumonia using chest x-ray images: a systematic review andmeta-analysis. Computers in Biology and Medicine, page 103898, 2020. → page 83[134] Z. Li, I. Tong, L. Metcalf, C. Hennessey, and S. E. Salcudean. Free head movement eyegaze contingent ultrasound interfaces for the da vinci surgical system. IEEE Roboticsand Automation Letters, 3(3):2137–2143, 2018. → page 66[135] G. Liang, X. Wang, Y. Zhang, and N. Jacobs. Weakly-supervised self-training for breastcancer localization. In 2020 42nd Annual International Conference of the IEEEEngineering in Medicine & Biology Society (EMBC), pages 1124–1127. IEEE, 2020. →page 94[136] L. Liebel and M. Ko¨rner. Auxiliary tasks in multi-task learning. arXiv preprintarXiv:1805.06334, 2018. → pages 84, 97[137] D. J. Liebling and S. T. Dumais. Gaze and mouse coordination in everyday work. InProceedings of the 2014 ACM International Joint Conference on Pervasive andUbiquitous Computing: Adjunct Publication, pages 1141–1150. ACM, 2014. → pages68, 70, 73[138] G. W. Lindsay. Attention in psychology, neuroscience, and machine learning. Frontiersin computational neuroscience, 14:29, 2020. → page 91[139] C. Liu, Y. Chen, M. Liu, and B. E. Shi. Using eye gaze to enhance generalization ofimitation networks to unseen environments. IEEE Transactions on Neural Networksand Learning Systems, 32(5):2066–2074, 2020. → page 93118[140] X. Liu and M. Milanova. Visual attention in deep learning: a review. Int Rob Auto J, 4(3):154–155, 2018. → page 81[141] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXivpreprint arXiv:1608.03983, 2016. → page 85[142] L. M. Mack, S. Y. Kim, S. Lee, H. Sangi-Haghpeykar, and W. Lee. Automatedfractional limb volume measurements improve the precision of birth weight predictionsin late third-trimester fetuses. Journal of Ultrasound in Medicine, 36(8):1649–1655,2017. → pages 8, 37[143] N. A. Macmillan and C. D. Creelman. Detection theory: A user’s guide. Psychologypress, 2004. → page 30[144] P. Majaranta and A. Bulling. Eye tracking and eye-based human–computer interaction.In Advances in physiological computing, pages 39–65. Springer, 2014. → page 4[145] M. F. Marmor and E. Zrenner. Standard for clinical electro-oculography. DocumentaOphthalmologica, 85(2):115–124, 1993. → page 2[146] S. Martinez-Conde, S. L. Macknik, X. G. Troncoso, and D. H. Hubel. Microsaccades: aneurophysiological analysis. Trends in neurosciences, 32(9):463–475, 2009. → pages12, 31[147] S. Martinez-Conde, J. Otero-Millan, and S. L. Macknik. The impact of microsaccadeson vision: towards a unified theory of saccadic function. Nature Reviews Neuroscience,14(2):83, 2013. → page 14[148] D. Martinez-Marquez, S. Pingali, K. Panuwatwanich, R. A. Stewart, and S. Mohamed.Application of eye tracking technology in aviation, maritime, and constructionindustries: a systematic review. Sensors, 21(13):4289, 2021. → page 4[149] L. Matin, E. Matin, and D. G. Pearce. Eye movements in the dark during the attempt tomaintain a prior fixation position. Vision Research, 10(9):837–857, 1970. → page 13[150] I. Mehmood, N. Ejaz, M. Sajjad, and S. W. Baik. Prioritization of brain mri volumesusing medical image perception model and tumor region segmentation. Computers inbiology and medicine, 43(10):1471–1483, 2013. → page 7[151] M. L. Mele and S. Federici. A psychotechnological review on eye-tracking systems:towards user experience. Disability and Rehabilitation: Assistive Technology, 7(4):261–281, 2012. → page 4[152] A. Mihali, B. van Opheusden, and W. J. Ma. Bayesian microsaccade detection. Journalof vision, 17(1):13–13, 2017. → pages xi, 12, 19, 30, 32[153] N. Moacdieh and N. Sarter. Clutter in electronic medical records: examining itsperformance and attentional costs using eye tracking. Human factors, 57(4):591–606,2015. → page 6119[154] G. Moody, R. Mark, and A. Goldberger. Physionet: A research resource for studies ofcomplex physiologic and biomedical signals. In Computers in Cardiology 2000. Vol. 27(Cat. 00CH37163), pages 179–182. IEEE, 2000. → page 87[155] C. H. Morimoto and M. R. Mimica. Eye gaze tracking techniques for interactiveapplications. Computer vision and image understanding, 98(1):4–24, 2005. → page 4[156] S. Murphey. Work related musculoskeletal disorders in sonography. Journal ofDiagnostic Medical Sonography, 2017. → page 8[157] S. Murphy and D. E. Need. Voice control of a medical ultrasound scanning machine,Aug. 13 1996. US Patent 5,544,654. → page 54[158] G. P. Mylonas, A. Darzi, and G. Zhong Yang. Gaze-contingent control for minimallyinvasive robotic surgery. Computer Aided Surgery, 11(5):256–266, 2006. → page 6[159] G. P. Mylonas, K.-W. Kwok, A. Darzi, and G.-Z. Yang. Gaze-contingent motorchannelling and haptic constraints for minimally invasive robotic surgery. InInternational Conference on Medical Image Computing and Computer-AssistedIntervention, pages 676–683. Springer, 2008. → page 6[160] D. P. Noonan, G. P. Mylonas, A. Darzi, and G.-Z. Yang. Gaze contingent articulatedrobot control for robot assisted minimally invasive surgery. In 2008 IEEE/RSJInternational Conference on Intelligent Robots and Systems, pages 1186–1191. IEEE,2008. → page 5[161] H. E. A. of British Columbia/Health Science Professionals BargainingAssociation Recruitment and R. Committee. Recommendation report: Diagnosticmedical sonographers. URLhttps://hsabc.org/sites/default/files/uploads/2016-08-31%20HEABC%20HSPBA%20RRC%20Recommendations%20Report%20FV.pdf. → page 41[162] J. Otero-Millan, J. L. A. Castro, S. L. Macknik, and S. Martinez-Conde. Unsupervisedclustering method to detect microsaccades. Journal of vision, 14(2):18–18, 2014. →page 12[163] J. Otero-Millan et al. An oculomotor continuum from exploration to fixation.Proceedings of the National Academy of Sciences, 110(15):6175–6180, 2013. → page11[164] T. Oyama and T. Yamanaka. Fully convolutional densenet for saliency-map prediction.In 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR), pages 334–339.IEEE, 2017. → page 82[165] T. Oyama and T. Yamanaka. Influence of image classification accuracy on saliency mapestimation. CAAI Transactions on Intelligence Technology, 3(3):140–152, 2018. →page 82[166] O. Pallotta and A. Roberts. Musculoskeletal pain and injury in sonographers, causes andsolutions. Sonography, 4(1):5–12, 2017. → page 8120[167] J. Pan, E. Sayrol, X. Giro-i Nieto, K. McGuinness, and N. E. O’Connor. Shallow anddeep convolutional networks for saliency prediction. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 598–606, 2016. → page82[168] A. Papoutsaki, A. Gokaslan, J. Tompkin, Y. He, and J. Huang. The eye of the typer: abenchmark and analysis of gaze behavior during typing. In Proceedings of the 2018ACM Symposium on Eye Tracking Research & Applications, page 16. ACM, 2018. →pages 68, 69, 70, 71, 75, 76, 77, 78, 80[169] J. Park, S. Woo, J.-Y. Lee, and I. S. Kweon. Bam: Bottleneck attention module. arXivpreprint arXiv:1807.06514, 2018. → page 103[170] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin,N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deeplearning library. Advances in neural information processing systems, 32:8026–8037,2019. → page 88[171] A. Patra, Y. Cai, P. Chatelain, H. Sharma, L. Drukker, A. T. Papageorghiou, and J. A.Noble. Efficient ultrasound image analysis models with sonographer gaze assisteddistillation. In International Conference on Medical Image Computing andComputer-Assisted Intervention, pages 394–402. Springer, 2019. → page 7[172] L. Pelissier, B. Zhang, and T. Bobovsky. Highly configurable medical ultrasoundmachine and related methods, Aug. 9 2016. US Patent 9,408,587. → page 54[173] E. Pons, L. M. Braun, M. M. Hunink, and J. A. Kors. Natural language processing inradiology: a systematic review. Radiology, 279(2):329–343, 2016. → pages 8, 37[174] D. Purves, G. Augustine, D. Fitzpatrick, L. Katz, A. LaMantia, J. McNamara, andS. Williams. Neuroscience 2nd edition. sunderland (ma) sinauer associates. Types ofEye Movements and Their Functions, 2001. → pages 1, 2[175] M. E. Raichle. Two views of brain function. Trends in cognitive sciences, 14(4):180–190, 2010. → page 1[176] I. Rakhmatulin and A. T. Duchowski. Deep neural networks for low-cost eye tracking.Procedia Computer Science, 176:685–694, 2020. → page 3[177] P. K. Ray and E. Saha. Ergonomic performance measurement and evaluation forworksystems in healthcare. In Advances in Human Factors and Ergonomics inHealthcare, pages 329–342. Springer, 2017. → page 8[178] N. Reddy, S. Jain, P. Yarlagadda, and V. Gandhi. Tidying deep saliency predictionarchitectures. In 2020 IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS), pages 10241–10247. IEEE, 2020. → page 89[179] B. Reiner. Visually directed human-computer interaction for medical applications,Dec. 12 2017. US Patent 9,841,811. → pages 6, 55121[180] M. Reyes, R. Meier, S. Pereira, C. A. Silva, F.-M. Dahlweid, H. v. Tengg-Kobligk,R. M. Summers, and R. Wiest. On the interpretability of artificial intelligence inradiology: challenges and opportunities. Radiology: artificial intelligence, 2(3):e190043, 2020. → page 6[181] M. Rolfs. Microsaccades: small steps on a long way. Vision research, 49(20):2415–2441, 2009. → page 11[182] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedicalimage segmentation. In International Conference on Medical image computing andcomputer-assisted intervention, pages 234–241. Springer, 2015. → page 82[183] K. Ruhland, C. E. Peters, S. Andrist, J. B. Badler, N. I. Badler, M. Gleicher, B. Mutlu,and R. McDonnell. A review of eye gaze in virtual agents, social robotics and hci:Behaviour generation, user interaction and perception. In Computer Graphics Forum,volume 34, pages 299–326. Wiley Online Library, 2015. → pages 11, 55[184] A. Saad, T. Loupas, D. Hull, and S. J. Hill. Ultrasound system with dynamicallyautomated doppler flow settings as a sample volume is moved, Jan. 1 2019. US Patent10,166,006. → pages 8, 37[185] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Invertedresiduals and linear bottlenecks. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 4510–4520, 2018. → page 102[186] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, andD. Rueckert. Attention gated networks: Learning to leverage salient regions in medicalimages. Medical image analysis, 53:197–207, 2019. → page 91[187] C. Scholl and H. Salisbury. Barriers to performing ergonomic scanning techniques forsonographers. Journal of Diagnostic Medical Sonography, 33(5):406–411, 2017. →page 8[188] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.Grad-cam: Visual explanations from deep networks via gradient-based localization. InProceedings of the IEEE international conference on computer vision, pages 618–626,2017. → pages 91, 103[189] K. Semmelmann and S. Weigelt. Online webcam-based eye tracking in cognitivescience: A first look. Behavior Research Methods, 50(2):451–465, 2018. → page 3[190] O. Sener and V. Koltun. Multi-task learning as multi-objective optimization. arXivpreprint arXiv:1810.04650, 2018. → page 84[191] S. Serte, A. Serener, and F. Al-Turjman. Deep learning in medical imaging: A briefreview. Transactions on Emerging Telecommunications Technologies, page e4080,2020. → page 83122[192] H. Sharma, L. Drukker, A. T. Papageorghiou, and J. A. Noble. Multi-modal learningfrom video, eye tracking, and pupillometry for operator skill characterization in clinicalfetal ultrasound. In 2021 IEEE 18th International Symposium on Biomedical Imaging(ISBI), pages 1646–1649. IEEE, 2021. → page 93[193] D. Sheynikhovich, M. Be´cu, C. Wu, and A. Arleo. Unsupervised detection ofmicrosaccades in a high-noise regime. Journal of vision, 18(6):19–19, 2018. → page 12[194] L. Sidenmark and A. Lundstro¨m. Gaze behaviour on interacted objects during handinteraction in virtual reality for eye tracking calibration. In Eleventh edition of the ACMSymposium on Eye Tracking Research & Applications (ETRA 2019), 2019. → page 67[195] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scaleimage recognition. arXiv preprint arXiv:1409.1556, 2014. → page 102[196] P. Sinn and R. Engbert. Small saccades versus microsaccades: Experimental distinctionand model-based unification. Vision research, 118:132–143, 2016. → page 11[197] R. Smith-Bindman, M. L. Kwan, E. C. Marlow, M. K. Theis, W. Bolch, S. Y. Cheng,E. J. Bowles, J. R. Duncan, R. T. Greenlee, L. H. Kushi, et al. Trends in use of medicalimaging in us health care systems and in ontario, canada, 2000-2016. Jama, 322(9):843–856, 2019. → page 9[198] S. R. Snare, H. Torp, F. Orderud, and B. O. Haugen. Real-time scan assistant forechocardiography. IEEE Transactions on Ultrasonics, Ferroelectrics, and FrequencyControl, 59(3):583–589, 2012. → page 8[199] J. N. Stember, H. Celik, D. Gutman, N. Swinburne, R. Young, S. Eskreis-Winkler,A. Holodny, S. Jambawalikar, B. J. Wood, P. D. Chang, et al. Integrating eye trackingand speech recognition accurately annotates mr brain images for deep learning: proof ofprinciple. Radiology: Artificial Intelligence, 3(1):e200047, 2020. → page 6[200] D. Stoyanov, G. P. Mylonas, and G.-Z. Yang. Gaze-contingent 3d control for focusedenergy ablation in robotic assisted surgery. In International Conference on MedicalImage Computing and Computer-Assisted Intervention, pages 347–355. Springer, 2008.→ page 5[201] H. Straka and N. Dieringer. Basic organization principles of the vor: lessons from frogs.Progress in neurobiology, 73(4):259–309, 2004. → page 2[202] D. A. Strassner, C. Ji, V. T. Shamdasani, and I. A. M. Mohammed. Ultrasound systemsand methods for automated fetal heartbeat identification, June 12 2018. US Patent9,993,224. → pages 8, 37[203] Y. Sun, M. Zhao, K. Hu, and S. Fan. Visual saliency prediction using multi-scaleattention gated network. Multimedia Systems, pages 1–9, 2021. → page 82[204] G. Sze, E. I. Bluth, C. E. Bender, and J. R. Parikh. Work-related injuries of radiologistsand possible ergonomic solutions: recommendations from the acr commission on123human resources. Journal of the American College of Radiology, 14(10):1353–1358,2017. → page 8[205] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi. Inception-v4, inception-resnetand the impact of residual connections on learning. In Thirty-first AAAI conference onartificial intelligence, 2017. → pages 88, 102[206] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neuralnetworks. In International Conference on Machine Learning, pages 6105–6114. PMLR,2019. → pages 83, 91[207] M. Tan and Q. V. Le. Efficientnetv2: Smaller models and faster training. arXiv preprintarXiv:2104.00298, 2021. → pages 89, 98[208] T. Tarkiainen, M. Turpeinen, M. Haapea, E. Liukkonen, and J. Niinima¨ki. Investigatingerrors in medical imaging: medical malpractice cases in finland. Insights into Imaging,12(1):1–10, 2021. → page 9[209] T. Tieleman, G. Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by a runningaverage of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26–31, 2012. → page 85[210] G. Tourassi, S. Voisin, V. Paquit, and E. Krupinski. Investigating the link betweenradiologists’ gaze, diagnostic decision, and image content. Journal of the AmericanMedical Informatics Association, 20(6):1067–1075, 2013. → page 6[211] S. Tripathi and B. Guenter. A statistical approach to continuous self-calibrating eye gazetracking for head-mounted virtual reality systems. In 2017 IEEE Winter Conference onApplications of Computer Vision (WACV), pages 862–870. IEEE, 2017. → pages 67, 72[212] D. USA. Diagnostic medical sonography & ultrasound technicians. URLhttps://datausa.io/profile/cip/diagnostic-medical-sonography-ultrasound-technician#demographics. → page 41[213] J. N. van der Geest and M. A. Frens. Recording eye movements with video-oculographyand scleral search coils: a direct comparison of two methods. Journal of neurosciencemethods, 114(2):185–195, 2002. → page 3[214] A. Van Opstal and J. Van Gisbergen. Skewness of saccadic velocity profiles: a unifyingparameter for normal and slow saccades. Vision research, 27(5):731–745, 1987. →page 14[215] H. L. Van Trees. Detection, estimation, and modulation theory, part I: detection,estimation, and linear modulation theory. John Wiley & Sons, 2004. → pages 16, 22[216] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, andL. Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEETransactions on Pattern Analysis and Machine Intelligence, 2021. → page 84124[217] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, andI. Polosukhin. Attention is all you need. In Advances in neural information processingsystems, pages 5998–6008, 2017. → page 91[218] M. Visentini-Scarzanella, G. P. Mylonas, D. Stoyanov, and G.-Z. Yang. i-brush: Agaze-contingent virtual paintbrush for dense 3d reconstruction in robotic assistedsurgery. In International Conference on Medical Image Computing andComputer-Assisted Intervention, pages 353–360. Springer, 2009. → page 6[219] B. A. Wandell. Foundations of vision, volume 8. sinauer Associates Sunderland, MA,1995. → page 11[220] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu.Score-cam: Score-weighted visual explanations for convolutional neural networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognitionworkshops, pages 24–25, 2020. → page 91[221] S. Wang, X. Ouyang, T. Liu, Q. Wang, and D. Shen. Follow my eye: Using gaze tosupervise computer-aided diagnosis. IEEE Transactions on Medical Imaging, 2022. →pages 95, 96[222] W. Wang, J. Shen, J. Xie, M.-M. Cheng, H. Ling, and A. Borji. Revisiting videosaliency prediction in the deep learning era. IEEE transactions on pattern analysis andmachine intelligence, 43(1):220–237, 2019. → page 82[223] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8:Hospital-scale chest x-ray database and benchmarks on weakly-supervised classificationand localization of common thorax diseases. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 2097–2106, 2017. → pages 82, 92[224] J. Ward and J. Coats. Comparison of the backjoy sitsmart relief and spine buddy lt1 h/cergonomic chair supports on short-term neck and back pain. Journal of Manipulativeand Physiological Therapeutics, 40(1):41–49, 2017. → pages 8, 37[225] M. J. Washburn, B. M. Hawley, and S. D. Prichard. Voice control of a generic inputdevice for an ultrasound system, Apr. 13 2010. US Patent 7,698,142. → page 54[226] M. B. Weinger, D. J. Gardner-Bonneau, and M. E. Wiklund. Handbook of humanfactors in medical device design. CRC Press, 2010. → page 54[227] M. Wichrowski. Usability engineering in the prototyping process of software userinterfaces for mobile medical ultrasound devices. Computer Science, 16, 2015. → page54[228] Wikipedia. Saccade, 2021. URL https://en.wikipedia.org/wiki/Saccade. accessed17-Nov-2021. → page 1[229] Wikipedia. Smooth pursuit, 2021. URLhttps://en.wikipedia.org/wiki/Smooth pursuit. accessed 17-Nov-2021. → page 1125[230] C. K. Williams and C. E. Rasmussen. Gaussian processes for machine learning,volume 2. MIT press Cambridge, MA, 2006. → pages 71, 72, 74[231] D. F. Williamson, R. A. Parker, and J. S. Kendrick. The box plot: a simple visualmethod to interpret data. Annals of internal medicine, 110(11):916–921, 1989. → pagesxiv, 88[232] K. C. Wong, M. Moradi, J. Wu, A. Pillai, A. Sharma, Y. Gur, H. Ahmad, M. S.Chowdary, J. Chiranjeevi, K. K. R. Polaka, et al. A robust network architecture to detectnormal chest x-ray radiographs. In 2020 IEEE 17th International Symposium onBiomedical Imaging (ISBI), pages 1851–1855. IEEE, 2020. → pages 92, 102[233] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon. Cbam: Convolutional block attentionmodule. In Proceedings of the European conference on computer vision (ECCV), pages3–19, 2018. → page 103[234] C.-C. Wu and J. M. Wolfe. Eye movements in medical image perception: a selectivereview of past, present and future. Vision, 3(2):32, 2019. → page 9[235] J. T. Wu, K. C. Wong, Y. Gur, N. Ansari, A. Karargyris, A. Sharma, M. Morris,B. Saboury, H. Ahmad, O. Boyko, et al. Comparison of chest radiograph interpretationsby artificial intelligence algorithm vs radiology residents. JAMA network open, 3(10):e2022779–e2022779, 2020. → pages 92, 102[236] H. J. Wyatt and J. Pola. Smooth pursuit eye movements under open-loop andclosed-loop conditions. Vision research, 23(10):1121–1131, 1983. → page 2[237] Z. Xiaoli. Gaze contingent control system for a robotic laparoscope holder, Jan. 232014. US Patent App. 13/941,632. → page 5[238] X. Yang and I. Krajbich. Webcam-based online eye-tracking for behavioral research.2020. → page 3[239] L. M. Yudkovitch, F. Farrokhnia, and R. Chiao. Method and apparatus for natural voicecontrol of an ultrasound machine, July 24 2007. US Patent 7,247,139. → page 54[240] S. Zhai, C. Morimoto, and S. Ihde. Manual and gaze input cascaded (magic) pointing.In Proceedings of the SIGCHI conference on Human Factors in Computing Systems,pages 246–253. ACM, 1999. → page 54[241] R. Zhang, A. Saran, B. Liu, Y. Zhu, S. Guo, S. Niekum, D. Ballard, and M. Hayhoe.Human gaze assisted artificial intelligence: a review. In IJCAI: Proceedings of theConference, volume 2020, page 4951. NIH Public Access, 2020. → page 93[242] X. Zhang, N. Smith, and A. Webb. Medical imaging. In Biomedical InformationTechnology, pages 3–27. Elsevier, 2008. → page 9[243] Y. Zhang and Q. Yang. A survey on multi-task learning. IEEE Transactions onKnowledge and Data Engineering, 2021. → page 82126[244] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features fordiscriminative localization. In Proceedings of the IEEE conference on computer visionand pattern recognition, pages 2921–2929, 2016. → pages 91, 93[245] H. Zhu. Statistical Modeling and Tracking of Artery Wall Motion in Ultrasound Images.PhD thesis, McGill University Libraries, 2018. → page 17[246] H. Zhu, S. Salcudean, and R. Rohling. The neyman pearson detection of microsaccadeswith maximum likelihood estimation of parameters. Journal of vision, 19(13):17–17,2019. → pages 68, 72[247] H. Zhu, S. Salcudean, and R. Rohling. The neyman pearson detection of microsaccadeswith maximum likelihood estimation of parameters. Journal of vision, 19(13):17–17,2019. → pages v, 91[248] H. Zhu, S. E. Salcudean, and R. N. Rohling. A novel gaze-supported multimodalhuman–computer interaction for ultrasound machines. International journal ofcomputer assisted radiology and surgery, 14(7):1107–1115, 2019. → pagesvi, xiv, 8, 11, 37, 40, 66, 75, 76, 87, 91[249] H. Zhu, R. N. Rohling, and S. E. Salcudean. Hand-eye coordination-based implicitre-calibration method for gaze tracking on ultrasound machines: a statistical approach.International journal of computer assisted radiology and surgery, 15(5):837–845, 2020.→ pages vi, 37, 91[250] H. Zhu, Y. Halwani, R. Rohling, S. Fels, and T. Salcudean. A unified representation ofcontrol logic in human-ultrasound machine interaction. IEEE Journal of Biomedicaland Health Informatics, 2022. → page v[251] B. Zuber, L. Stark, and G. Cook. Microsaccades and the velocity-amplitude relationshipfor saccadic eye movements. Science, 150(3702):1459–1460, 1965. → pages 11, 31127Appendix ADerivation of the Equations (2.27) and(2.29)From Equation (2.26), we have:ln f(Ck2:k1θ ,H1)=−12lndet(2piΣ)− 12[(Ck2:k1−µ)T·Σ−1 (Ck2:k1−µ)] (A.1)and we have (Ck2:k1−µ)T ·Σ−1 (Ck2:k1−µ)=Ck2:k1TΣ−1Ck2:k1−2Ck2:k1 TΣ−1µ +µ TΣ−1µ(A.2). By substitute Equation (A.2) to Equation (A.1), and drop constant terms, we can simplifyEquation (2.27) as:θˆML = argmaxθ(2µ TΣ−1Ck2:k1−µ TΣ−1µ). (A.3)Since for µ , the parameter θ1 is a scaling factor, we can defineµ¯ =1θ1µ (A.4)such that each element in µ¯ is independent of θ1:µ¯i =exp[−(t1+ i−1θ2)θ3]−exp[−(t1+ iθ2)θ3] (A.5)128where µ¯i is the ith element in µ¯ . Therefore, we can rewrite Equation (2.28) as:∂ ln f(Ck2:k1θ ,H1)∂θ1=∂∂θ1[2θ1µ¯ TΣ−1Ck2:k1−θ12µ¯ TΣ−1µ¯]=2µ¯ TΣ−1Ck2:k1−2θ1µ¯ TΣ−1µ¯=0(A.6)and, solving it, we can come to equation (2.29) as:θˆ1ML =µ¯ TΣ−1Ck2:k1µ¯ TΣ−1µ¯. (A.7)129Appendix BUS machine function usage frequencyscoresUsage frequency scores for each function are: B-mode Gain (98), Focus (97), Depth (92),Freeze (89), Colour Doppler (88), TGC (80), Measurements (78), Annotation (78), Capture(74), PW Doppler (63), Frequency (49), Zoom (46), Transducer Options (42), Cine Tools (42),Preset Selection (41), Patient Tools (34), Width (34), M-Mode (22), Dynamic Range (20), SplitScreen (15), Auto Optimization (15), Scale (15), End Exam (13), Panoramic View (13), OutputPower (12), PRF (10), Angle Steer (10), Body Markers (8), Virtual Convex (8), CalculationPackages (8), Tilt (6), Harmonics (6), 3D View (6), Logiq View (5), Resolution (5), Persistence(5), Baseline (4), Color Frequency (4), Rejection (4), Contrast (3), Record (3), Send Exam (3),Invert (3), B Flow (3), Compression (3), Filters (3), Cross Beam (3), Print Image (Paper) (3),Gray Maps (3), Frame Speed Average (3), Color Maps (3), Post-processing (2), High LevelMachine Settings (2), Screen Size (2), Spectral Trace (2), Duplex (2), Dual Mode (2), ColorGain (2), Penetration (2), Compound Mode (2), Packet Size (1), Spatial Compounding (1), Col-orization (1), B-scan (1), Doppler Energy (1), Volume (1), PW Sample Gate Size (1), Protocol(1).130Appendix CNasa task load indexIn the NASA TLX, the workload of a given task is evaluated from 6 sources, which are mentaldemand, physical demand, temporal demand, effort, performance, and frustration level [83, 84].To quantify the overall workload of a task with NASA TLX, two steps are involved in theevaluation process.The first step is to identify the sources of workload for the given task. As different tasks mayimpose different demands for the task handler, we need to understand to what percentage theoverall workload is composed of the workloads from the 6 sources. To quantify the percentage,the study participant is asked to compare all pairs of the 6 workload sources and determinewhich kind of workload is higher in the given task. To be more specific, the study participant isask to compare the select which workload source is more dominant in the task in the followingquestions:1. Effort or performance2. Temporal demand or frustration3. Temporal demand or effort4. Physical demand or frustration5. Performance or frustration6. Physical demand or temporal demand7. Physical demand or performance8. Temporal demand or mental demand9. Frustration or effort13110. Performance or mental demand11. Performance or temporal demand12. Mental demand or effort13. Mental demand or physical demand14. Effort or physical demand15. Frustration or mental demandBased on the answers to the 15 questions, we can determine the weights for different workloadsources.The second step is to let the study participant rate the intensity of the 6 sources of workloadon a scale from low to high with a scoring system. Then the overall workload can be calculatedas the weighted average of the scores to the 6 sources of workload.132
Thesis/Dissertation
eng
Biomedical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-ShareAlike 4.0 International
Graduate
Rohling, Robert N.
Salcudean, Septimiu Edmund
Gaze tracking for human-ultrasound machine interaction and medical image understanding
Text