Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Verma, Shreya
Master of Applied Science - MASc
University of British Columbia
Measuring vital signs is crucial in healthcare as they provide essential information about a human body's physiological state. Particularly, the initial months following birth are crucial for the survival and well-being of newborns. Existing methods face challenges such as estimating incorrect respiratory rate values in low-light conditions, the inability to detect shallow breathing, overestimating respiratory values in darker skin tones and the need for contact sensors that may disturb sensitive infants. In this study, we propose a novel video-based respiratory rate estimation method for infants in the Neonatal Intensive Care Unit. Our method overcomes challenges by utilizing non-contact depth-sensing camera technology, which mitigates the impact of skin pigmentation on measurement accuracy. We conducted a study with 13 infants, recording uninterrupted 4-hour video sessions. We evaluated the algorithm's performance and robustness in various light conditions by analyzing randomly selected videos. We compared the respiratory rate estimated by our depth-sensing camera method with both a nurse practitioner's manually annotated reference respiratory rate and the respiratory rate estimated from red-green-blue (RGB) videos of the infants recorded during the same period as the depth videos. Our results record a bias of 0.125 bpm and a root mean squared difference accuracy of 1.058 bpm. Moreover, our proposed method exhibits a 26% increase in accuracy compared to the RGB camera-based method for respiratory rate estimation in infants. The Pearson correlation coefficient of 0.87 (p < 0.001) indicates a strong correlation, and the intraclass correlation coefficient of 0.77 (95%CI: 0.68 - 0.84 ) demonstrates good reliability. The Bland-Altman analysis of the proposed algorithm shows higher agreement between the values estimated by our developed method and the visually counted respiratory rate than the agreement between the respiratory rate obtained from the impedance sensors and the reference respiratory rate, the agreement between a former EVM-based method, and the respiratory rate estimated during the RGB-camera-based study. Our preliminary study shows that the proposed algorithm can successfully estimate the respiratory rate in the regular clinical care environment.
One Breath at a TimeA Feasibility StudybyShreya VermaB.Tech, Vellore Institute of Technology, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Electrical and Computer Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)May 2024© Shreya Verma 2024The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:One Breath at a Time: A Feasibility Studysubmitted by Shreya Verma in partial fulfillment of the requirements forthe degree of Master of Applied Science in Electrical and ComputerEngineering.Examining Committee:Guy Dumont, Professor, Electrical and Computer Engineering, UBCSupervisorLiisa Holsti, Professor, Faculty of Medicine, UBCSupervisory Committee MemberPeyman Servati, Professor, Electrical and Computer Engineering, UBCSupervisory Committee MemberiiAbstractMeasuring vital signs is crucial in healthcare as they provide essential infor-mation about a human body’s physiological state. Particularly, the initialmonths following birth are crucial for the survival and well-being of new-borns. Existing methods face challenges such as estimating incorrect res-piratory rate values in low-light conditions, the inability to detect shallowbreathing, overestimating respiratory values in darker skin tones and theneed for contact sensors that may disturb sensitive infants. In this study,we propose a novel video-based respiratory rate estimation method for in-fants in the Neonatal Intensive Care Unit. Our method overcomes challengesby utilizing non-contact depth-sensing camera technology, which mitigatesthe impact of skin pigmentation on measurement accuracy. We conducteda study with 13 infants, recording uninterrupted 4-hour video sessions. Weevaluated the algorithm’s performance and robustness in various light condi-tions by analyzing randomly selected videos. We compared the respiratoryrate estimated by our depth-sensing camera method with both a nurse prac-titioner’s manually annotated reference respiratory rate and the respiratoryrate estimated from red-green-blue (RGB) videos of the infants recordedduring the same period as the depth videos.Our results record a bias of 0.125 bpm and a root mean squared dif-ference accuracy of 1.058 bpm. Moreover, our proposed method exhibits a26% increase in accuracy compared to the RGB camera-based method forrespiratory rate estimation in infants. The Pearson correlation coefficient of0.87 (p < 0.001) indicates a strong correlation, and the intraclass correlationcoefficient of 0.77 (95%CI: 0.68 − 0.84 ) demonstrates good reliability. TheBland-Altman analysis of the proposed algorithm shows higher agreementbetween the values estimated by our developed method and the visuallycounted respiratory rate than the agreement between the respiratory rateobtained from the impedance sensors and the reference respiratory rate,the agreement between a former EVM-based method, and the respiratoryrate estimated during the RGB-camera-based study. Our preliminary studyshows that the proposed algorithm can successfully estimate the respiratoryrate in the regular clinical care environment.iiiLay SummaryThe respiratory rate is a vital sign of overall health and well-being. Partic-ularly, the initial months following birth are crucial for the survival of thenewborns. Existing methods face challenges such as estimating incorrectrespiratory rate values in low-light conditions, the inability to detect shal-low breathing, overestimating respiratory values in darker skin tones andthe need for contact sensors that may disturb sensitive infants. To addressthis need, we conducted a feasibility study to investigate and assess the useof depth-sensing cameras for monitoring respiratory rates. Our methodol-ogy mitigates challenges, such as motion artifacts and heartbeat extraction,leveraging signal processing techniques. The algorithm’s performance wasevaluated using statistical methods for 70 randomly selected videos.ivPrefaceThe data was acquired for an observational study conducted between March2019 and November 2019 in the Neonatal Intensive Care Unit at the BCWomen’s Hospital is located in Vancouver, British Columbia, Canada.• Informed consent was obtained from every infant’s parents or LegalGuardians. The University approved the study of British ColumbiaChildren’s & Women’s Research Ethics Board (H18-02406)• Shreya Verma is the primary researcher and author for all the re-search contributions made in this thesis. I reviewed the literature todevelop the original idea and identify the research gaps and problems.I formulated the research problems, coordinated with Medtronics toprocure the data, prepared the data for analysis, developed solutionapproaches, and performed mathematical analysis.• Dr. Guy Dumont, as my supervisor, helped me identify the researchproblem and provided valuable guidance and technical feedback through-out various stages of the project. He helped me steer the research andexplore wavelet transforms for denoising the respiratory volume signal.• Dr. Soodeh Ahani helped formulate the original approach and led thedata collection process. She also reviewed the results and providedconstructive feedback. Her research on RGB videos helped me under-stand the requirements and milestones of my preliminary study. Thiswork is an extension of her work based on Video-based RespiratoryRate Estimation for Infants in the NICU.• Dr. Pascal Lavoie’s opinions and feedback as a neonatologist shapedthe research. Having a clinician on the team was extremely valuablein helping us understand the requirements and keep us in check.• Dr. Liisa Holsti mentored me at every step and helped me brainstormthe correct methods to evaluate my algorithm. She helped me decideon ideal evaluation methods and also structure my thesis.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xivDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1.1 The need for Non-Contact Methods . . . . . . . . . . 21.1.2 Clinical Use Cases . . . . . . . . . . . . . . . . . . . . 21.1.2.1 Sepsis . . . . . . . . . . . . . . . . . . . . . 21.1.2.2 Apnea . . . . . . . . . . . . . . . . . . . . . 31.1.2.3 Respiratory Distress Syndrome (RDS) . . . 31.2 Research Question and Contribution . . . . . . . . . . . . . . 31.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Contact Based Respiratory Rate Monitoring . . . . . . . . . 62.1.1 Impedance Pneumography . . . . . . . . . . . . . . . 62.1.2 Acoustic Monitoring . . . . . . . . . . . . . . . . . . . 62.1.3 Airflow Monitoring . . . . . . . . . . . . . . . . . . . 7viTable of Contents2.1.4 Carbon dioxide Monitoring . . . . . . . . . . . . . . . 72.1.5 Photoplethysmogram . . . . . . . . . . . . . . . . . . 72.1.6 Electrocardiogram . . . . . . . . . . . . . . . . . . . . 82.2 Non-Contact Based Respiratory Rate Monitoring . . . . . . 82.2.1 Laser . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2.2 Thermal Camera . . . . . . . . . . . . . . . . . . . . 82.2.3 Radar . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.4 RGB Camera . . . . . . . . . . . . . . . . . . . . . . 102.2.5 Depth Camera . . . . . . . . . . . . . . . . . . . . . . 112.2.5.1 Respiratory Rate Benchtop/Lab studies . . 112.2.5.2 Clinical Studies . . . . . . . . . . . . . . . . 122.2.6 Comparison Study . . . . . . . . . . . . . . . . . . . . 133 Estimation of Respiratory Rate of Infants in Neonatal In-tensive Care Unit . . . . . . . . . . . . . . . . . . . . . . . . . . 153.1 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Key Contributions . . . . . . . . . . . . . . . . . . . . . . . . 153.3 Introduction and Methodology . . . . . . . . . . . . . . . . . 163.4 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.4.1 Limitations and Opportunities . . . . . . . . . . . . . 173.5 Overview of Dataset . . . . . . . . . . . . . . . . . . . . . . . 183.5.1 Introduction to Depth Camera . . . . . . . . . . . . . 193.5.2 Data Collection . . . . . . . . . . . . . . . . . . . . . 213.5.3 Challenges and Limitations . . . . . . . . . . . . . . . 233.5.3.1 Data interpretation and transfer . . . . . . . 233.5.3.2 Gold Reference Values . . . . . . . . . . . . 243.5.4 Data Pre-processing and Visualization . . . . . . . . 253.5.5 Introduction to Optimal and Suboptimal Videos . . . 273.6 Requirement of Region of Interest . . . . . . . . . . . . . . . 313.6.1 Selecting Ideal Region of Interest . . . . . . . . . . . 333.7 Respiratory Volume Analysis . . . . . . . . . . . . . . . . . . 333.7.1 Related work . . . . . . . . . . . . . . . . . . . . . . . 343.7.2 Extracting Respiratory Volume Signal . . . . . . . . . 343.8 Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.8.1 Related work . . . . . . . . . . . . . . . . . . . . . . . 373.8.1.1 Foundational Background . . . . . . . . . . 373.8.1.2 Literarture Review . . . . . . . . . . . . . . 393.8.2 Denoising Methodology . . . . . . . . . . . . . . . . . 413.9 Peak Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 493.9.1 Related work . . . . . . . . . . . . . . . . . . . . . . . 49viiTable of Contents3.9.2 Adaptive Peak Prominence . . . . . . . . . . . . . . . 504 Evaluation of the Proposed Algortihm . . . . . . . . . . . . 564.1 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 574.1.1 Bias and Root Mean Squared Difference . . . . . . . 574.1.2 Bland Altman Analysis . . . . . . . . . . . . . . . . . 584.1.3 Pearson Correlation . . . . . . . . . . . . . . . . . . . 584.1.4 ICC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 Results of Proposed Method . . . . . . . . . . . . . . . . . . 594.2.1 Bias and Root Mean Squared Difference . . . . . . . 604.2.2 Bland Altman Analysis . . . . . . . . . . . . . . . . . 604.2.3 Pearson Correlation . . . . . . . . . . . . . . . . . . . 624.2.4 ICC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635 Discussion and Future Works . . . . . . . . . . . . . . . . . . 645.1 Our Contributions . . . . . . . . . . . . . . . . . . . . . . . . 645.1.1 Neonatal Intensive Care Unit Tool . . . . . . . . . . . 675.2 Limitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72AppendixA MATLAB Code . . . . . . . . . . . . . . . . . . . . . . . . . . . 86viiiList of Tables1.1 Overview of Structure of the Thesis . . . . . . . . . . . . . . 52.1 Overview of Camera-based technologies for Neonatal Respi-ratory Rate (RR) measurement. . . . . . . . . . . . . . . . . . 143.1 Overview of Neonatal Data Demographics . . . . . . . . . . . 223.2 Specifications of the analyzed short-videos of each case of thedataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.3 Overview of signal processing techniques used in camera-basedfor neonatal respiratory monitoring.. . . . . . . . . . . . . . . . 394.1 Interpretation of the Intraclass Correlation Coefficient val-ues.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.2 Results of BA Analysis (MOD±LOAs) of the proposed methodcompared with RGB-camera based methodology, impedancesensor [47], EVM based method [33] .. . . . . . . . . . . . . . . 624.3 ICC value of the proposed depth-sensing RR estimation al-gorithm .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63ixList of Figures3.1 Block Diagram of the proposed Depth-video based RR esti-mation method. . . . . . . . . . . . . . . . . . . . . . . . . . . 163.2 Frame samples captured using RGB Camera . . . . . . . . . . 193.3 Frame samples captured using Depth Camera . . . . . . . . . 203.4 Frame samples captured using Depth Camera, converted us-ing false colours . . . . . . . . . . . . . . . . . . . . . . . . . . 213.5 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.6 Neonatal Intensive Care Unit camera set up . . . . . . . . . . 233.7 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.8 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.9 Challenges faced visualizing the depth data . . . . . . . . . . 273.10 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.11 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.12 Region of Interest Selection . . . . . . . . . . . . . . . . . . . 323.13 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.14 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.15 Respiratory Volume Signal . . . . . . . . . . . . . . . . . . . . 363.16 Process flow of wavelet decomposition using Discrete Wavelettransform. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.17 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.18 Multilevel wavelet decomposition in Discrete Wavelet Trans-form. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42xList of Figures3.19 Application of Discrete Wavelet Transforms Filter bank the-ory using daubechie 4 mother wavelet as discussed in [25]. . . 433.20 Application of Discrete Wavelet Transforms Filter bank the-ory using daubechie 10 level 4 mother wavelet as discussed in[29]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.21 Application of Discrete Wavelet Transforms Filter bank the-ory using daubechie 3 level 4 mother wavelet as discussed in[49]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.22 Application of Discrete Wavelet Transforms Filter bank the-ory using Haar mother wavelet as discussed in [49] . . . . . . 463.23 Application of Discrete Wavelet Transforms Filter bank the-ory using Symlet 4 Level 6 mother wavelet as discussed in[96] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.24 Application of Discrete Wavelet Transforms Filter bank the-ory using Symlet 4 Level 3 mother wavelet as discussed in ourproposed methodology . . . . . . . . . . . . . . . . . . . . . . 483.25 Extracted Respiratory Volume Signal denoised applying Dis-crete Wavelet Transform. . . . . . . . . . . . . . . . . . . . . 493.26 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.27 An example of the upper and lower envelopes of the breathpulse for automatic peak prominence threshold value estima-tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.28 Detected Peaks . . . . . . . . . . . . . . . . . . . . . . . . . . 533.29 Representative frame samples captured in two different light-ing conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.30 An example of the process of the proposed methodology. . . 554.1 BA plot of RRref vs RRdepth . . . . . . . . . . . . . . . . . . 604.2 BA plot of RRref vs RRrgb . . . . . . . . . . . . . . . . . . . 614.3 Scatterplot of RRdepth against RRref aggregated over allsubjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62xiList of AbbreviationsBA Bland-Altman.bpm breaths per minute.CO2 carbon-dioxide.DWT discrete wavelet transform.ECG electrocardiogram.FFT Fast Fourier Transform.FOV field of view.HR heart rate.ICA Independent Component Analysis.ICC intraclass correlation coefficient.IP Impedance Pneumography.IR Infrared.IRB Institutional review board.LOA limit of agreements.MOD Mean of the Differences.NICU Neonatal Intensive Care Unit.PCA Principal Component Analysis.xiiList of AbbreviationsPPG Photoplethysmogram.RDS Respiratory Distress Syndrome.RGB Red Green Blue.RMSD Root Mean Square Deviation.RMSE Root Mean Squared Error.ROI Region of Interest.RR Respiratory Rate.RV Respiratory Volume.SNR Signal to Noise Ratio.TV Tidal Volume.xiiiAcknowledgementsI would not have been able to lead this multi-year cross-department workwithout the constant support and assistance of many individuals, the re-search team, and the department.I want to thank my research supervisor, Dr. Guy Dumont, for this once-a-lifetime opportunity to join his very talented research team. His invaluablesupport and guidance have made this research possible. My deepest grat-itude to him, as this thesis would have remained a thought without hisguidance, mentorship and patience. I am extremely proud and grateful formy years working with him. Professor Guy’s motivation, vision, integrity,and generosity are things I will take to my future career.Soodeh’s immense knowledge and support have been beyond helpful.Her technical feedback was instrumental in presenting my work. She guidedme at every research step and was open to insightful discussions. The brain-storming sessions and guidance Dr. Liisa and Dr. Lavoie provided pro-foundly shaped my research focus during my master’s degree. The researchteam’s unwavering belief in my abilities and the academic freedom theygranted me are aspects of my education that I will always hold dear. I amgrateful for the opportunity to explore my ideas and directions within myprojects. As a graduate student, I could not have asked for better supervi-sion.I could not have been successful in this journey without the unwaveringsupport of my family and friends. This journey has taught me a lot, includ-ing keeping loved ones close and believing in myself; anything is possible,being the biggest one.xivDedicated to Shakti Dutt Kondal, Sudesh Sharma, Shailja Verma, SaryuSharma, Anubhav Sharma, Kabir Sharma, Vir Sharma, Aprajita Arora andRohith Achanta, who have been my source of inspiration and strength.xvChapter 1IntroductionMeasuring vital signs is crucial in healthcare as they provide essential in-formation about a person’s physiological state. They are a set of objectivemeasurements that reflect the basic functions necessary for life. Vital signssuch as body temperature, blood oxygen levels, and respiratory rate (RR)help clinicians assess acutely ill patients and predict the need for immediateclinical intervention [17]. Changes in vital signs can sometimes indicate earlyhealth problems. Monitoring these changes can aid in the early detectionof potential issues, enabling timely care. For example, the normal RR canvary according to the age. While the normal adult’s RR can range between12 to 20 breaths per minute (bpm), the RR of pre-term infants is generallybetween 40 to 60 bpm [59], but it could go as high as 140 bpm for ill infants.In comparison to other age groups, an ill infant’s health can deterioraterapidly and can result in high mortality in extreme cases. Abnormal RRvalues in infants can be related to high levels of carbon dioxide in the blood,low levels of oxygen in the blood or high levels of acidity in the blood. Theimportance and need for monitoring respiratory rates in clinical practice areundeniable. However, uncertainties remain regarding the optimal methodsfor monitoring and interpreting these rates, as well as the ideal frequency ofmeasurement [17].1.1 MotivationThe initial months following the birth are critical for the infant’s survival.Respiratory diseases are one major cause of life-threatening illness in infants,specifically preterm neonates, since they are not fully developed. They oftenrequire specialized Neonatal Intensive Care Unit (NICU) care immediatelyafter birth [46]. Early diagnosis and assessment of respiratory dysfunctionhelp with required medical treatment, and monitoring the respiratory rateof infants can save their lives. [60].[72] discussed two methods based on a wearable sensor predominantlyused for RR measurement. The first method measures the changes to chestand abdomen movement using a change in impedance. A small electrical11.1. Motivationsignal is passed through adhesive skin electrodes, which then measures thechanges in the impedance due to breathing. Removal and placement ofthese adhesive skin sensors(electrodes) can cause discomfort and pain. Thesecond method derives the breathing rate from the Photoplethysmogram(PPG) signal obtained from a pulse oximeter exploiting respiratory sinusarrhythmia.1.1.1 The need for Non-Contact MethodsNeonates, especially those delivered before 29 weeks of gestation, have veryfragile and sensitive skin. The attachment between the sensor and the der-mis might be stronger than the bond between the dermis and epidermis.Therefore, attaching sensors may damage the skin and increase the risk ofdeveloping infection.[97] Adhesive sensors also present additional challenges,such as sensor dislocation, motion artifacts, false connections, and calibra-tion shifts.[60] Motion artifacts due to sensor dislocation pose a significantchallenge in continuously detecting respiratory rate in neonates. Addition-ally, irregular or shallow breathing patterns can impact chest impedancereadings. This variability may make it challenging to accurately distinguishnormal breathing from apnea events. Hence, a contactless approach wouldaddress challenges in monitoring neonatal respiratory rate and be suitablefor use in intensive care units and home environments.1.1.2 Clinical Use CasesTimely detecting respiratory issues through monitoring respiratory rate canhelp prevent complications such as respiratory failure, sepsis, or long-termrespiratory conditions. A few clinical use cases are discussed below.1.1.2.1 SepsisAccording to the World Sepsis report [80], every year, approximately 49million people globally are impacted by sepsis, with over 20 million beingchildren under the age of 5 and almost 5 million being older children andadolescents aged 5-19. The risk of mortality due to sepsis increases by 4-9%for every hour that treatment is delayed. The changes in respiratory rate canbe one of the early signs of sepsis. Sepsis often leads to systemic inflamma-tion and an immune response, affecting various organs, including the lungs.An elevated respiratory rate may indicate an increased demand for oxygen,reflecting the body’s attempt to compensate for the metabolic changes as-21.2. Research Question and Contributionsociated with sepsis. Early diagnosis and treatment could potentially saveas many as 80% of patients experiencing septic shock.1.1.2.2 ApneaPremature infants are particularly prone to apnea, a pause in breathing last-ing more than 20 seconds. Monitoring respiratory rate helps in the earlydetection of apnea episodes, allowing healthcare professionals to initiate in-terventions to support and stabilize the infant’s breathing.[67]1.1.2.3 Respiratory Distress Syndrome (RDS)The RR is a crucial parameter in the early detection of Respiratory DistressSyndrome (RDS). RDS is characterized by inadequate surfactant produc-tion in the premature infant’s lungs, leading to difficulties in breathing andpotentially respiratory distress. Infants with RDS typically exhibit signsof respiratory distress shortly after birth. Symptoms may include rapid orshallow breathing, retractions (visible pulling in of the chest muscles duringbreathing), grunting sounds, and a bluish tint to the skin (cyanosis) dueto insufficient oxygen. [67] An increased or abnormal respiratory rate andother clinical signs are early warning signs of respiratory distress, promptinghealthcare providers to take swift action to support the infant’s respiratoryfunction and overall well-being.1.2 Research Question and ContributionThe thesis presents a feasibility study investigating the non-contact method-ology for estimating RR of infants in NICU. The primary research questionsthat we intend to address are summed up as follows:• Can a depth-sensing camera estimate respiratory rates, particularlyin challenging scenarios replicating real-life conditions, like an infantmoving under covers or swaddled and in low-light settings?• Could a depth-sensing camera outperform an RGB camera in estimat-ing infant’s respiratory rates in the NICU? The advantage of depth-sensing cameras lies in their ability to mitigate biases related to skinpigment, which have been observed with RGB cameras. This potentialadvancement could revolutionize non-contact vital sign monitoring.• How significant is data quality when estimating infant’s respiratoryrates using non-contact methods?31.2. Research Question and Contribution• What is the importance of defining a Region of Interest (ROI)?• Why is a gold standard reference essential when assessing an algo-rithm’s performance? Can we rely on manually counted reference val-ues provided by nurse practitioners? What are the limitations associ-ated with this approach?• Is it feasible to utilize the wavelet transform technique for signal de-noising, and what advantages does it provide? Why is the peak detec-tion algorithm crucial for estimating infants’ respiratory rates in theNICU?Our primary contribution is demonstrating the effective usage of a depth-sensing camera to estimate RR. We also introduce the usability of wavelettransforms and adaptive peak-prominence threshold values while estimatingthe respiratory rate of infants. Additionally, we present the following pivotalcontributions to investigating non-contact methods.• We captured videos based on parental consent, therefore our datasetdoes not only include videos of sleeping infants but also instances ofsuboptimal conditions( i.e. crying baby, presence of clinicians hand,limbs or head movement). It is common in the literature to excludesuboptimal videos from databases due to motion artifacts causing in-accuracies in respiratory rate estimation.• The sample size of our testing dataset is 70 randomly selected videos.Typically, only 7-9 videos are analyzed in the literature.• Our proposed method is reliable and effective across different lightingscenarios.• We address the possibility of heart rate (HR) extraction while estimat-ing the respiratory rate (RR) by defining a minimum peak distancewhile applying a peak detection algorithm.• Our method eliminates the effect of skin pigments on the accuracy ofmeasurement.• Our method allows users to choose the region of interest they wish touse for calculating the respiratory rate.41.3. Thesis Outline1.3 Thesis OutlineThe structure of the thesis is as follows:Chapter 2 Deep - dive into the RR monitoring Methods : ContactMethodology and Non-Contact MethodologyChapter 3• Related work in monitoring RR of infants on NICU.• We also explore the limitations of the current method-ologies and discuss opportunities.• Introduction to the dataset and the importance ofdefining ROI.• We then dive into Respiratory Volume Analysis andour proposed method. We also explore the denoisingmethods to remove motion artifacts.Chapter 4 We discuss the experimental evaluations of the proposedmethod using the following metrics.• Bias and Root Mean Standard Deviation• Bland-Altman Analysis• Pearson Correlation• Intra Class CorrelationChapter 5 We end the thesis by summarizing our work and the chal-lenges faced.Chapter 6 Discuss potential future directions and concluding remarksTable 1.1: Overview of Structure of the Thesis5Chapter 2Related WorksThe necessity for monitoring changes in respiratory rate has led to extensiveresearch in this domain. This chapter discusses the current methodologiesfor measuring RR, providing an overview of contact and non-contact meth-ods and outlining their existing challenges.2.1 Contact Based Respiratory Rate MonitoringThe conventional methods utilized for measuring RR involve contact-basedtechniques requiring a sensor attached to the body. These methods typicallymeasure parameters like the sounds related to respiration, the motion of thetorso, blood oxygen saturation, respiratory carbon-dioxide (CO2) emission,and electrocardiogram (ECG).2.1.1 Impedance PneumographyTraditional monitoring devices measure RR from the Impedance Pneumog-raphy (IP) waveform and can depict variations in the patient’s chest elec-trical impedance via an ECG electrode [97]. However, disposable adhesivesensors have some disadvantages, including their maintenance cost and thepotential to cause skin irritation and damage, especially in preterm or ill in-fants since they have fragile skin [72]. Moreover, false monitor alarms fromthe IP method related to the patient moving and not breathing momentar-ily can disturb the infant and the staff. Another challenge that needs to beconsidered is the infant’s chest size, which makes it very difficult to place asensor(electrode).2.1.2 Acoustic MonitoringAcoustic-based approaches utilize a microphone positioned near the airwaysto capture respiratory sounds, and by analyzing their variations through fre-quency analysis, RR can be estimated. In study [68], using spectral analysis,breath sounds were isolated from an acoustic signal affected by background62.1. Contact Based Respiratory Rate Monitoringnoise. Researchers observed that external noise exhibits a distinct distribu-tion of frequency that enables the differentiation of breath sounds throughappropriate filtering within the relevant range. However, it is noteworthythat previous studies, such as [50], [38], reported that extraneous noises likesnoring, speaking, crying, and coughing exerted a detrimental impact on theefficacy of this methodology.2.1.3 Airflow MonitoringMeasuring RR can be achieved by monitoring respiratory airflow, leveragingthe fact that exhaled air is warm and humid [30]. A thermistor that canbe nasal or oral can detect changes in the temperature of breath-in air andbreath-out air, thus helping estimate airflow and further the respiratoryrate. However, the thermistor’s placement can influence the methodology’saccuracy, as highlighted in the study by [89]. Additionally, a few patientsmay find the sensor placement uncomfortable, especially over the mouthand/or upper lip area.2.1.4 Carbon dioxide MonitoringThe CO2 monitoring technique assesses general variations in CO2 levels.The process is carried out by placing a heated electrode on the skin andstudying diffusion. A major drawback of this methodology is that it couldburn an infant’s delicate skin. In an alternative approach, Folke et al. [31]suggest a CO2 sensor designed for measuring RR, but it leads to significantvariations in sensor performance. [50]2.1.5 PhotoplethysmogramPPG-based measurement is a cost-effective and contactless method that canestimate RR via sensing the blood volume pulse.The signal is recorded using a pulse oximeter and derives from the lightvariations reflected from the tissues as mentioned in the work by [12]. Theinhalation and exhalation influence the ”pulse wave” as discussed in [55].Respiration impacts the amplitude and frequency of RR pulses, whichthen helps us in the estimation of respiratory rate using various parameterssuch as the width of the pulse, heart rate, and changes in amplitude ofthe PPG signal [56]. In [51], wavelet transforms were used to identify therespiratory signal of ten healthy adults using PPG signals. Additionally,[102] identified the peaks and calculated the respiratory rate of the infant’sPPG signals.72.2. Non-Contact Based Respiratory Rate MonitoringLazaro et al. [56] utilized a smartphone (iPhone 4s) to record a PPGsignal. The participants placed their fingers on the flashlight and switchedit on. The researchers selected a region of interest and integrated it overa period of time to get the respiratory rate. However, it was observedthat this method was ineffective for dark-skinned infants and resulted in anoverestimation of respiratory rate.2.1.6 ElectrocardiogramInhaling and exhaling can influence the ECG signal, and hence, we canutilize this property to estimate the respiratory rate, as mentioned in [7].The body surface ECG is impacted by breathing-related electrode move-ments, causing fluctuations in thoracic electrical impedance while inhalingand exhaling. Consequently, variations in the mean cardiac electrical axisare linked to respiration, enabling RR estimation. [61] applied PrincipalComponent Analysis (PCA) to anaylse the ECG leads that contribute thetowards change in respiratory rate. However, [6] highlighted that the infant’smovements could introduce errors while estimating RR.2.2 Non-Contact Based Respiratory RateMonitoring2.2.1 LaserEarly studies proposed the usage of a laser diode and calculated the dis-tance between the torso and the sensor to estimate the RR. They capturedrespiratory waveforms by plotting the variations in this distance over timeto estimate RR. In a different approach, [14] utilized an ultrasonic proxim-ity sensor to track chest-wall movement. [50] further studied this method’saccuracy to measure RR and reported that it could be influenced by thetexture of the subject’s clothing.2.2.2 Thermal CameraThermal imaging cameras present novel possibilities for implementing non-contact methods to measure vital signs. Exhaled air typically exhibits ahigher temperature compared to inhaled air.Yang et al. [74] employed a thermal camera to monitor periodic tem-perature changes near superficial blood vessels and the nasal area, enabling82.2. Non-Contact Based Respiratory Rate Monitoringthe measurement of HR and RR through infrared light modulation. [65] em-ployed a cooled mid-wave infrared Phoenix camera with a thermal sensitivityof 0.025°C to capture videos and utilized statistical methods to measure RRwithout physical contact with the subjects. Their experiments were con-ducted in a dimly lit room to minimize reflection issues. The researcherschose a region approximately 6-8 feet from the tip of the nose as the ROI toobserve temperature variations during inhalation and exhalation. Throughan adaptive thresholding algorithm, their approach distinguished facial skinfrom non-skin areas in the subject’s profile view. Their methodology com-prised two phases: training and testing. During training, the system learnedthe patient’s breathing temperature patterns, calculating parameters suchas Tmin, Tmax, and the average of Tmin and Tmax. They defined temper-ature distributions for inhalation (between Tmin and Tave) and exhalation(within Tave and Tmax), assuming a normal distribution. The mean andvariance of these distributions were calculated across multiple breathing cy-cles. During the testing phase, the system compared incoming temperaturedistributions with those from the training phase. If the difference was lowerfor inhalation, the pixels in the ROI were classified as inspiration; otherwise,they were classified as expiration. This process updated parameters (weight,mean, and variance) for the inspiration and expiration distributions. Upondetecting a complete breathing cycle, RR was estimated using timestampsfrom all frames in the cycle. The variation in the room temperature couldresult in incorrect values. The subjects were limited in their natural move-ments for the experiments of both studies, [65] and [93].2.2.3 RadarChen et al. [21] employed a Doppler radar system to assess both HR andRR. This system exposes the participant to a low-intensity microwave signalreflected in the radar’s sensor. As respiration induces movement in the upperbody, it influences the signal, enabling the estimation of RR. On the otherhand, Uenoyama et al. [93] used a 1,215 MHz-microwave radar antenna boxwith a maximum output power of 70 mW and two micro stripe antennas forradiating and receiving to measure the RR of sleeping subjects. The radarsystem was positioned 200 cm away from clothed, healthy subjects lying inbed. The researchers identified the chest as the region of interest (ROI) tomonitor upper body movements. They calculated RR by analyzing the FastFourier Transform (FFT) of chest wall motions captured by the radar, withthe peak of the FFT signal indicating RR.92.2. Non-Contact Based Respiratory Rate Monitoring2.2.4 RGB CameraWe can measure the respiratory rate using a Red Green Blue (RGB) cameraby either monitoring the movement of the upper body or extracting therespiratory volume signal.[53] and [92] utilized a camera to capture alterations in movements dur-ing respiration to estimate respiratory rate (RR). In Lin’s study, [53], theydetected the face to determine the relative upper body position and mod-elled upper body motion caused by respiration based on vertical variationsin optical flow, subsequently calculating RR. On the other hand, Tran et al.[92] localized the chest and abdomen using body segmentation and calcu-lated the average breathing signal by mapping the optical flows of the upperbody. [108] studied the changes in imaging due to respiration movement tomeasure the respiratory rate. They utilized a near-infrared camera whichwas sensitive to both visible and near-infrared light, They based the researchon changes in reflected light correlated with respiratory events, facilitatingRR estimation.Shao et al. [85] utilized a method to estimate RR by monitoring the edgeof the shoulder and employed an algorithm that tracked motion. However,it was recorded that the motion artifacts led to low accuracy.Al-Naji et al. [8] used the wavelet decomposition technique followed byan elliptic filter to magnify the subjects’ motion. To detect the motion andestimate the respiratory rate, they also implemented a subtraction of theframes.Wei et al. [100] calculated two regions of interest: the subject’s face andneck. They leveraged a kurtosis-based approach to select the signal thatcontained information on all the decomposed signals.Jorge et al. [43] proposed a supervised learning for skin segmentation.They implemented Independent Component Analysis (ICA) over the visibleskin region to extract the RR. Additionally,[74] demonstrated the feasibilityof calculating the RR based on the movement of the nostrils, essentially bymeasuring the position of the nose.Wu et al.[104] amplified the motion of the subject in the videos to es-timate the respiration. They faced challenges in differentiating betweenrespiratory-induced motion and other unrelated movements.Utilizing the principle that the blood can absorb more light comparedto the tissues surrounding it, [44] studied the effects of change in blood vol-ume on the light reflected from the skin and obtained PPG. Furthermore,previous literature has suggested and proven the feasibility of using a RGBcamera for rPPG monitoring [22], [95]. Karlen et al. [44] averaged inde-102.2. Non-Contact Based Respiratory Rate Monitoringpendent respiratory rate values to calculate the respiratory rate from 14PPG. Additionally, [94] used a combination of normalized colour channelsto extract the respiratory rate. The algorithm detected the face and furtherdivided it into sub-regions. They then calculated the Signal to Noise Ratio(SNR) for each sub-region, selecting the one with the highest SNR to esti-mate RR. However, it was noted that the methodology is noise-sensitive [95].A Hue-based methodology was proposed in Sanyal and Nundy [81] wherethe time-dependent fluctuation of hue was averaged to calculate pulse and,eventually, respiratory rate. Soodeh Ahani proposed a methodology that isbased on RGB camera videos. She utilized the Eulerian video magnificationalgorithm and also included suboptimal instances. Our work is inspired byher findings. Our proposed algorithm outperforms her methodology by 26%.The work has not been published yet.2.2.5 Depth CameraDepth cameras have exhibited remarkable effectiveness in capturing the mo-tion patterns associated with respiration. Respiratory Volume (RV) signalcan be calculated, which helps measure both respiratory rate and tidal vol-ume. Additionally, combining RR and Tidal Volume (TV), and its non-contact methodology, could be very helpful during pandemics, such as covid[3]. Depth sensing respiratory rate studies majorly fall into two reasonablecategories:• Laboratory Research• Clinical healthy volunteers and patient studiesThese are discussed as follows:2.2.5.1 Respiratory Rate Benchtop/Lab studiesThe studies generally comprise ad hoc investigations, which are carried outto understand the technology’s fundamental aspects or provide limited proofof principle.A study conducted by [15] focused on healthy volunteers. The ideal sam-pling frequency was chosen based on various experiments. Furthermore, thealgorithm’s robustness was evaluated by performing the study in differentlight conditions, such as room lighting, clothes texture and the subject’sposition. A high Pearson Correlation was reported between the referencespirometer value and Depth Camera values.112.2. Non-Contact Based Respiratory Rate Monitoring[23] conducted a study using a Prime Sense camera on six sleeping par-ticipants. Visual counting was referred to as the gold reference for evaluatingthe methodology. The volunteers breathed within 1-3 min, ranging from 14to 25 breaths/min. However, they did not perform any statistical analy-ses, but a difference of 1 breath/min was reported between the estimatedrespiratory rate from the depth camera and reference RR.Lin et al. [53] studied 41 videos from 9 subjects. They utilized theregion of interest capability in estimating the respiratory rate. A correlationcoefficient of 0.98 was reported when compared against the gold reference.A bias of 0.82 breaths/min was noted. This novel study introduced theconcept of more than one region of interest.2.2.5.2 Clinical StudiesClinical studies are usually conducted on a wider scale using various instru-ments and technology. They often require Institutional review board (IRB)approval and comprise many participants.[3]Yu et al.[106] conducted two related experiments to study the feasibilityand accuracy of depth cameras for continuous monitoring. During the firstexperiment, eight participants were required to change their sleeping posi-tion every 15 breathing cycles to study the effect of their sleeping positionon the depth camera. 86.3% accuracy was reported against the referenceRespiratory Inductance Plethysmography bands. A single participant wasmonitored during the second study. Forty two hours of sleep data werecollected in 10 days, with 5 days when the participant was covered in ablanket and the other 5 days when the blanket was removed. 92% accuracywas reported, highlighting that the average accuracy for detecting the torsoand the head in the depth video was between 96.4% and 98.4%. A sleeplab study involving 67 healthy patients and 3239 segments of data was per-formed by Martinez and Stiefelhagen [57]. Features such as the thickness ofthe blanket, size of the cushion and visible items were studied. An accuracyof 88.7% was reported. The reference in the study was a thermistor placedunder the nose.Seppanen et al.[84] monitored various breathing patterns of eight healthysubjects. 2.5 cm wide regions of interest were selected, and the value calcu-lated from the spirometer was considered a gold reference.Medtronics[1] utilized a Kinect camera to estimate respiratory rate. ARoot Mean Square Deviation (RMSD) and bias of 0.66 breaths/min and0.04, respectively, were found. The study reported a high correlation be-tween the RR calculated using the depth camera and the reference recorded122.2. Non-Contact Based Respiratory Rate Monitoringusing a capnograph. The studies conducted on the younger, non-adult pop-ulation will be discussed in Chapter 4.Considering the comprehensive body of evidence, the clinical studies de-scribed in this section, which involve healthy volunteers and patients, clearlydemonstrate the potential of using depth-sensing equipment to monitor res-piratory rates.2.2.6 Comparison StudyThis section compares camera-based techniques for determining neonatalrespiratory rates (RR). Categorization is performed based on the methodol-ogy and sample size. Furthermore, their limitations and clinical applicationsare presented [59]. In the table below, ”PI” denotes preterm infants, ”N”represents neonates, and ”L” signifies the location.It’s worth mentioning that the majority of the studies [97], [48], [11],[41] reviewed here have a sample size of 2-5 infants. This trend has beencommonly observed in previous literature. Additionally, many of the studiesare focused solely on healthy neonates. In contrast, our study comprises 13infants, with a testing dataset containing 70 randomly selected videos.Moreover, the studies [91], [54], [13], [28] hare light dependent and high-light a limitation regarding susceptibility to inaccuracies caused by externalnoise, motion artifacts, and variations in lighting conditions. Our method ef-fectively tackles and overcomes these challenges by employing wavelet trans-forms and an adaptive peak prominence threshold. We want to highlightthat our processing time per video is less than 2 seconds.132.2. Non-Contact Based Respiratory Rate MonitoringCamera-based technologies for Neonatal RR measurementAuthor Sample Size(PI/N, L)Method Application LimitationVillarroelet al.[98](2, NICU) ReflectancePPG-basedCardio-respiratorymonitoring.The higher error be-tween BR from PPGand reference.Koolentet al.[48](2, NICU) ReflectancePPG-basedDetection ofapnea in Home-based settings.The higher time re-quired for computationAlinoviet al.[11]Neonates = 2 Motion-Based Sleep studies The study was limitedto healthy neonates.Janssenet al.[41](2, NICU) Motion-based Cardiorespiratorymonitoring inneonates as wellas adultsNot MentionedSun etal. [91](5, NICU) Motion-Based RR Monitoring ofpremature infantsInefficient filtering notsufficient in case of mo-tion noise (caregivers’movement)Loratoet al.[54](5, NICU) Motion-based Apnea alert sys-tem along withRR monitoringEven after modifica-tion, the algorithmdidn’t turn out to benoise resilient.Antognoliet al.[13](40, NICU) Motion-based CardiorespiratoryMonitoringProne to inaccuracy byextraneous motions andLight dependentEastwoodet al.[28]Mannequin Motion-baseddepth sensingCardiorespiratoryMonitoringDepends on the lu-minance of backgroundconcerning the subjectGastel etal.[94](2, NICU) PPG-based RR monitoring The study was limitedto healthy neonates.Table 2.1: Comparison between Camera-based technologies for NeonatalRR measurement14Chapter 3Estimation of RespiratoryRate of Infants in NeonatalIntensive Care Unit3.1 AimNon-contact respiratory rate estimation is highly desirable for infants be-cause of their sensitive skin. This thesis aims to conduct a feasibility studythat will help us determine the viability and potential benefits of using depthcameras to estimate the respiratory rate of infants in the NICU, paving theway for future research and clinical implementation.3.2 Key Contributions• We captured videos based on parental consent. Therefore, our datasetincludes videos of sleeping infants and instances of suboptimal con-ditions( i.e., crying baby, presence of clinician hand, limbs or headmovement). It is common in the literature to exclude suboptimalvideos from databases due to motion artifacts causing inaccuracies inrespiratory rate estimation.• The sample size of our testing dataset is 70 randomly selected videos.Typically, only 7-9 videos are analyzed in the literature.• Our proposed method is reliable and effective across different lightingscenarios.• We address the possibility of HR extraction while estimating the respi-ratory rate (RR) by defining a minimum peak distance while applyinga peak detection algorithm.• Our method eliminates the effect of skin pigments on the accuracy ofmeasurement. Indeed, the infants belong to different ethnicities.153.3. Introduction and Methodology• Our method allows users to choose the region of interest they wish touse for calculating the respiratory rate.• We introduce the usability of wavelet transforms for removing motionartifacts and denoising the signal.• Our method applies an adaptive peak prominence threshold value forpeak detection.3.3 Introduction and MethodologyThe respiratory rate in newborns allows us to know the health and well-being of the infant. Classical monitoring methods can monitor the vitalsigns in adults, but having contact with newborns can cause injuries andinfections.We propose a non-contact approach for estimating infants’ respiratoryrates in the Neonatal Intensive Care Unit (NICU). The method involves ex-tracting the respiratory volume signal by integrating it over a user-selectedRegion of Interest (ROI). Subsequently, Wavelet Transforms are employedto denoise the data and remove motion artifacts. We also introduce a min-imum peak distance to prevent heart rate extraction or other erroneousvalues. Furthermore, an adaptive peak prominence technique is appliedto mitigate motion artifacts and enhance the algorithm’s robustness acrossvarious lighting conditions. The process flow is described in Figure 3.1.Figure 3.1: Block Diagram of the proposed Depth-video-based RR estima-tion method.163.4. Related work3.4 Related workQuite a few research groups have concentrated on a younger, non-adult pop-ulation. Cenci et al. [19] monitored the respiratory rates of three neonatesfor five 30-second intervals each. They calculated the mean breath periodand reported excellent agreement with the gold reference. A study thatmonitored the respiratory rate of a 4-month-old and 1-year-old infant in theNICU [79] demonstrated that the depth camera can calculate the respiratoryrate. They collected five one-minute videos and utilized the ventilator valueas the reference. A RMSDs of 0.77 and 0.68 breaths/min were reported.Al-Naji et al.[9] reported agreement between the respiratory rate estimatedusing the depth camera and the reference values calculated using a piezobelt while studying five children between the ages of 1 and 5 years old. Thelimits of the agreement for the study ranged from [-0.91 to +1.0] to [-1.3 to+2.3] breaths/min for cases with and without the blanket.3.4.1 Limitations and OpportunitiesIn the previous chapters, we have briefly discussed the implications of exist-ing methods developed to estimate an infant’s respiratory rate in the NICU.The respiratory rate of infants in the NICU is monitored by bedsidemonitors. The ECG is the reference since it provides stable and robust mon-itoring for a NICU. However, the pressure applied by the contact sensor canalter local skin perfusion, leading to inaccurate measurements compared tonon-contact sensors. The electrodes put pressure on the skin, which resultsin tissue compression and could lead to vascular insufficiency. Prolongedapplication of ECG electrodes on infant skin could result in skin infectionor trauma. Furthermore, removing the adhesives from the delicate skinof preterm infants during routine care procedures can inflict damage andinduce stress and pain [91]. Moreover, the current methodology proves im-practical for home care due to the requirement for skilled caregivers to placethe sensors and the resultant inconvenience posed to the daily lives of theindividuals involved.When creating a NICU tool, it is crucial to consider clinical settings andensure our algorithm is robust and suitable for hospital use. We take a steptoward this goal by capturing videos based on parental consent. Therefore,our dataset includes videos of sleeping infants and instances of suboptimalcon- ditions( i.e., crying baby, presence of clinician hand, limbs or headmovement). It is common in the literature to exclude suboptimal videosfrom databases due to motion artifacts causing inaccuracies in respiratory173.5. Overview of Datasetrate estimation. We aim to develop a reliable and robust methodology thatis effective under different lighting conditions. This is achieved by applyingan adaptive peak prominence algorithm and using wavelet transforms todenoise the data. These techniques help eliminate motion artifacts fromroutine baby care, examinations, interventions, or limb movements.While video-based RR estimation methods for infants in the NICU rep-resent a promising start, several limitations still need to be addressed. Manyproposed methods have been tested under ideal conditions, such as well-litenvironments and visible skin. One critical factor impacting RR estimationmethods is the level of environmental lighting, which restricts the effective-ness of non-contact vital sign estimation methods. Although using a farinfrared (FIR) camera can address some of these challenges, these camerasare expensive and restrict the algorithm’s adaptability to more accessiblehandheld devices. [67]Furthermore, infants are frequently swaddled or wrapped in a blanket,making it challenging to monitor changes in their chest and abdomen, espe-cially during shallow breathing. This poses difficulties for prominent non-contact methods utilizing RGB cameras, as the lack of visible skin hamperstheir performance.Additionally, skin colour influences accuracy, leading to SpO2 overesti-mation. Studies have shown that video PPG, reliant on skin tone, facessimilar challenges in SpO2 estimation. [5]. While the RGB-camera-basedmethod shows promise in detecting chest movements in RGB videos, it ishindered by darker skin tones. Depth cameras offer a solution to this issue!Most studies in the current literature primarily concentrate on healthyadult subjects. However, the initial months of life are crucial for humansurvival, emphasizing the necessity for developing non-contact algorithmscapable of estimating respiratory rate.3.5 Overview of DatasetThe data was acquired for an observational study conducted between March2019 and November 2019 in the Neonatal Intensive Care Unit at the BCWomen’s Hospital in Vancouver, British Columbia, Canada.Informed consent was obtained from every infant’s parents or LegalGuardians. The study was approved by the University of British ColumbiaChildren’s & Women’s Research Ethics Board (H18-02406). The local ethicscommittee also approved the data collection set-up. The study included 13infants with a gestational age between 24 to 39 weeks at birth. The infant’s183.5. Overview of Datasetpostnatal age varied from 7 to 116 days, and their weight at the time thevideo was captured ranged from 1950 to 3370 grams.Figure 3.2: Frame samples captured using RGB Camera.3.5.1 Introduction to Depth CameraThe technology of depth cameras has great potential for non-contact res-piratory rate monitoring, identifying breathing patterns, studying motionactivity, calculating tidal volume, detecting sleep apnea, etc. Moreover, non-contact monitoring introduces the advantages of reduced patient restriction,avoidance of single-use consumables, and mitigated risks of disease transmis-sion. It works on a simple principle where the camera measures the distanceto the surface of the objects in its field of view (FOV) and then returnsa single matrix, which can also translate as depth for every frame. Theconcept varies from that of a RGB camera, as those generate three matrics:red, green and blue. Figure 3.2 and Figure 3.3 display each example. Wecan visualize the final depth information by converting the depth values intoa false-colour image as shown in Figure 3.4. The resolution of the IntelRealSense D415 camera is 640 x 360 with a frame rate of 30 frames per second.The depth-sensing cameras can be based on one of three principles:stereoscopic, structured light, or time-of-flight [3]. Each of these principleshas advantages and is suited to different applications based on accuracy,range, and cost. Based on a stereoscopic system, the depth-sensing cam-era generates barely two different perspectives with the same view. Depthis then measured by calculating the differences between the features of theview. Based on the time-of-flight principle, the camera measures distance193.5. Overview of DatasetFigure 3.3: Frame samples captured using Depth Cameraby calculating the time the signal returns after reflecting. They do generallyalso have an Infrared (IR) laser. In a light-based camera system, a decidedpattern is projected. The depth is then calculated by studying the changesin the pattern.As mentioned, depth maps are often converted into colour maps to facil-itate easier human interpretation. Different colours are assigned to differentdistances, such as cyan for closer objects and red for farther ones. It’s im-portant to note that the specific colour mapping used for depth maps isarbitrary and only serves the purpose of visualization.When focusing on an infant’s torso, the variation in the distance causedby the chest expanding and contracting during breathing results in changesin pixel values in the depth videos. These changes can then be utilized toderive a respiratory volume (RV) signal, as discussed in Addison et al.’swork [2] on continuous respiratory monitoring in infants.For our study, the videos were recorded using Intel Real Sense D415,which uses stereovision to calculate depth. The equipment was mounted ona gantry-like tripod approximately 1 m away from the baby to minimizedisturbance to the regular infant’s care. The Neonatal Intensive Care Unitsetup is shown in Figure 3.6.203.5. Overview of DatasetFigure 3.4: Frame samples captured using Depth Camera, converted usingfalse colours3.5.2 Data Collection           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.5: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.We captured videos of nine infants in natural daylight conditions. How-ever, three were recorded in dimly lit environments in the evening withclosed curtains. Figure 3.7 represent those captured in two different lightingconditions in a lit environment and a low-lit environment, respectively. Fourinfants were recorded at bedtime as shown in 3.1. It’s important to highlightthat the videos were recorded according to the parent’s consent. Therefore,our dataset includes videos of sleeping infants and instances of suboptimalconditions( i.e., crying baby, presence of clinician hand, limbs or head move-ment). It is common in the literature to exclude suboptimal videos fromdatabases due to motion artifacts causing inaccuracies in respiratory rate213.5. Overview of DatasetData DemographicsStudyIDGestationalAgeAge(days)Sex RespiratorySupportSelf-declaredEthinicityTime of theDay001 01 39,1 14 M N N/A Day Time002 01 31 22 M N N/A Dark thenLight003 01 33 7 M N European,ColumbianDay time004 01 26,6 72 F N/A N/A Day Time005 01 24,2 90 M Y N/A Day Time006 01 35,3 29 M N/A Fillipino,Finn Day Time007 01 35,4 40 F N Caucasian N/A008 01 36 17 M N Caucasian,CanadianBed Time009 01 35,1 18 F N Caucasian Day Time0010 01 35 43 F N N/A Day Time0011 01 24,5 116 F N Caucasian Bed Time0012 01 31,4 30 F N Asian, Fil-ipinoDay Time0013 01 39,3 30 F N N/A Bed TimeTable 3.1: Overview of Neonatal Data Demographicsestimation. Each infant’s recording spanned up to four consecutive hours.In one instance, the recording extended over two days, with four consecu-tive hours each day due to corrupted bedside monitor information on thefirst day. Additionally, one infant was receiving CPAP treatment during thevideo recording.In addition to video monitoring, we concurrently collected vital signs us-ing a standard bedside patient monitor (Phillips IntelliVue MX800, Philips,Amsterdam, Netherlands). [17] Electronic data from the bedside monitorwere acquired through a laptop equipped with the Vital Recorder data ac-quisition program. The Vital Recorder, a free program, facilitates capturingtime-synchronized data from various devices, enabling integrated analysis ofvital signs data. A nurse practitioner visually annotated the captured NICUdata.223.5. Overview of DatasetFigure 3.6: Neonatal Intensive Care Unit set up3.5.3 Challenges and LimitationsOur main objective of this study is to design and develop a reliable andreproducible methodology. To achieve this, we maintain high data quality.This helps us draw credible conclusions and validate the evaluation met-rics performed. Furthermore, good data quality promotes reproducibility,allowing peers to replicate our work and verify the results.Our methodology overcomes the challenge and effectively estimates therespiratory rate, aiding clinicians in determining the necessity of medicalintervention. In the future, we plan to annotate more data manually tobroaden and enhance the scope of our study.3.5.3.1 Data interpretation and transferThe data collection process was a collaborative effort with Medtronic. AnIntel RealSense D415 depth-sensing camera was utilized to capture high-resolution videos of infants in the Neonatal Intensive Care Unit. Theserecordings were stored as binary files. This methodology demonstrates aneffective data compression ability and the capability to handle the extendedduration of the videos, which, in our case, lasted up to four hours. However,233.5. Overview of DatasetFigure 3.7: Representative frame samples captured in two different lightingconditionswhile performing the experiments, we encountered a limitation related tostoring high-resolution files as binary data.• The risk of data corruption is significant while using binary formatsince even a single bit of corruption can render the entire file un-usable. Data transfer issues corrupted the data for Infant 1, Infant5, and Infant 13 during transmission between the research team andMedtronic.• The interpretation of data can be influenced by the byte order in whichthe data is stored, also known as endianness. Ensuring data qualitywas our primary objective, especially while developing algorithms forclinical settings. When reading and processing the binary files, weaimed to handle endianness and any potential compatibility issues ef-fectively. To address this challenge, we implemented appropriate con-version methods to transform the binary data into a compatible formatfor analysis and integration with other tools. Specifically, we convertedthe data into HDF5 format, which offered enhanced compatibility andstreamlined data processing and analysis.3.5.3.2 Gold Reference ValuesEstablishing reference values is crucial during algorithm evaluation. Theyserve as benchmarks, offering insights into the proposed algorithm’s accuracy243.5. Overview of Datasetand reliability. We set up a bedside monitor (Phillips IntelliVue MX800,Philips, Amsterdam, Netherlands) to collect vital signs. The respiratory rate(RR) provided by the bedside monitor via IP sensors was the reference goldstandard RR value. The data from the monitor was acquired using a freeprogram, The Vital Recorder. Additionally, a nurse practitioner manuallycounted the respiratory rate by observing the RGB-camera-based videos.However, we decided to use the respiratory rate determined through vi-sual assessment by the nurse and the values estimated from our proposedRGB method suggested in Video-based Respiratory Rate Estimation for In-fants in the NICU as our reference gold standard, aligning with the approachtaken in [47]. Notable disparities between the RR values provided by visualassessment and those reported by the bedside monitor have been reportedin [98], [47], and [33], supporting our claim. However, manually countingrespiratory rate as a gold reference presented its challenges.The manual annotation process was conducted offline, where the nurseobserved the RGB videos to record the values. We omitted cases 3, 7,and 9 due to insufficient documentation in the manually annotated videos.Notably, RGB cameras have a frame rate of 65 per second, while depth datais captured at 30 per second. Furthermore, there is a 3-second lag betweenthe depth-sensing and RGB-based cameras. This discrepancy results inmissed frames and potentially missed breaths.3.5.4 Data Pre-processing and Visualization           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.8: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.HDF5 allows for organizing data in a structured manner, with sup-253.5. Overview of Datasetport for hierarchical groups and datasets. It is important to note thatdepth videos are high-dimensional. We adopted this structured approachto streamline the organization and management of intricate depth-cameradatasets. As we were dealing with a large HDF5 dataset, we opted to loadthe data in chunks rather than loading the entire dataset into memory atonce. This approach proved advantageous as it helped us manage memoryusage and enhance performance, especially since our analysis did not requiresimultaneous access to the entire dataset.Our strategy involved iterating the dataset in chunks based on the videolength we analyzed and loading specific frames using array slicing. Withineach iteration, we would analyze or process the chunk of data. For instance,in this thesis, we calculated the mean pixel value over a defined region ofinterest (ROI).In an attempt to visualize the videos, we encountered metadata andattributes occasionally while converting frames into videos. As a result, wenoticed blue frames that could potentially introduce erroneous values intorespiratory rate calculations, as shown in Figure 3.9a. We resolved thisissue by removing the attributes using a MATLAB command. This helpedus remove any outlier depth values, as well.Furthermore, during data visualization without applying thresholding,we observed that pixel intensity values ranged up to nearly 7000, with fewerthan 100 values falling between 3000 and 7000. Figure 3.9b shows the outputimage without thresholding. Thresholding depth data during visualizationcan help improve its interpretability, clarity, and usefulness by highlight-ing relevant information, removing noise, enhancing contrast, simplifyingvisualization, and facilitating interpretation. Therefore, we incorporatedthresholding specifically for visualization to tackle this issue. It’s essentialto emphasize that thresholding was solely utilized for visualization. As men-tioned, depth maps are often converted into colour maps to facilitate easierhuman interpretation. Different colours are assigned to different distances,such as cyan for closer objects and red for farther ones. It’s important tonote that the specific colour mapping used for depth maps is arbitrary andonly serves the purpose of visualization. We enhanced the contrast and visi-bility within the depth data through thresholding, effectively mapping depthvalues to an intensity range. This method allowed us to distinguish betweenvarious depth levels, making the visualization more informative and visuallyappealing. We want to highlight that we utilize the raw, high-dimensionaldepth data to estimate the respiratory rate.263.5. Overview of Dataset(a) Attribute in HDF5data format(b) High dimensional depthdataFigure 3.9: Challenges faced visualizing the depth data3.5.5 Introduction to Optimal and Suboptimal VideosWe captured videos based on parental consent. To achieve this, we placedour setup, as shown in Figure 3.6, such that the regular care of the infantis not interrupted. Therefore, our dataset includes videos of sleeping in-fants and instances of suboptimal conditions ( i.e., crying baby, presence ofclinician hand, limbs or head movement). It is common in the literature toexclude suboptimal videos from databases due to motion artifacts causinginaccuracies in respiratory rate estimation.Focusing on the data quality, we removed the segments in case the fol-lowing events were visually detected:• Routine care for the baby by parents or nurses, such as changing cloth-ing or adjusting bedding.• Medical examinations and interventions include assessing heartbeat,conducting auditory and visual tests, measuring temperature, and per-forming blood tests.• Infant removed from the crib or incubator for nurturing, includingskin-to-skin contact and feeding by the mother.273.5. Overview of Dataset           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.10: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.• The occurrence of significant motion artifacts, where the baby’s breathing-related movements are indistinguishable for annotators, such as duringcrying episodes.• The presence of external objects significantly covering the baby’s chest,head, or abdomen, such as the presence and movement of parents’ orclinicians’ hands or heads.We categorized periods not involving the above events as valid data basedon the methodology proposed by Villarroel et al. (2019) [97]. We randomlyselected multiple videos from each case, each lasting around one minute,with the number of videos per case ranging from 19 to 25, as explained indetail in Table 3.2.The videos with significant movements that made distinguishing betweenrespiratory-related-movement and motion artifacts difficult were excludedfrom the dataset. However, there were instances of acceptable motion ar-tifacts, defined as motions that do not hinder the annotator from visuallycounting the respiratory rate. It is common in the literature to excludesuch videos from databases due to motion artifacts causing inaccuracies inrespiratory rate estimation. [33]Hence, the video segments were classified into optimal and suboptimalbased on the baby’s movement. Optimal videos included short-durationsegments with minimal infant motion or slight caregiver shadow movement.On the other hand, suboptimal videos encompassed segments with upperor lower limb movement within permissible limits, the presence of parent’s283.5. Overview of Datasetor nurse’s hands that did not obstruct the breathing-related area, or videosshowing mild infant crying without causing high motion artifacts. Table 3.2presents the percentage distribution of videos analyzed within the subopti-mal set.We analysed a testing dataset comprising 70 short-duration videos ofsuboptimal and optimal instances, to evaluate our feasibility study, totallingan analyzed data duration of 155.9 minutes, nearly 2.5 hours. Table 3.2details specific information for each case, such as the number of randomlyselected videos, duration per case, percentage of videos in the suboptimalset, percentage recorded in low-lit environments, and percentage of videoswhere the infant is swaddled or covered with a blanket.Optimal and suboptimal categorization of videos was done by simul-taneously reviewing RGB and depth videos on a second-by-second basis.However, during the analysis of depth videos, we had to exclude cases 1,5, and 13 due to the unavailability of depth videos for these cases. Cases3, 7, and 9 were omitted due to insufficient documentation in the manuallyannotated videos.It is important to highlight that to evaluate the robustness of our algo-rithm, we included instances with various light conditions. For example, inTable 3.2, the fifth row indicates that cases 8, 9, 11, and 13 were recordedin a dark environment. In contrast, 26.3% of the videos from case 2 (i.e., 5videos out of a total of 19 videos) were recorded in a low-lit environment.Casenumber2 3 4 6 7 8 9 10 11 12No. ofvideos19 0 23 25 0 20 0 20 23 21Duration(min)20.7 24.7 24 25.2 26.4 20.3 25.3 19.9 23.5 22.3Suboptimalclass %51.7 17.7 34.6 48.6 16.1 34.8 33.4 19.7 56.3 51.7Low-lit % 26.3 0 0 0 28 100 100 0 100 0% covered 100 100 100 100 100 95 100 10 86.9 38.1Table 3.2: Specifications of the analyzed short videos of each case of thedatasetIn this feasibility study focused on assessing reliability, we randomlyselected 19 to 25-minute segments from the entire 4-hour video for eachcase. The remaining data from our Neonatal Intensive Care Unit dataset293.5. Overview of Datasetwill be analyzed and reported in future work.303.6. Requirement of Region of Interest3.6 Requirement of Region of Interest           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.11: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.The Region of Interest (ROI) represents an area in a dataset, image orvideo where we want to focus our attention for further analysis. The regionis selected based on its significance or relevance to a specific technologicaltask or operation. Identifying a ROI enables a more targeted and efficientapplication of algorithms, processes, or tools in the context of the overalltechnological objective. Numerous techniques exist in the literature for ROIselection and processing, for example, manual selection, automatic ROI orpattern recognition, and predefined ROI selected based on the dataset toname a few. [71]As stated above, the purpose of a ROI selection algorithm is to concen-trate attention on a defined area within an image, video or dataset deemedmost important for the study, which in the case of our study is the torso ofthe baby. By selectively focusing on specific portions of an image, we opti-mized processes, conserved computational resources, and enhanced analysis.This targeted approach did not require processing an entire image or dataset,allowing us more efficient and streamlined operations. One important thingto note in our study is that we do not only record videos while the conditionsare ideal ( i.e. baby is sleeping or calm; therefore, selecting ROI also helpsus to avoid additional noise that we might encounter if the baby is moving.We define a MATLAB function manualROIselectionstrategy. It is de-signed to facilitate the manual selection of regions of interest (ROIs) withinan image, particularly aimed at identifying the chest area in medical orphysiological contexts, in our case of estimating respiratory rate.313.6. Requirement of Region of InterestThe function takes a single input argument, an image from which ROIsare to be selected. The user is prompted to input the number of ROIsthey want to select through the input function. Our algorithm then guidesthe user to manually select a rectangular mask that covers the region theywant to analyze, essentially the chest area for this study’s purposes. This isachieved using the getrect function, which allows the user to draw a rectangleon the displayed image interactively. Additionally, a mask is created basedon the selected rectangle coordinates to highlight the chosen area within theimage. This mask can extract the pixel values from the original colour imagewithin the Region of Interest. The process of averaging pixel values acrossall chosen regions of interest (ROIs) results in the Respiratory Volume (RV)signal, which is integral to estimating the final respiratory rate. The manualROI selection interface is illustrated in Figure 3.12.Our approach allows the user to select multiple regions of interest, pro-viding the flexibility to combine results from all focused areas. This enhancesRR estimation accuracy compared to selecting a singular large ROI. Cur-rently, our methodology uses a manual ROI selection strategy. Future workwill automate the process. Figure 3.12: Region of Interest Selection323.7. Respiratory Volume Analysis3.6.1 Selecting Ideal Region of InterestAs mentioned above, the ideal ROIs encompass breathing-related move-ments while excluding movements associated with the heartbeat or upperand lower limbs. The quality of the RV depends on the selection of the ROI.Hence, it plays a crucial role in the estimation algorithm. [3] reported thata bigger ROI might pick up on the background motion and noise, whichcould lead to an incorrect noisy signal. Additionally, a smaller ROI couldeliminate important information, even though it might seem less noisy. Eventhough the algorithms detecting the entire torso might be ideal in literature,in clinical settings, selecting an ROI works more efficiently as they eliminateadditional noise. Bed sheets and clothing may also produce different derivedand actual tidal volumes. After the end user defines the region of interest,our algorithm performs spatial averaging within that ROI, leading to ex-tracting a Respiratory Volume Signal. We further mitigate the impact ofmotion artifacts (e.g., heartbeat-related movements) on RR estimation byfiltering the extracted breath pulse (respiratory volume signal), followed byan adaptive peak detection algorithm to ensure accurate RR estimation.           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.13: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.3.7 Respiratory Volume AnalysisThe volume of the chest cavity is changed due to inhalation and exhalationof the breath, which then results in the movement of the torso, specificallythe chest cavity. We analyze this change in the distance between the torsoand the depth camera to calculate the respiratory volume signal.333.7. Respiratory Volume Analysis3.7.1 Related workNumerous researchers have focused on understanding the properties of RVsignals. The methods are assessed against the references that are usuallyrecorded using a spirometer. Most of the literature integrates the change indepth across a user-defined ROI to calculate the respiratory volume signal.[92] utilized the machine learning approach to estimate the respiratoryrate using a volume signal recorded using a Kinect depth camera. Theystudied four participants and demonstrated the feasibility of using a depthcamera.In a real-time respiratory motion monitoring experiment conducted on10 volunteers by [103]. Their methodology used a model based in PCA toremove additional noise. They reported a correlation of 0.97 and used aspirometer as a reference instrument.[52] conducted a single volunteer study to estimate the respiratory rateusing a depth-sensing camera. The group used a respiratory belt to recordreference values and reported a correlation coefficient of 0.74.[45] investigated the influence of various parameters, such as distanceand sampling rate, on a signal generated using a depth-sensing camera.They reported that the respiratory SNR ratio deteriorated significantly asthe distance from the person to the depth camera increased, particularlyapproaching 4 meters. Additionally, decreasing the sampling rate providedmore time for processing but negatively impacted the SNR ratio.3.7.2 Extracting Respiratory Volume Signal           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.14: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.343.7. Respiratory Volume AnalysisOur methodology utilizes a depth-sensing camera and remains reliableacross various lighting conditions for estimating respiratory rate. In sub-sequent sections, we evaluate the robustness of our method by randomlyselecting 70 video segments from various infant recordings, each exhibitingdifferent lighting conditions. Table 3.2 gives an overview of different casesand the lighting conditions while the videos were recorded.The acquired depth data is converted into a compatible HDF5 format foranalysis and respiratory rate estimation. We iterate the dataset in chunksbased on the video length and load specific frames using array slicing. Thespecific frames, or the duration of the video segment, are decided based onthe length of videos used by the review to count the respiratory rate man-ually. The video segments were also classified into optimal and suboptimalbased on the baby’s movement. Optimal videos included short-durationsegments with minimal infant motion or slight caregiver shadow movement.On the other hand, suboptimal videos encompassed segments with upperor lower limb movement within permissible limits, the presence of parentsor nurse’s hands that did not obstruct the breathing-related area, or videosshowing mild infant crying without causing high-motion artifacts. [97] Table3.2 presents the percentage distribution of videos analyzed within the sub-optimal set. It is worth mentioning that in the literature, suboptimal videosare excluded from databases due to motion artifacts causing inaccuraciesin respiratory rate estimation. Our methodology allows users to select theregion they want to analyze manually. As an infant breathes in or out, thechest cavity’s volume changes, causing a change in the distance between thechest and the depth camera, consequently altering pixel intensities. Thisvariation in depth (pixel intensity) within the specified ROI positioned onthe infant’s chest can be summed over time, producing a Respiratory VolumeSignal.In the upcoming section, we will discuss the need to denoise the signalto remove motion artifacts, followed by applying an adaptive peak promi-nence algorithm. We evaluated the methodology by conducting a compar-ative analysis between the respiratory rate estimated using a depth cam-era, RRDepth and RRRef reference respiratory rate obtained through manualcounting by a nurse practitioner. Furthermore, we investigated the perfor-mance of RRDepth compared to respiratory rate estimates derived from RGBcamera data, RRRGB aiming to determine whether depth camera technologycould outperform RGB camera. Dr. Soodeh Ahani suggested the method-ology, based on utilizing the RGB camera for the estimation of respiratoryrate for infants in NICU. The work has not been published yet.353.8. DenoisingFigure 3.15: Respiratory Volume Signal extracted by integrating the depth(pixel intensity) variation across a specified region of interest.3.8 DenoisingAfter calculating the average of pixels over the selected region of interest(ROI) to determine the breath pulse, our focus shifted to addressing noiseintroduced by motion artifacts in the signal. The resulting respiratory vol-ume signal was non-stationary, as depicted in Figure 3.15. Through an ex-tensive literature review, we opted for a wavelet-based denoising approach,which has been widely used across various fields. Table 3.3 provides anoverview of techniques for removing motion artifacts from an extracted res-piratory volume signal. Discrete wavelet transforms prove highly effectivewhen analyzing non-stationary signals like the Respiratory Volume Signal.Furthermore, the computational complexity of a discrete wavelet transformis O(n), which is lower than that of the Fast Fourier Transform. We choseSymlet Wavelet as the mother wavelet to decompose the Respiratory VolumeSignal. [25]363.8. Denoising3.8.1 Related workThe application of wavelet transforms continues to evolve across disciplines,showcasing its versatility and ongoing advancements. Our denoising method-ology was implemented using MATLAB’s Wavelet Toolbox, leveraging thepower and flexibility of wavelet-based signal processing techniques. Re-cent literature discusses two main types of wavelet transforms: discrete andcontinuous. MATLAB’s Wavelet Toolbox makes it easier to use wavelettransforms.[76] The wavelet transform denoising methodology representsboth time and frequency domains. This is helpful for localized analysisof the signal’s frequency content. When the wavelet transform is appliedto a signal, it generates a time-frequency representation of the signal, oftenreferred to as the time-frequency amplitude matrix. This matrix representshow the frequency content of the signal changes over time. [37]The goal of denoising a wavelet is to remove noise without distortingthe signal. The fundamental assumption underlying denoising with wavelettransform is that respiratory volume signal and additional noise can be dis-tinguished and separated based on their distinct frequency characteristics.[4] In many scientific measurements, the desired signal components often oc-cupy lower frequency ranges, while noise tends to be more prevalent at higherfrequencies. The methodology requires defining a wavelet type and number,which then can influence the denoising process. Different wavelets havedifferent frequency localization properties, affecting how effectively theyseparate signals from noise. Common wavelet families include Daubechies,Coiflets, Symlets, and Haar wavelets, each with advantages and limitations.[90] The wavelet level refers to the number of decomposition levels used inthe wavelet transform. Increasing the level results in finer frequency resolu-tion but also increases the computational complexity. The choice of waveletlevel is crucial as it determines the frequency resolution of the analysis andhow effectively the signal and noise can be separated. Higher levels allowfor finer distinctions between signal and noise frequencies. [36] There hasbeen much research in image denoising and signal denoising using wavelettransforms [64], [10], [86], [37], [70].3.8.1.1 Foundational BackgroundThe discrete wavelet transform (DWT) is a valuable technique for denoisingsignals. The DWT offers time-frequency analysis capabilities, making itparticularly beneficial for applications in biomedical signal processing. [4].Selecting discrete values for dilation (a) and translation (b) parameters to373.8. Denoisingenhance computational efficiency is implemented through MultiresolutionAnalysis. [82]A waveform of effectively limited duration with an average value of zerois described as a wavelet. [90] It can be defined as. . .ψa,b(t) =1√aψ(t− ba)(3.1)a and b are real numbers, called Dilation (Scale) and Translation (Posi-tion) parameters, respectively.The Discrete Wavelet Transform (DWT) efficiently uses filter banks toconstruct multi-resolution analyses, as shown in Figure 3.16. CD is thedetailed coefficient, and CA is the approximation coefficient. Two filtersare employed within the DWT framework: low-pass (LPF) and high-pass(HPF). The approach integrates filtering and down-sampling, reducing thesignal by a factor of 2. [82]        Figure 3.16: Process flow of wavelet decomposition using Discrete Wavelettransform.Passing the original signal through these filters and down-sampling re-sults in the first level of wavelet transform. Two filters are also present inthe reconstruction step. But instead of downsampling, the signal is upsam-pled, increasing its sampling rate by a factor of 2 and inserting zeros duringthis process.383.8. DenoisingWavelet-based denoising utilizes the thresholding algorithms introducedby Johnstone and Donoho. [26] It typically uses hard thresholding or softthresholding. Universal, Minimax, and Steins unbiased estimates of riskand minimum description length are the commonly used threshold selectionmethods. [42]Our proposed methodology removes the motion artifacts (noise) in theRespiratory Volume signal by applying wavelet decomposition up to level 3,using Symlet 4 as the mother wavelet. We then apply the universal empiri-cal Bayesian method, incorporating soft thresholding and level-independentnoise estimation, to effectively denoise the signal.3.8.1.2 Literarture ReviewAuthor ROI Motion ArtefactsRemovalRelevant Signal Extrac-tionVillarroel etal.[98]Automatic None ICAKoolent etal.[48]Manual None PCA & ICAAlinovi etal. [11]Not clearlymentionedBinarization MLJanssen etal. [41]Automatic Upper-LimitThresholdingPeak DetectionEastwoodet al. [28]Automatic None Frequency Based Meth-odsGastel etal.[94]Manual Not mentioned Peak DetectionSun etal.[91]Manual None PCALorato etal.[54]Automatic Fourth OrderButterworthFilterPeak DetectionAntognoliet al.[13]Manual Keep a check onROIAveraging three chan-nelsTable 3.3: Overview of techniques used in camera-based neonatal respira-tory monitoring.The selection of the mother wavelet is crucial for the denoising method-393.8. Denoisingology. The Daubechies wavelet is one of the most widely used. However, itlacks symmetry.[99] developed an algorithm that used it as a mother waveletand successfully removed noise.V. N. Raj, Dr. T. Vekaswarlu[77] utilized undecimated wavelet transformto decompose the ECG signal and then applied methodology to limit noise.[10] proposed an approach based on wavelet transform coefficients. Sim-ilarly, [26] also used wavelet transform to remove noise.[25] studied different approaches for thresholding and various ”motherwavelets.” He reported that picking the correct ”mother wavelet” and thresh-olding technique for denoising is essential. [20] demonstrated that db4wavelets obtain the ideal results.[96] performed a comparative study on 17,520 images, mainly investi-gating two basic wavelet bases, Daubecheis and Symlet, for noise reductionin ultrasound images. The methods were evaluated based on Root MeanSquared Error (RMSE), SNR, and Q-index. The best results were reportedwhile using Daubechies at a lower level of decomposition.It is important to note that no universal mother wavelet would workfor all noises. Depending on the study’s nature, we can select the ”motherwavelet,” generally based on the region of interest, types of noise present,and the type of videos/images studied.[49] drew a comparison between haar and DB3 wavelets to remove thespeckle noise from ultrasound, CT Scan, MRI and X-ray, whereas [49] hascompared Haar, daubechie, Symlet, Coiflet wavelets with different threshold-ing techniques for removing additive white Gaussian noise. They reportedthat db3 is more efficient than Haar wavelet regarding visual and peak SNR.In [87], deduced among all the discrete wavelet bases, Coiflet performs wellin image denoising.The general process flow is segmented into three stages. First, the respi-ratory volume signals are decomposed into wavelet coefficients. Next, thesecoefficients undergo modification using threshold values. The final step in-volves adjusting the coefficients to produce a denoised signal.Many studies [66], [109], [40], [73] have utilized the thresholding tech-nique to efficiently denoise the ECG signals.There is also a noticeable trend among researchers using wavelet trans-forms with effective techniques to denoise the signal.Zhou et al.[101] preprocessed the signal using a combined soft thresh-olding and whitening algorithm. Further, he utilized ICA to remove motionartifacts from ECG signal. Inuso et al. [39] demonstrated the usage of ICA.[29] proposed an algorithm that decomposes the ECG signal using discretewavelet transform (DWT) up to 9th and 10th level. They reported an ab-403.8. Denoisingsolute error of 6.8 %. In [74], a new method is recommended that mixes themulti-resolution property of DWT with an Artificial Neural Network as anon-linear Adaptive filter.Our proposed methodology removes the motion artifacts (noise) in theRespiratory Volume signal by applying wavelet decomposition up to level 3,using Symlet 4 as the mother wavelet.3.8.2 Denoising Methodology           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.17: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.Based on an extensive literature review, [25], [64], [78], [37], [76], [88],[34], [18], [35], [83], [36] we discovered the utilization of wavelet transformfor denoising ECG signals. Drawing from this knowledge base, our researchaimed to investigate the application of wavelet transform for denoising therespiratory volume signal, specifically targeting the removal of motion arti-facts. This exploration was motivated by the need to enhance the qualityand reliability of respiratory volume measurements in physiological monitor-ing systems. As discussed above, wavelet transforms enable multiresolutionanalysis, allowing for the decomposition of signals into different frequencybands, as shown in 3.18. This provides a comprehensive view of signalcharacteristics at various scales, facilitating identifying and removing noisecomponents. The wavelet transform methodology is efficient and allows forreal-time and close-to-real-time denoising. This efficiency is advantageousfor our applications since we require rapid data processing and analysis.We leveraged MATLAB’s capabilities by defining custom functions forwavelet denoising wdenoise, specifically tailoring it according to the require-ments of the study. As mentioned, the discrete wavelet transform is based413.8. Denoising      Figure 3.18: Multilevel wavelet decomposition in Discrete Wavelet Trans-form.on the concept of wavelets, widely known as mother wavelets, which aresmall, localized wave-like functions well-suited for analyzing signals with lo-calized and global characteristics. We applied and analyzed the output ofthe methodology suggested in [25] in Figure 3.19, an algorithm developedin [29] in Figure 3.20, methodologies designed [49] in Figure 3.21 and Fig-ure 3.22. After a comprehensive literature review and thorough analysisof the performance of various mother wavelets and decomposition levels,we ultimately selected Symlet 4 as the mother wavelet for our methodol-ogy. Symlet 4 wavelets balance time-frequency localization, smoothness,adaptability, and compression efficiency, making them ideal for denoisingbiomedical signals. We further analyzed various levels of Symlet, for exam-ple, Figure 3.23, and based on the output of approximation coefficient, choseSymlet 4, decomposition level 3 as shown in Figure 3.24. We then apply theuniversal empirical Bayesian method, incorporating soft thresholding andlevel-independent noise estimation, to effectively remove motion artifactsand denoise the Respiratory Volume Signal, as shown in Figure 3.25.423.8. DenoisingFigure 3.19: Application of Discrete Wavelet Transforms Filter bank theoryusing daubechie 4 mother wavelet as discussed in [25].433.8. DenoisingFigure 3.20: Application of Discrete Wavelet Transforms Filter bank theoryusing daubechie 10 level 4 mother wavelet as discussed in [29].443.8. DenoisingFigure 3.21: Application of Discrete Wavelet Transforms Filter bank theoryusing daubechie 3 level 4 mother wavelet as discussed in [49].453.8. DenoisingFigure 3.22: Application of Discrete Wavelet Transforms Filter bank theoryusing Haar mother wavelet as discussed in [49].463.8. DenoisingFigure 3.23: Application of Discrete Wavelet Transforms Filter bank theoryusing Symlet 4 Level 6 mother wavelet as discussed in [96].473.8. DenoisingFigure 3.24: Application of Discrete Wavelet Transforms Filter bank theoryusing Symlet 4 Level 3 mother wavelet as discussed in our methodology.483.9. Peak DetectionFigure 3.25: Extracted Respiratory Volume Signal denoised applying Dis-crete Wavelet Transform.3.9 Peak DetectionWe feed the denoised breath pulse to the peak detection unit to estimate theRR. The peak detection unit then returns the local peaks of the denoisedbreath pulse. A local peak refers to a data sample that is greater in valuethan its two adjacent samples. Even after denoising the breath signal, alocal peak can occur near its neighbouring peak, mostly related to heartbeat-related movement; this peak should not be counted as a breath. We definea minimum distance between neighbouring peaks to eliminate such peaksthat are physically impossible.3.9.1 Related workAchieving accuracy while performing peak detection is essential. Peak de-tection algorithms are used in a wide range of signals, e.g. ECG, EEG, andAudio signal. In [75] a robust algorithm was adopted to calculate the localmaxima which helped locate each R peak. [76] suggests using wavelet and493.9. Peak DetectionHilbert transforms and employing adaptive thresholding technique to detectR peak. [107] breaks down the process into three stages: preprocessing thesignal, complete elimination of the motion artifacts, and R-peak detectionthrough an efficient ”area over the curve” method. The discrete wavelettransform removes the noise component of the time-frequency domain, andthe Hilbert Transform, using the adaptive thresholding technique, detectsthe peak. The performance of the proposed method is found to be betterin detecting R-peaks, having a sensitivity of 99.71 %. [24] removed high-frequency noises using a low pass filter to detect the peaks by collecting Rpeak localization points. A five-processing block methodology is presentedin [63]: Motion artifacts detection, deep learning denoising network, respi-ratory cycle segmentation, separation of discontinuous adventitious soundsfrom vesicular sounds and crackle peak detection. Our methodology adoptsan algorithm developed by a researcher in our lab based on adaptive peakprominence, but the work has not yet been published.3.9.2 Adaptive Peak Prominence           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.26: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.Although the breath signal was denoised, local peaks still occurred veryclosely to represent neighbouring peaks. To remove such a peak, which isphysically impossible to breathe, i.e. a peak caused by heart-related move-ments, we considered a minimum acceptable distance between two adjacentdetected breaths. An infant’s expected breathing rate range is 40 to 60bpm. According to the Intel Sense 415 documentation, the frame rate of thecamera is 30 frames per second; hence, the distance of two adjacent peaks503.9. Peak Detectionis less than 0.5 seconds or equivalent to 15 frames in our videos; the peakswith the lower amplitude will not be considered for RR counting. [71]To address various lighting conditions ( i.e. ambient light affects breathpulse amplitude), shallow breath detection(i.e. depth of breaths affects thebreath pulse), and also removing the small peaks (e.g. the heartbeat-relatedpeaks), we also considered the peak prominence threshold value in the breathcounting. [105]Our goal is to focus on the more significant breath-related peaks. Peakprominence can be defined as the height of the peak above the surroundingnoise or baseline level. The strength or significance of a peak is determinedby its peak prominence. The peak prominence threshold is the minimumamount a peak must stand out from its surroundings baseline to be consid-ered a distinct peak. [75]We calculate an adaptive setting of the peak prominence threshold. Weonly count the prominent peaks to the adjacent local peaks and ignorethe ones with a prominence below this threshold(e.g. the heartbeat-relatedpeaks).Figure 3.27: An example of the upper and lower envelopes of the breathpulse for automatic peak prominence threshold value estimation.513.9. Peak DetectionTo ensure that the algorithm is robust, we consider various signal am-plitudes to determine a suitable peak prominence threshold value for RRestimation.Ambient light levels and the extent of chest or abdomen movement dur-ing breathing influence the amplitude of each calculated breath pulse.The respiratory signal has high amplitude when the baby is quiet andthe room is well-lit. Similarly, during suboptimal instances the amplitude islow. The videos captured during low-lit and night time have less amplitudethan well-lit videos. We wanted to ensure we captured the videos withoutaltering infants’ environments, resulting in optimal and suboptimal videos.Therefore, an appropriate peak prominence threshold should be dynamicallyadjusted based on the breath pulse amplitude. Taking inspiration from ourRGB methodology, suggested by Soodeh Ahani, we build on an automaticpeak prominence threshold estimation methodology. by identifying the up-per and the lower envelopes of the breath pulse. Breath Pulse is depictedin Figure 3.27. The envelope function calculates the upper and lower en-velopes of the input signal using a moving average filter with a window sizeof 30 samples and peak-based envelope calculation. After computing theupper and lower envelopes, the envelope function returns two vectors: theupper envelope vector containing the maximum values within each windowof the moving average filter. These values represent the peaks or highestpoints in the signal within each window. lo: The lower envelope vectorcontaining the minimum values within each window of the moving aver-age filter. These values represent the valleys or lowest points in the signalwithin each window. The difference between the upper and lower envelopesrepresents the ”prominence” of each peak in the signal. It contains the am-plitude differences between the upper and lower envelopes, which representthe range of variation in the signal within each window of the moving av-erage filter. Peaks with larger differences between their peaks and valleysare more prominent. dm is the mean of these differences, providing an av-erage measure of peak prominence in the signal. MinPeakProminence isthen set to 5% (0.05) of the mean peak prominence (dm), which serves as athreshold for detecting significant peaks. Peaks with a prominence greaterthan this threshold are considered significant. We determined the optimalscaling factor of 0.5 based on the facts above and some trial and error. Weexperimented with different values and evaluated the results to find the mostsuitable value.The findpeaks function analyzes the signal and detects peaks based ontheir prominence and minimum distance criteria. Peaks are identified aslocal maxima that satisfy the specified conditions, including minimum peak523.9. Peak DetectionFigure 3.28: Detected Peaksprominence and minimum peak distance. The estimated RR is derived fromthe number of detected peaks occurring within a minute, as shown in Fig-ure 3.28. Furthermore, apnea and shallow breathing can be identified bycalculating the distance between two adjacent peaks. Abnormally long dis-tances between peaks may indicate apnea episodes, while unusually shortdistances may suggest shallow breathing patterns, which we will address infuture works.Our proposed methodology effectively estimated the respiratory rate us-ing the depth-sensing camera, as demonstrated in Figure 3.30. First, weloaded the raw high-dimensional depth data as chunks based on the optimaland suboptimal videos. Next, we manually selected a region of interest. Weextract the respiratory volume signal by computing the average pixel inten-sities, which change based on the chest’s distance from the depth-sensingcamera during an infant’s breathing cycles.However, the resulting respiratory volume signal was highly noisy dueto environmental factors, motion artifacts, and other sources of interference.We employed a discrete wavelet transform algorithm to mitigate this noise,decomposing the original respiratory volume signal using Symlet 4 up to level533.9. Peak Detection           !"#$ ! $  %""! $!  " & ' "% () !!(     Figure 3.29: Proposed steps for Depth-video-based Respiratory Rate (RR)estimation.3. Additionally, we established criteria for the minimum distance betweeneach peak and applied an adaptive peak prominence technique for peakdetection.We estimated the respiratory rate by analyzing the number of peaks inthe denoised respiratory volume signal. The process flow of our methodologyis depicted in Figure 3.1, outlining the steps involved. Additionally, Figure3.30 presents the milestone outputs at various stages within this processflow.543.9. Peak DetectionFigure 3.30: An example of the process of the proposed methodology.55Chapter 4Evaluation of the ProposedAlgortihmEstablishing reference values is crucial during algorithm evaluation. Theyserve as benchmarks, offering insights into the proposed algorithm’s accuracyand reliability. We set up a bedside monitor (Phillips IntelliVue MX800,Philips, Amsterdam, Netherlands) to collect vital signs. The respiratoryrate (RR) provided by the bedside monitor via IP sensors was recordedas the reference gold standard RR value. The data from the monitor wasacquired using a free program, The Vital Recorder. Additionally, a nursepractitioner manually counted the respiratory rate by observing the RGB-camera-based videos.However, we decided to use the respiratory rate determined through vi-sual assessment by the nurse and the values estimated from our proposedRGB method suggested in Video-based Respiratory Rate Estimation for In-fants in the NICU as our reference gold standard, aligning with the approachtaken in [47]. Notable disparities between the RR values provided by visualassessment and those reported by the bedside monitor have been reportedin [98], [47], and [33], supporting our claim.The manual annotation process was conducted offline, where the nursefrom NICU observed the RGB videos to record the values. During thedata collection phase, RGB camera videos of the infants were concurrentlyrecorded. However, manually counting respiratory rate as a gold referencepresented its challenges. We excluded certain cases due to missing manualannotations and corrupt data. It’s essential to highlight that this exclu-sion doesn’t introduce bias, as the remaining cases encompass diverse lightconditions and involve infants of varying ethnicities and gestational ages.[58] Notably, RGB cameras have a frame rate of 65 per second, while depthdata is captured at 30 per second. Furthermore, there is a 3-second lagbetween the depth-sensing and RGB-based cameras. This discrepancy re-sults in missed frames and potentially missed breaths. Subsequently, ourlab member developed an algorithm utilizing these videos, whose work hasnot been published yet. We intend to leverage the data obtained from this564.1. Evaluation Metricsalgorithm to evaluate whether the depth camera can surpass the RGB cam-era in performance. The respiratory rate estimated using the RGB camerawill be referred to as RRRGB. [69]4.1 Evaluation MetricsWe compared our proposed method’s results RR against the reference goldstandard RR, visually counted by our reviewer, a nurse practitioner. Itis important to note that the annotations are conducted offline and notduring video-capturing. It’s worth emphasizing that cases 3, 7, and 9 hadto be excluded from our analysis because the documentation in the man-ually annotated videos was insufficient. This decision was made to upholdthe accuracy and reliability of our research findings. By recognizing theseconstraints, we aim to maintain transparency and credibility in our dataanalysis.Using RGB camera-based videos is a common practice in computer vi-sion research. Depth-sensing cameras provide additional information aboutthe scene by capturing the distance of objects from the camera, which canbe valuable for various applications. We systematically evaluate the perfor-mance of RGB and depth camera-based videos on the same subjects anddetermine their relative strengths and weaknesses for the task.4.1.1 Bias and Root Mean Squared DifferenceUsing the statistical methods, Bias and accuracy, we evaluate the methodol-ogy against the reference. The bias represents the mean difference betweenthe test (RRdepth) and reference values (RRRef), while the root quantifies ac-curacy mean squared difference (RMSD) between the two datasets. RMSDquantifies the difference between the estimated and reference respiratoryrates [58], [3], [5]. They can be calculated by using the following formula:Bias =1nn∑i=1(RRdepthi − RRRefi) (4.1)andRMSD =√√√√ 1nn∑i=1(RRdepthi − RRRefi)2 (4.2)574.1. Evaluation Metricswhere i corresponds to the index of each pair of respiratory rate (RR)values analyzed. Random 70 respiratory rate pairs were computed, whereeach pair represented the average over 1 minute, optimal and suboptimalconditions.Furthermore, we compared the performance of the proposed method re-ported in [3]. Addison et al.[3] studied 14 healthy adult volunteers andprocessed depth-sensing data to generate RR. The respiratory rate valuesused as a reference were computed using a capnograph (RRcap). They re-ported the bias a of 0.04 bpm and a 0.66 bpm root mean squared difference(RMSD) accuracy.Additionally, we conducted a comparative analysis with an in-houseRGB camera-based feasibility study that was independently carried out byour research team. The objective is to determine whether depth camerasoutperform RGB cameras for the same subjects.4.1.2 Bland Altman AnalysisA Bland–Altman analysis methodology is utilized to evaluate the agreementbetween the values calculated using the depth-sensing camera and the man-ually annotated reference values [16], [32]. It provides information about thebias (i.e., the Mean of the Differences (MOD) of the estimated and referencevalues) and limit of agreements (LOA) [16], [32]. The LOA is defined as arange that 95% of the differences points fall within it [16], [32]. The LOAshows how precise the estimates are [32]. In Bland-Altman (BA) analysis,good agreement is when the points lie relatively close to the mean bias linewith diminished scattering [27].4.1.3 Pearson CorrelationWe also evaluate the performance of the proposed method by reporting thePearson correlation coefficients (i.e., Pearson’s r) that show whether andhow strongly the estimated (RRdepth) and the reference RR values (RRRef)are linearly correlated [16], [32]. The experiment is considered statisticallysignificant if Pearson’s p-value is smaller than the significance level (i.e., p <0.05)[16]. We also compare it with the in-house RGB-camera study, whichreports the Pearson correlation coefficients (i.e., Pearson’s r).4.1.4 ICCIntra-class correlation (ICC) is a statistical measure that quantifies the de-gree of similarity or consistency among observations made on the same in-584.2. Results of Proposed Methoddividuals or objects [62]. It is commonly used to evaluate the reliability andconsistency of measurements taken by different raters, assessors, or instru-ments. There are several types of ICC, each designed for a specific type ofdata and research design. The most commonly used types of ICC are:• Single measure ICC: used when a single measurement is taken on eachsubject or object. This type of ICC is also known as the consistencyICC.• Two-way random effect ICC: used when multiple measurements aretaken on each subject or object, and there are multiple raters or as-sessors. This type of ICC is also known as the agreement ICC.• Two-way mixed effect ICC: used when multiple measurements aretaken on each subject or object, and there are both multiple ratersor assessors and multiple subjects or objects. This type of ICC is alsoknown as the average measure ICC.The ICC values can range from 0 to 1, with 1 indicating perfect agree-ment or consistency among the observations and 0 indicating no agreementor consistency. Table 4.1 enlists the interpretation of intraclass correlationcoefficient (ICC)ICC value 0 to 0.5 0.5 to 0.75 0.75 to 0.9 0.9 to 1Reliability Level poor moderate good excellentTable 4.1: Interpretation of the Intraclass Correlation Coefficient values.It is widely used in various fields, such as medicine, psychology, andeducation. Different types of ICC are used for different types of data andresearch designs, and the interpretation of ICC values depends on the con-text. It is always recommended to report the type of ICC used, the ICCvalue, and the sample size used to calculate it.4.2 Results of Proposed MethodWe report the results (RRdepth) of our method against the gold standard,(RRRef)and the respiratory rates estimated using the RGB-camera methodology(RRRGB).594.2. Results of Proposed Method4.2.1 Bias and Root Mean Squared DifferenceThe bias was found to be 0.125 bpm, and the root mean squared difference(RMSD) accuracy is 1.058 bpm. These are calculated between RRdepth andRRRef. Furthermore, the bias and accuracy (RMSD) between RRdepth andRRRGB were found to be 0.164 bpm and 1.38 bpm, respectively.4.2.2 Bland Altman AnalysisFigure 4.1: BA plot of RRref vs RRdepthFigure 4.2 and Figure 4.1 show the BA plots of the proposed depthsensing-based method and RGB-camera-based method compared to the ref-erence values.The BA analysis reveals a similar underestimation by usingthe proposed method RRdepth (i.e., MOD = - 0.13 bpm), compared withthe EVM-based RGB RRRGB (MOD = - 0.16 bpm).Table 4.2 reports details of the BA plots (MOD ± LOAs) of our pro-posed method in contrast to the (i) RGB-based approach. Furthermore, weconduct a comparative analysis of our Bland-Altman findings with thosepresented for the (ii) impedance sensor as discussed in [47], and (iii) theEVM-based RR estimation method as described in [33].604.2. Results of Proposed MethodFigure 4.2: BA plot of RRref vs RRrgbWe report that the LOA of the proposed method (i.e.≤ 8.64) are rela-tively lower than the LOAs of RGB-camera-based methodology (i.e.≤ 11.49).Additionally, both of the impedance sensors [47] (i.e.≤ 20.96), and theEVM-based method in [33] (i.e. ≤ 22.8) report higher LOAs.Therefore, our proposed methods lower LOA suggests that our methodoutperforms the others, i.e. [47], [33] and RGB-camera-based preliminarystudy.We conclude that the estimated respiratory values generated using theproposed method have a better agreement with the reference gold standard(manually annotated in our case) than the respiratory rate calculated us-ing impedance sensor [47], the method in [33] and the RGB-camera-basedpreliminary study.We want to highlight that our study also includes a suboptimal datasetwith several motion artifacts. Previous methodology and literature (e.g., [3],[47] [97]) focus only on optimal datasets and remove the motion artifacts.We wanted to test the feasibility in real-life scenarios; hence, our datasetincludes both optimal and suboptimal videos.614.2. Results of Proposed MethodMeasurement 1 Measurement 2 MOD ±LOAsRRref RRdepth -0.13 ±8.64RRref RRrgb -0.16 ±11.49RRref [47] Impedance sensor[47]-0.4 ±20.96RR obtained byECG [33]EVM-based [33] 0.8 ±22.8Table 4.2: Results of BA Analysis (MOD ±LOAs) of the proposed methodcompared with RGB-camera based methodology, impedance sensor [47],EVM based method [33]4.2.3 Pearson CorrelationWe consider p < 0.05 statistically significant. The Pearson’s correlationvalue, r, and Pearson’s p-value of the estimated RR using the proposedalgorithm versus the reference respiratory rate are 0.87 and < .001, respec-tively. A linear relationship is suggested between RRref and RRdepth. ThePearson’s correlation value r for RGB-camera-based methodology is 0.75.Figure 4.3: Scatter plot of RRdepth against RRref aggregated over allsubjects624.2. Results of Proposed Method4.2.4 ICCEstimationMethodICC 95%ConfidenceInterval-LowerBound95% ConfidenceInterval-UpperBoundRRdepth 0.77 0.68 0.84Table 4.3: ICC value of the proposed depth-sensing RR estimation algo-rithmTable 4.3 reports the ICC values of the estimated RR by using a depth-sensing camera and RGB camera over the total dataset.We report the upper and lower bounds of the ICC 95% confidence in-terval for reliability analysis. The ICC value for respiratory estimation overthe total set is 0.77, indicating good reliability as suggested in Table 4.1.The 95% confidence interval is 0.68 to 0.84, which indicates a good level ofreliability based on the ranges reported in Table 4.1.63Chapter 5Discussion and Future WorksWe designed and developed a non-contact methodology for estimating therespiratory rate of infants in the Neonatal Intensive Care unit. The method-ology utilizes a depth-sensing camera, as demonstrated in Figure 3.30. First,we loaded the raw high-dimensional depth data as chunks based on the opti-mal and suboptimal videos. Next, we manually selected a region of interest.We extract the repository volume signal by computing the average pixel in-tensities, which change based on the chest’s distance from the depth-sensingcamera during an infant’s breathing cycles.However, the resulting respiratory volume signal was highly noisy dueto environmental factors, motion artifacts, and other sources of interference.We employed a discrete wavelet transform algorithm to mitigate this noise,decomposing the original respiratory volume signal using Symlet 4 up to level3. Additionally, we established criteria for the minimum distance betweeneach peak and applied an adaptive peak prominence technique for peakdetection. We estimated the respiratory rate by analyzing the number ofpeaks in the denoised respiratory volume signal.We evaluated the methodology’s robustness in various lighting condi-tions by analyzing results from 70 randomly chosen video segments coveringvarious infant cases. In the next sections, we discuss our key contributions.5.1 Our ContributionsThe infants have very sensitive skin. hence, non-contact respiratory rateestimation is highly desirable in infants. We conducted a feasibility studyand determined the viability and benefits of using depth-sensing camerasto estimate the respiratory rate of infants in the NICU, paving the way forfuture research and clinical implementation.Our achievements are as follows:• We captured videos based on parental consent. Therefore, our datasetincludes videos of sleeping infants and instances of suboptimal condi-tions( i.e., crying baby, presence of clinician hand, limbs or head move-645.1. Our Contributionsment). It is common in the literature to exclude suboptimal videosfrom databases due to motion artifacts causing inaccuracies in respi-ratory rate estimation. The proposed algorithm provided an accurateRR even if the infant is covered in a swaddle or a blanket.• The sample size of our testing dataset is 70 randomly selected videos.Typically, only 7-9 videos are analyzed in the literature.• Our proposed method is reliable and effective across different lightingscenarios.• We address the possibility of HR extraction while estimating the respi-ratory rate (RR) by defining a minimum peak distance while applyinga peak detection algorithm.• Our method eliminates the effect of skin pigments on the accuracy ofmeasurement. Indeed, the infants belong to different ethnicities.• Our method allows users to choose the region of interest they wish touse for calculating the respiratory rate.• We introduce the usability of wavelet transforms, applying Symlet 4as the mother wavelet for removing motion artifacts and denoising thesignal.• Our method successfully applies an adaptive peak prominence thresh-old value for peak detection.• The algorithm outperforms the RGB-camera-based method, which wassuggested Video camera-based respiratory rate estimation of infants inNICU. The paper is not published yet.The depth-sensing camera captures the variation in the distance causedby the chest expanding and contracting during breathing. Depth maps areoften converted into colour maps to facilitate- easier human interpretation.Different colours are assigned to different distances, such as cyan for closerobjects and red for farther ones. This change in the distance between thechest and the camera leads to a change in the intensity of pixel values in thedepth videos, resulting in the extraction of the Respiratory Volume Signal.Therefore, our methodology is effective for individuals of all skin colours andethnicities since it does not rely on analyzing skin pigment colour, which isnot captured by the depth-sensing camera.655.1. Our ContributionsTo evaluate the preliminary study’s feasibility of using depth-sensingvideos for respiratory rate estimation, we reported the results of 70 ran-domly selected short-time videos from the recruited cases. The videos wererecorded under various lighting conditions, including infants in low-lit, well-lit, and dark environments. Typically, in literature, the videos of the subjectsare recorded under well-lit optimal conditions. The studies often focus onhealthy subjects. Since our dataset includes videos recorded in various lightconditions, one of the main objectives of our methodology is to automat-ically adapt and adjust to changes in lighting, shallow breaths and severemotion artifacts. Inspired by the RGB-camera-based preliminary study, weutilized the adaptive peak prominence threshold value calculation algorithm.Furthermore, a ”minimum peak distance” was defined, which further helpedus remove unwanted exacted values, such as the heartbeat.Furthermore, we recorded the videos based on the parents’ consent.Therefore, our dataset includes videos of sleeping infants and instances ofsuboptimal conditions( i.e., crying baby, presence of clinician hand, limbs orhead movement ). Various benchmarking studies [71], [47], [1], [98] excludesuboptimal videos from dataset. The movement of the infant, also known asmotion artifacts and environmental conditions around the crib, introducednoise. Based on an extensive literature review, we investigated the appli-cation of discrete wavelet transforms to denoise a respiratory volume signalextracted by averaging over a user-selected ROI. A key highlight of our pro-posed methodology is the ability for users to choose the specific region theywish to analyze, such as the torso or chest area, as demonstrated in ourstudy. To remove environmental noise and motion artifacts, we leveragedMATLAB’s capabilities by defining custom functions for wavelet denoisingwdenoise, specifically tailoring it according to the requirements of the study.We demonstrated the use of Symlet 4 as the mother wavelet and decomposedthe noisy signal using the filter bank theory to level 3. To the best of ourknowledge, this methodology outperformed other algorithms mentioned inthe literature on our dataset. It is important to note that the type of motherwavelet can affect the quality of results and that suitable mother waveletscan vary for various applications.We used the visually annotated values as the gold reference and com-pared our methodology to an RGB-camera-based methodology that studiedthe same subjects. The performance of the proposed algorithm was evalu-ated over a region of interest that can be manually selected. The ICC valuesreported in Table 4.3 indicate good reliability of the proposed RR estima-tion method. Our method reports the lower bounds of the 95% confidenceintervals of the ICC values (i.e., 0.68 and 0.84).665.1. Our ContributionsWe also evaluated the performance using the Bland-Altman Analysis andreported the mean difference to be − 0.13 bpm between our estimated valueand the reference manually annotated values. Additionally, our methodologyhas a lower limit of agreement than that reported in [47] and [33] It isimportant to note that the proposed depth-sensing algorithm outperformsthe RGB-camera-based algorithm. Table 4.2 indicated that the LOAs ofour estimation method (i.e.≤ 8.64) are relatively lower than the LOAs ofRGB-camera-based methodology (i.e.≤ 11.49). Therefore, there is a higheragreement between our methodology and the reference gold standard valuesthan those calculated using RGB-camera data.Our methodology showcased that depth-sensing cameras can be utilizedto estimate respiratory rates in a real-world clinical setting.5.1.1 Neonatal Intensive Care Unit ToolWe designed the study to simulate real-world scenarios to create a toolsuitable for clinical use.As mentioned, we captured the videos based on the parent’s consent.Therefore, the infant was not always sleeping. Our dataset includes instancesof suboptimal conditions( i.e., crying baby, presence of clinician hand, limbsor head movement ). Furthermore, the lighting conditions of a clinical wardchange throughout the day. To address this challenge, we applied an adap-tive peak-prominence algorithm.The COVID-19 pandemic highlighted the necessity for non-contact meth-ods to monitor vital signs. This, in addition to the vulnerability of infants’delicate skin and susceptibility to infections, emphasized the necessity of anon-contact approach to estimating vital signs, including respiration rate,in our study.Our methodology estimates the respiratory rate of infants using a depth-sensing camera. It is effective for individuals of all skin colours and ethnic-ities since it does not rely on analyzing skin pigment colour. The depth-sensing camera can be installed on the infant’s crib, limiting the requirementfor a contact device. Hourly videos can be analyzed to monitor the infant’srespiratory rate. The estimated respiratory rate can help clinicians monitorand decide if further urgent medical intervention is required.675.2. Limitation5.2 LimitationThere are various limitations of our study:• The study is currently extended to relatively stable infants. In futurework, we want to evaluate the performance of a dataset that includesinfants who were very sick, premature, or on a ventilator.• Currently, we have limited samples during night-time. We will extendvarious light conditions in upcoming studies.• The sample size was 70 videos. We want to expand our studies, eventhough our sample size is commendable according to previous litera-ture.• The absence of documented gold reference values also hindered ourstudy. Despite having access to videos, a significant amount of datawas lost during project transfers. As a next step, we intend to expandthe study and analyze the remaining videos.• In our study, we analyzed different mother wavelets to denoise thedata. We can further refine the results by applying various threshold-ing algorithms and filters.• The adaptive peak-prominence algorithm was designed using an RGBdataset. Even though the results are promising, and our algorithmoutperforms the RGB camera, we can further explore customizing thealgorithm based on depth videos.68Chapter 6ConclusionMeasuring vital signs is crucial in healthcare as they provide essential in-formation about a person’s physiological state. They are a set of objectivemeasurements that reflect the basic functions necessary for life. Vital signssuch as body temperature, blood oxygen levels, and respiratory rate (RR)help clinicians assess acutely ill patients and predict the need for immediateclinical intervention. Changes in vital signs can sometimes indicate earlyhealth problems. Monitoring these changes can aid in the early detectionof potential issues, enabling timely care. The initial months following thebirth are critical for the infant’s survival. Respiratory diseases are one ma-jor cause of life-threatening illness in infants, specifically preterm neonates,since they are not fully developed. They often require specialized NeonatalIntensive Care Unit (NICU) care immediately after birth. As mentioned,early diagnosis and assessment of respiratory dysfunction help with requiredmedical treatment, and monitoring the respiratory rate of infants can savetheir lives. Neonates, especially those delivered before 29 weeks of gestation,have very fragile and sensitive skin. Therefore, a contactless approach wouldbe ideal for monitoring neonatal respiratory rates and be suitable for use inintensive care units and home environmentsFor our preliminary study, we have designed a depth-camera-based respi-ratory estimation methodology that also addresses the limitations recordedin past works, such as the need for visible skin, optimal lighting conditions,and removal of motion artifacts. Our analysis demonstrates that the pro-posed algorithm effectively overcomes all the challenges and estimates therespiratory rate within the typical clinical care setting without specific re-quirements, showing excellent potential for practical application in clinicalenvironments.One of the primary advantages of using depth-sensing cameras is themitigation of skin pigmentation effects when calculating values. Existingliterature has highlighted instances where a bias is introduced while usingcolour-based cameras. This specifically affects the pulse oximetry values forpatients with darker skin tones.Currently, end-users can select the region they wish to analyze. We69Chapter 6. Conclusiondefine a MATLAB function manualROIselectionstrategy. It is designed tofacilitate the manual selection of regions of interest (ROIs) within an image,particularly aimed at identifying the chest area in medical or physiologicalcontexts, in our case of estimating respiratory rate. To automate this pro-cess, we aim to employ machine learning algorithms to detect the region ofinterest automatically in the future.It is important to highlight that captured videos are based on parentalconsent. Therefore, our dataset includes videos of sleeping infants and in-stances of suboptimal conditions( i.e., crying baby, presence of clinicianhand, limbs or head movement). It is common in the literature to excludesuboptimal videos from databases due to motion artifacts causing inaccu-racies in respiratory rate estimation. The proposed algorithm provided anaccurate RR even if the infant is covered in a swaddle or a blanket. Wesuccessfully remove the motion artifacts (noise) in the extracted signal byapplying wavelet decomposition up to level 3, using Symlet 4 as the motherwavelet. We intend to explore independent component analysis (ICA) tomitigate motion artifacts and overcome the challenges associated with sig-nificant body movements.One of the main objectives of our methodology is to automatically adaptand adjust to changes in lighting, shallow breaths and severe motion arti-facts. Inspired by the RGB-camera-based preliminary study, we utilized theadaptive peak prominence threshold value calculation algorithm. There-fore, our algorithm is effective in various lighting conditions. We evaluatedthe algorithm’s robustness by randomly selecting 70 short-duration videos,totalling an analyzed data duration of 155.9 minutes, nearly 2.5 hours, span-ning different infant cases. Furthermore, a ”minimum peak distance” wasdefined, which further helped us remove unwanted exacted values, such asthe heartbeat.Our results record a bias of 0.125 bpm and a root mean squared differenceaccuracy of 1.058 bpm.Moreover, our proposed method exhibits a 26% increase in accuracycompared to the RGB camera-based method for respiratory rate estimationin infants, that is suggested by Soodeh Ahani. The paper is not publishedyet. The Pearson correlation coefficient of 0.87 bpm (p ¡ 0.001) indicatesa strong correlation, and the intraclass correlation coefficient of 0.77 bpm(95%CI: 0.68 bpm − 0.84 bpm) demonstrates good reliability.The Bland-Altman analysis of the proposed algorithm shows higheragreement between the values estimated by our developed method and thevisually counted respiratory rate than the agreement between the respira-tory rate obtained from the impedance sensors and the reference respiratory70Chapter 6. Conclusionrate [47], the agreement between a former EVM-based method [33], and therespiratory rate estimated during the RGB-camera-based study.Our preliminary study shows that the proposed algorithm can success-fully estimate the respiratory rate in the regular clinical care environment.71Bibliography[1] Anthony P. Addison, Paul S. Addison, Philip Smit, DominiqueJacquel, and Ulf R. Borg. Noncontact Respiratory Monitoring UsingDepth Sensing Cameras: A Review of Current Literature. Sensors,21(4):1135, February 2021.[2] Paul S. Addison, Philip Smit, Dominique Jacquel, Anthony P. Ad-dison, Cyndy Miller, and Gardner Kimm. Continuous non-contactrespiratory rate and tidal volume monitoring using a Depth SensingCamera. Journal of Clinical Monitoring and Computing, 36(3):657–665, June 2022.[3] Paul S. Addison, Philip Smit, Dominique Jacquel, and Ulf R. Borg.Continuous respiratory rate monitoring during an acute hypoxic chal-lenge using a depth sensing camera. Journal of Clinical Monitoringand Computing, 34(5):1025–1033, October 2020.[4] Ali N. Akansu, Wouter A. Serdijn, and Ivan W. Selesnick. Emergingapplications of wavelets: A review. Physical Communication, 3(1):1–18, March 2010.[5] Raghda Al-Halawani, Peter H Charlton, Meha Qassem, and Panayio-tis A Kyriacou. A review of the effect of skin pigmentation on pulseoximeter accuracy. Physiological Measurement, 44(5):05TR01, May2023.[6] Farah AL-Khalidi, Reza Saatchi, Heather Elphick, and Derek Burke.Tracing the Region of Interest in Thermal Human Face for Respi-ration Monitoring. International Journal of Computer Applications,119(4):42–46, June 2015.[7] F.Q. AL-Khalidi, R. Saatchi, D. Burke, H. Elphick, and S. Tan. Res-piration rate monitoring methods: A review: Respiration Rate Moni-toring Methods. Pediatric Pulmonology, 46(6):523–529, June 2011.72Bibliography[8] Ali Al-Naji and Javaan Chahl. Remote respiratory monitoring sys-tem based on developing motion magnification technique. BiomedicalSignal Processing and Control, 29:1–10, August 2016.[9] Ali Al-Naji, Kim Gibson, Sang-Heon Lee, and Javaan Chahl. RealTime Apnoea Monitoring of Children Using the Microsoft Kinect Sen-sor: A Pilot Study. Sensors, 17(2):286, February 2017.[10] Mikhled Alfaouri and Khaled Daqrouq. ECG Signal Denoising ByWavelet Transform Thresholding. American Journal of Applied Sci-ences, 5(3):276–281, March 2008.[11] Davide Alinovi, Luca Cattani, Gianluigi Ferrari, Francesco Pisani,and Riccardo Raheli. Spatio-temporal video processing for respiratoryrate estimation. In 2015 IEEE International Symposium on MedicalMeasurements and Applications (MeMeA) Proceedings, pages 12–17,Torino, Italy, May 2015. IEEE.[12] John Allen. Photoplethysmography and its application in clinicalphysiological measurement. Physiological Measurement, 28(3):R1–R39, March 2007.[13] Luca Antognoli, Paolo Marchionni, Susanna Spinsante, Stefano No-bile, Virgilio Paolo Carnielli, and Lorenzo Scalise. Enanced videoheart rate and respiratory rate evaluation: standard multiparametermonitor vs clinical confrontation in newborn patients. In 2019 IEEEInternational Symposium on Medical Measurements and Applications(MeMeA), pages 1–5, Istanbul, Turkey, June 2019. IEEE.[14] Philippe Arlotto, Michel Grimaldi, Roomila Naeck, and Jean-MarcGinoux. An Ultrasonic Contactless Sensor for Breathing Monitoring.Sensors, 14(8):15371–15386, August 2014.[15] Flavia Benetazzo, Alessandro Freddi, Andrea Monteriu`, and SauroLonghi. Respiratory rate detection algorithm based on RGB-D cam-era: theoretical background and experimental results. HealthcareTechnology Letters, 1(3):81–86, September 2014.[16] J. M. Bland and D. G. Altman. Statistical methods for assessing agree-ment between two methods of clinical measurement. Lancet (London,England), 1(8476):307–310, February 1986.73Bibliography[17] Idar Johan Brekke, Lars H˚aland Puntervoll, Peter Bank Pedersen,John Kellett, and Mikkel Brabrand. The value of vital sign trends inpredicting and monitoring clinical deterioration: A systematic review.PloS One, 14(1):e0210875, 2019.[18] C. Brouse, G.A. Dumont, F.J. Herrmann, and J.M. Ansermino. Awavelet approach to detecting electrocautery noise in the ECG. IEEEEngineering in Medicine and Biology Magazine, 25(4):76–82, July2006.[19] Annalisa Cenci, Daniele Liciotti, Emanuele Frontoni, AdrianoMancini, and Primo Zingaretti. Non-Contact Monitoring of PretermInfants Using RGB-D Camera. In Volume 9: 2015 ASME/IEEE In-ternational Conference on Mechatronic and Embedded Systems andApplications, page V009T07A003, Boston, Massachusetts, USA, Au-gust 2015. American Society of Mechanical Engineers.[20] Gwo-Ching Chang. A comparative analysis of various respiratorysound denoising methods. In 2016 International Conference on Ma-chine Learning and Cybernetics (ICMLC), pages 514–518, Jeju Island,South Korea, July 2016. IEEE.[21] Kun-Mu Chen, Devendra Misra, Huei Wang, Huey-Ru Chuang, andElliot Postow. An X-Band Microwave Life-Detection System. IEEETransactions on Biomedical Engineering, BME-33(7):697–701, July1986.[22] Pierre Comon. Independent component analysis, A new concept? Sig-nal Processing, 36(3):287–314, April 1994.[23] Tin T. Dang, Canh Huynh, Khoa Tran, Anh Dinh, Huy D. Nguyen,and Quan M. Vo Le. Measuring human respiration rate using depthframes of PrimeSense camera. In 2015 International Conference onAdvanced Technologies for Communications (ATC), pages 411–416,Ho Chi Minh, Vietnam, October 2015. IEEE.[24] S. Das, S. Mukherjee, S. Chatterjee, and H. K. Chatterjee. Noiseelimination and ECG R peak detection using wavelet transform. In2016 IEEE 7th Annual Ubiquitous Computing, Electronics & MobileCommunication Conference (UEMCON), pages 1–5, New York City,NY, USA, October 2016. IEEE.74Bibliography[25] Cigdem Polat Dautov and Mehmet Sirac Ozerdem. Wavelet trans-form and signal denoising using Wavelet method. In 2018 26th SignalProcessing and Communications Applications Conference (SIU), pages1–4, Izmir, Turkey, May 2018. IEEE.[26] D.L. Donoho. De-noising by soft-thresholding. IEEE Transactions onInformation Theory, 41(3):613–627, May 1995.[27] Nurettin O¨zgu¨r Dog˘an. Bland-Altman analysis: A paradigm to un-derstand correlation and agreement. Turkish Journal of EmergencyMedicine, 18(4):139–141, December 2018.[28] Caillin Eastwood-Sutherland. Camera-based neonatal respiratorymonitoring. page 10032850 Bytes, 2023. Artwork Size: 10032850Bytes Publisher: University Of Tasmania.[29] A. Espiritu Santo and C. Carbajal. Respiration rate extraction fromECG signal via discrete wavelet transform. In 2010 2nd Circuitsand Systems for Medical and Environmental Applications Workshop(CASME), pages 1–4, Merida, Mexico, 2010. IEEE.[30] M. Folke, L. Cernerud, M. Ekstro¨m, and B. Ho¨k. Critical reviewof non-invasive respiratory monitoring in medical care. Medical &Biological Engineering & Computing, 41(4):377–383, July 2003.[31] Mia Folke, Fredrik Granstedt, Bertil Ho¨k, and H˚akan Scheer. Com-parative provocation test of respiratory monitoring methods. Journalof Clinical Monitoring and Computing, 17(2):97–103, February 2002.[32] Davide Giavarina. Understanding Bland Altman analysis. BiochemiaMedica, 25(2):141–151, 2015.[33] Kim Gibson, Ali Al-Naji, Julie-Anne Fleet, Mary Steen, Javaan Chahl,Jasmine Huynh, and Scott Morris. Noncontact Heart and Respira-tory Rate Monitoring of Preterm Infants Based on a Computer VisionSystem: Protocol for a Method Comparison Study. JMIR ResearchProtocols, 8(8):e13400, August 2019.[34] A. Graps. An introduction to wavelets. IEEE Computational Scienceand Engineering, 2(2):50–61, 1995.[35] Maximilian Grobbelaar, Souvik Phadikar, Ebrahim Ghaderpour,Aaron F. Struck, Nidul Sinha, Rajdeep Ghosh, and Md. Zaved Iqubal75BibliographyAhmed. A Survey on Denoising Techniques of ElectroencephalogramSignals Using Wavelet Transform. Signals, 3(3):577–586, August 2022.[36] Tiantian Guo, Tongpo Zhang, Enggee Lim, Miguel Lopez-Benitez, FeiMa, and Limin Yu. A Review of Wavelet Analysis and Its Applications:Challenges and Opportunities. IEEE Access, 10:58869–58903, 2022.[37] Aminou Halidou, Youssoufa Mohamadou, Ado Adamou Abba Ari,and Edinio Jocelyn Gbadoubissa Zacko. Review of wavelet denoisingalgorithms. Multimedia Tools and Applications, 82(27):41539–41569,November 2023.[38] Shaimaa Hameed Shaker and Farah Q. Al-Khalidi. Human Gender andAge Detection Based on Attributes of Face. International Journal ofInteractive Mobile Technologies (iJIM), 16(10):176–190, May 2022.[39] Giuseppina Inuso, Fabio La Foresta, Nadia Mammone, andFrancesco Carlo Morabito. Wavelet-ICA methodology for efficient ar-tifact removal from Electroencephalographic recordings. In 2007 In-ternational Joint Conference on Neural Networks, pages 1524–1529,Orlando, FL, USA, August 2007. IEEE. ISSN: 1098-7576.[40] Md Kafiul Islam, Amir Rastegarnia, and Zhi Yang. A Wavelet-BasedArtifact Reduction From Scalp EEG for Epileptic Seizure Detection.IEEE Journal of Biomedical and Health Informatics, 20(5):1321–1332,September 2016.[41] Rik Janssen, Wenjin Wang, Andreia Moc¸o, and Gerard De Haan.Video-based respiration monitoring with automatic region of interestdetection. Physiological Measurement, 37(1):100–114, January 2016.[42] Iain M. Johnstone and Bernard W. Silverman. Needles and straw inhaystacks: Empirical Bayes estimates of possibly sparse sequences.The Annals of Statistics, 32(4), August 2004.[43] Joa˜o Jorge, Mauricio ViIllarroel, Sitthichok Chaichulee, Kenny Mc-Cormick, and Lionel Tarassenko. Data fusion for improved camera-based detection of respiration in neonates. In Gerard L. Cote´, editor,Optical Diagnostics and Sensing XVIII: Toward Point-of-Care Diag-nostics, page 36, San Francisco, United States, February 2018. SPIE.[44] Walter Karlen, Ainara Garde, Dorothy Myers, Cornie Scheffer, J MarkAnsermino, and Guy A Dumont. Estimation of Respiratory Rate From76BibliographyPhotoplethysmographic Imaging Videos Compared to Pulse Oximetry.IEEE Journal of Biomedical and Health Informatics, 19(4):1331–1338,July 2015.[45] Walter Karlen, S. Raman, J. M. Ansermino, and G. A. Dumont. Mul-tiparameter Respiratory Rate Estimation From the Photoplethysmo-gram. IEEE Transactions on Biomedical Engineering, 60(7):1946–1953, July 2013.[46] Fatema-Tuz-Zohra Khanam, Asanka G. Perera, Ali Al-Naji, Kim Gib-son, and Javaan Chahl. Non-Contact Automatic Vital Signs Monitor-ing of Infants in a Neonatal Intensive Care Unit Based on NeuralNetworks. Journal of Imaging, 7(8):122, July 2021.[47] S Kohn, D Waisman, J Pesin, A Faingersh, I C Klotzman, C Levy,G Hirshberg, A Rotschild, and A Landesberg. Monitoring the res-piratory rate by miniature motion sensors in premature infants: acomparative study. Journal of Perinatology, 36(2):116–120, February2016.[48] Terry K. Koo and Mae Y. Li. A Guideline of Selecting and ReportingIntraclass Correlation Coefficients for Reliability Research. Journal ofChiropractic Medicine, 15(2):155–163, June 2016.[49] Manish Kumar and Sudhansu Kumar Mishra. A Comprehensive Re-view on Nature Inspired Neural Network based Adaptive Filter forEliminating Noise in Medical Images. Current Medical Imaging For-merly Current Medical Imaging Reviews, 16(4):278–287, May 2020.[50] Soojeong Lee, Hyeonjoon Moon, Chang-Hwan Son, and GangseongLee. Respiratory Rate Estimation Combining AutocorrelationFunction-Based Power Spectral Feature Extraction with GradientBoosting Algorithm. Applied Sciences, 12(16):8355, August 2022.[51] P Leonard. Standard pulse oximeters can be used to monitor respi-ratory rate. Emergency Medicine Journal, 20(6):524–525, November2003.[52] Wan Tin Lim, Andrew Hs Fang, Chian Min Loo, Kok Seng Wong,and Tharmmambal Balakrishnan. Use of the National Early WarningScore (NEWS) to Identify Acutely Deteriorating Patients with Sepsisin Acute Medical Ward. Annals of the Academy of Medicine, Singa-pore, 48(5):145–149, May 2019.77Bibliography[53] Kuan-Yi Lin, Duan-Yu Chen, Chen Yang, Kuo-Jui Chen, and Wen-Jiin Tsai. Automatic Human Target Detection and Remote Respira-tory Rate Monitoring. In 2016 IEEE Second International Conferenceon Multimedia Big Data (BigMM), pages 354–356, Taipei, Taiwan,April 2016. IEEE.[54] Ilde Lorato, Sander Stuijk, Mohammed Meftah, Deedee Kommers,Peter Andriessen, Carola Van Pul, and Gerard De Haan. Multi-camerainfrared thermography for infant respiration monitoring. BiomedicalOptics Express, 11(9):4848, September 2020.[55] Jesu´s La´zaro, Eduardo Gil, Raquel Bailo´n, Ana Minchole´, and PabloLaguna. Deriving respiration from photoplethysmographic pulsewidth. Medical & Biological Engineering & Computing, 51(1-2):233–242, February 2013.[56] Jesu´s La´zaro, Yunyoung Nam, Eduardo Gil, Pablo Laguna, and Ki HChon. Respiratory rate derived from smartphone-camera-acquiredpulse photoplethysmographic signals. Physiological Measurement,36(11):2317–2333, November 2015.[57] Manuel Martinez and Rainer Stiefelhagen. Breathing Rate Monitoringduring Sleep from a Depth Camera under Real-Life Conditions. In2017 IEEE Winter Conference on Applications of Computer Vision(WACV), pages 1168–1176, Santa Rosa, CA, USA, March 2017. IEEE.[58] Carlo Massaroni, Emiliano Schena, Sergio Silvestri, and SoumyajyotiMaji. Comparison of two methods for estimating respiratory wave-forms from videos without contact. In 2019 IEEE International Sym-posium on Medical Measurements and Applications (MeMeA), pages1–6, Istanbul, Turkey, June 2019. IEEE.[59] Lalit Maurya, Pavleen Kaur, Deepak Chawla, and Prasant Mahapa-tra. Non-contact breathing rate monitoring in newborns: A review.Computers in Biology and Medicine, 132:104321, May 2021.[60] Lalit Maurya, Reyer Zwiggelaar, Deepak Chawla, and Prasant Ma-hapatra. Non-contact respiratory rate monitoring using thermal andvisible imaging: a pilot study on neonates. Journal of Clinical Moni-toring and Computing, 37(3):815–828, June 2023.78Bibliography[61] B. Mazzanti, C. Lamberti, and J. De Bie. Validation of an ECG-derived respiration monitoring method. In Computers in Cardiology,2003, pages 613–616, Thessaloniki Chalkidiki, Greece, 2003. IEEE.[62] Kenneth O. McGraw and S. P. Wong. Forming inferences about someintraclass correlation coefficients. Psychological Methods, 1(1):30–46,March 1996.[63] Ian McLane, Eline Lauwers, Toon Stas, Ilene Busch-Vishniac, KrisIdes, Stijn Verhulst, and Jan Steckel. Comprehensive Analysis Sys-tem for Automated Respiratory Cycle Segmentation and CracklePeak Detection. IEEE Journal of Biomedical and Health Informat-ics, 26(4):1847–1860, April 2022.[64] B Molavi and G A Dumont. Wavelet based motion artifact removal forFunctional Near Infrared Spectroscopy. In 2010 Annual InternationalConference of the IEEE Engineering in Medicine and Biology, pages5–8, Buenos Aires, August 2010. IEEE.[65] R. Murthy, I. Pavlidis, and P. Tsiamyrtzis. Touchless monitoring ofbreathing function. In The 26th Annual International Conference ofthe IEEE Engineering in Medicine and Biology Society, volume 3,pages 1196–1199, San Francisco, CA, USA, 2004. IEEE.[66] Sasu Ma¨kela¨, Jan Kujala, and Riitta Salmelin. Removing ocular arti-facts from magnetoencephalographic data on naturalistic reading ofcontinuous texts. Frontiers in Neuroscience, 16:974162, December2022.[67] Andrea Nicolo`, Carlo Massaroni, Emiliano Schena, and Massimo Sac-chetti. The Importance of Respiratory Rate Monitoring: From Health-care to Sport and Exercise. Sensors, 20(21):6396, November 2020.[68] Y. Nishida, T. Hori, T. Suehiro, and S. Hirai. Monitoring of breathsound under daily environment by ceiling dome microphone. InSMC 2000 Conference Proceedings. 2000 IEEE International Confer-ence on Systems, Man and Cybernetics. ’Cybernetics Evolving to Sys-tems, Humans, Organizations, and their Complex Interactions’ (Cat.No.00CH37166), volume 3, pages 1822–1829, Nashville, TN, USA,2000. IEEE.[69] Catherine Muthoni Njeru, J. Mark Ansermino, William M. Macharia,and Dustin T. Dunsmuir. Variability of respiratory rate measurements79Bibliographyin neonates- every minute counts. BMC Pediatrics, 22(1):16, Decem-ber 2022.[70] Rajesh Patil and Surendra Bhosale. Medical Image Denoising Tech-niques: A Review. International Journal on Engineering, Science andTechnology, 4(1):21–33, January 2022.[71] M Paul, S Karthik, J Joseph, M Sivaprakasam, J Kumutha, S Leon-hardt, and C Hoog Antink. Non-contact sensing of neonatal pulse rateusing camera-based imaging: a clinical feasibility study. PhysiologicalMeasurement, 41(2):024001, March 2020.[72] Carina Barbosa Pereira, Xinchi Yu, Tom Goos, Irwin Reiss, ThorstenOrlikowsky, Konrad Heimann, Boudewijn Venema, Vladimir Blazek,Steffen Leonhardt, and Daniel Teichmann. Noncontact Monitoring ofRespiratory Rate in Newborn Infants Using Thermal Imaging. IEEETransactions on Biomedical Engineering, 66(4):1105–1114, April 2019.[73] Souvik Phadikar, Nidul Sinha, and Rajdeep Ghosh. Automatic Eye-blink Artifact Removal From EEG Signal Using Wavelet TransformWith Heuristically Optimized Threshold. IEEE Journal of Biomedi-cal and Health Informatics, 25(2):475–484, February 2021.[74] Mozhde Firoozi Pouyani, Mansour Vali, and Mohammad AminGhasemi. Lung sound signal denoising using discrete wavelet trans-form and artificial neural network. Biomedical Signal Processing andControl, 72:103329, February 2022.[75] Qin Qin, Jianqing Li, Yinggao Yue, and Chengyu Liu. An Adap-tive and Time-Efficient ECG R-Peak Detection Algorithm. Journal ofHealthcare Engineering, 2017:1–14, 2017.[76] Hossein Rabbani, M. Parsa Mahjoob, E. Farahabadi, and A. Fara-habadi. R peak detection in electrocardiogram signal based on an op-timal combination of wavelet transform, hilbert transform, and adap-tive thresholding. Journal of Medical Signals and Sensors, 1(2):91–98,May 2011.[77] V. Naga Prudhvi Raj and T Venkateswarlu. ECG signal denoising us-ing Undecimated Wavelet Transform. In 2011 3rd International Con-ference on Electronics Computer Technology, pages 94–98, Kanyaku-mari, India, April 2011. IEEE.80Bibliography[78] Rogelio Ramos, Benjamin Valdez-Salas, Roumen Zlatev, MichaelSchorr Wiener, and Jose Mar´ıa Bastidas Rull. The Discrete WaveletTransform and Its Application for Noise Removal in Localized Cor-rosion Measurements. International Journal of Corrosion, 2017:1–7,2017.[79] Haythem Rehouma, Rita Noumeir, Philippe Jouvet, Wassim Bouachir,and Sandrine Essouri. A computer vision method for respiratory mon-itoring in intensive care environment using RGB-D cameras. In 2017Seventh International Conference on Image Processing Theory, Toolsand Applications (IPTA), pages 1–6, Montreal, QC, November 2017.IEEE.[80] Kristina E Rudd, Sarah Charlotte Johnson, Kareha M Agesa,Katya Anne Shackelford, Derrick Tsoi, Daniel Rhodes Kievlan,Danny V Colombara, Kevin S Ikuta, Niranjan Kissoon, Simon Fin-fer, Carolin Fleischmann-Struzek, Flavia R Machado, Konrad KReinhart, Kathryn Rowan, Christopher W Seymour, R Scott Wat-son, T Eoin West, Fatima Marinho, Simon I Hay, Rafael Lozano,Alan D Lopez, Derek C Angus, Christopher J L Murray, and MohsenNaghavi. Global, regional, and national sepsis incidence and mortal-ity, 1990–2017: analysis for the Global Burden of Disease Study. TheLancet, 395(10219):200–211, January 2020.[81] Shourjya Sanyal and Koushik Kumar Nundy. Algorithms for Monitor-ing Heart Rate and Respiratory Rate From the Video of a User’s Face.IEEE Journal of Translational Engineering in Health and Medicine,6:1–11, 2018.[82] School of Mechatronics Engg Universiti Malaysia Perlis, Malaysia,P Karthikeyan, M Murugappan, School of Mechatronics Engg Uni-versiti Malaysia Perlis, Malaysia, S Yaacob, and School of Mechatron-ics Engg Universiti Malaysia Perlis, Malaysia. ECG Signal Denois-ing Using Wavelet Thresholding Techniques in Human Stress Assess-ment. International Journal on Electrical Engineering and Informat-ics, 4(2):306–319, June 2012.[83] V. Seena and Jerrin Yomas. A review on feature extraction and denois-ing of ECG signal using wavelet transform. In 2014 2nd InternationalConference on Devices, Circuits and Systems (ICDCS), pages 1–6,Combiatore, India, March 2014. IEEE.81Bibliography[84] Tiina M. Seppanen, Janne Kananen, Kai Noponen, Olli-Pekka Alho,and Tapio Seppanen. Accurate measurement of respiratory airflowwaveforms using depth data. In 2015 37th Annual International Con-ference of the IEEE Engineering in Medicine and Biology Society(EMBC), pages 7857–7860, Milan, August 2015. IEEE.[85] Dangdang Shao, Yuting Yang, Chenbin Liu, Francis Tsow, Hui Yu,and Nongjian Tao. Noncontact Monitoring Breathing Pattern, Ex-halation Flow Rate and Pulse Transit Time. IEEE Transactions onBiomedical Engineering, 61(11):2760–2767, November 2014.[86] Farnaz Shikhsarmast, Tingting Lyu, Xiaolin Liang, Hao Zhang, andThomas Gulliver. Random-Noise Denoising and Clutter Eliminationof Human Respiration Movements Based on an Improved Time Win-dow Selection Algorithm Using Wavelet Transform. Sensors, 19(1):95,December 2018.[87] Atsushi Sofuni, Takayoshi Tsuchiya, and Takao Itoi. Ultrasound di-agnosis of pancreatic solid tumors. Journal of Medical Ultrasonics,47(3):359–376, July 2020.[88] Qingkun Song, Li Ma, Jiankun Cao, and Xiao Han. Image DenoisingBased on Mean Filter and Wavelet Transform. In 2015 4th Inter-national Conference on Advanced Information Technology and SensorApplication (AITS), pages 39–42, Harbin, China, August 2015. IEEE.[89] K. Storck, M. Karlsson, P. Ask, and D. Loyd. Heat transfer evaluationof the nasal thermistor technique. IEEE Transactions on BiomedicalEngineering, 43(12):1187–1191, December 1996.[90] Mallat Ste´phane. CHAPTER 7 - Wavelet Bases. In Mallat Ste´phane,editor, A Wavelet Tour of Signal Processing (Third Edition), pages263–376. Academic Press, Boston, third edition edition, 2009.[91] Yue Sun, Wenjin Wang, Xi Long, Mohammed Meftah, Tao Tan,Caifeng Shan, Ronald M. Aarts, and Peter H. N. De With. Respira-tion Monitoring for Premature Neonates in NICU. Applied Sciences,9(23):5246, December 2019.[92] Quoc-Viet Tran, Shun-Feng Su, and Van-Truong Nguyen. Pyra-midal Lucas—Kanade-Based Noncontact Breath Motion Detection.IEEE Transactions on Systems, Man, and Cybernetics: Systems,50(7):2659–2670, July 2020.82Bibliography[93] Maki Uenoyama, Takemi Matsui, Kouske Yamada, Satoshi Suzuki,Bonpei Takase, Shinya Suzuki, Masayuki Ishihara, and MitsuyukiKawakami. Non-contact respiratory monitoring system using a ceiling-attached microwave antenna. Medical & Biological Engineering &Computing, 44(9):835–840, September 2006.[94] Mark Van Gastel, Sander Stuijk, and Gerard De Haan. Robust respira-tion detection from remote photoplethysmography. Biomedical OpticsExpress, 7(12):4941, December 2016.[95] Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. Re-mote plethysmographic imaging using ambient light. Optics Express,16(26):21434, December 2008.[96] Dominik Vilimek, Jan Kubicek, Milos Golian, Rene Jaros, Radana Ka-hankova, Pavla Hanzlikova, Daniel Barvik, Alice Krestanova, MarekPenhaker, Martin Cerny, Ondrej Prokop, and Marek Buzga. Compar-ative analysis of wavelet transform filtering systems for noise reductionin ultrasound images. PLOS ONE, 17(7):e0270745, July 2022.[97] Mauricio Villarroel, Sitthichok Chaichulee, Joa˜o Jorge, Sara Davis,Gabrielle Green, Carlos Arteta, Andrew Zisserman, Kenny Mc-Cormick, Peter Watkinson, and Lionel Tarassenko. Non-contact phys-iological monitoring of preterm infants in the Neonatal Intensive CareUnit. npj Digital Medicine, 2(1):128, December 2019.[98] Mauricio Villarroel, Alessandro Guazzi, Joa˜o Jorge, Sara Davis, PeterWatkinson, Gabrielle Green, Asha Shenvi, Kenny McCormick, andLionel Tarassenko. Continuous non-contact vital sign monitoring inneonatal intensive care unit. Healthcare Technology Letters, 1(3):87–91, September 2014.[99] Zhaoyang Wang, Junjiang Zhu, Tianhong Yan, and Lulu Yang. A newmodified wavelet-based ECG denoising. Computer Assisted Surgery,24(sup1):174–183, October 2019.[100] Bing Wei, Xuan He, Chao Zhang, and Xiaopei Wu. Non-contact,synchronous dynamic measurement of respiratory rate and heart ratebased on dual sensitive regions. BioMedical Engineering OnLine,16(1):17, December 2017.[101] Weidong Zhou and J. Gotman. Removal of EMG and ECG artifactsfrom EEG based on wavelet transform and ICA. In The 26th Annual83BibliographyInternational Conference of the IEEE Engineering in Medicine andBiology Society, volume 3, pages 392–395, San Francisco, CA, USA,2004. IEEE.[102] D Wertheim, C Olden, E Savage, and P Seddon. Extracting respira-tory data from pulse oximeter plethysmogram traces in newborn in-fants. Archives of Disease in Childhood - Fetal and Neonatal Edition,94(4):F301–F303, July 2009.[103] Udaya Wijenayake and Soon-Yong Park. Real-Time External Respi-ratory Motion Measuring Technique Using an RGB-D Camera andPrincipal Component Analysis. Sensors, 17(8):1840, August 2017.[104] Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Fre´doDurand, and William Freeman. Eulerian video magnification for re-vealing subtle changes in the world. ACM Transactions on Graphics,31(4):1–8, August 2012.[105] Hongqiang Xu, Malikeh P. Ebrahim, Kareeb Hasan, Fatemeh Hey-dari, Paul Howley, and Mehmet Rasit Yuce. Accurate Heart Rateand Respiration Rate Detection Based on a Higher-Order HarmonicsPeak Selection Method Using Radar Non-Contact Sensors. Sensors,22(1):83, December 2021.[106] Meng-Chieh Yu, Huan Wu, Jia-Ling Liou, Ming-Sui Lee, and Yi-PingHung. Multiparameter Sleep Monitoring Using a Depth Camera. InJoaquim Gabriel, Jan Schier, Sabine Van Huffel, Emmanuel Conchon,Carlos Correia, Ana Fred, and Hugo Gamboa, editors, BiomedicalEngineering Systems and Technologies, volume 357, pages 311–325.Springer Berlin Heidelberg, Berlin, Heidelberg, 2013. Series Title:Communications in Computer and Information Science.[107] Unai Zalabarria, Eloy Irigoyen, Raquel Martinez, and Andrew Lowe.Online robust R-peaks detection in noisy electrocardiograms using anovel iterative smart processing algorithm. Applied Mathematics andComputation, 369:124839, March 2020.[108] Fang Zhao, Meng Li, Yi Qian, and Joe Z. Tsien. Remote Measure-ments of Heart and Respiration Rates for Telemedicine. PLoS ONE,8(10):e71384, October 2013.[109] T. Zikov, S. Bibian, G.A. Dumont, M. Huzmezan, and C.R. Ries. Awavelet based de-noising technique for ocular artifact correction of84the electroencephalogram. In Proceedings of the Second Joint 24thAnnual Conference and the Annual Fall Meeting of the BiomedicalEngineering Society] [Engineering in Medicine and Biology, volume 1,pages 98–105, Houston, TX, USA, 2002. IEEE.85Appendix AMATLAB CodeThe code was developed on MATLAB2002a by Shreya Verma. Kindly reachout to shreyav1@gmail.com for any future queries.Pre-requisites: Signal Processing ToolboxCode:% each .h5 file is a hdf5 file with the depth datafrom a capture.%% load info:dr = '/Users/';fname = '.h5';% Information about the file:info = h5info(fname);for x = 1: numel(info)ds = info.Datasets(x);fprintf('%s:\n\t size=', ds.Name);disp(ds.Dataspace.Size)% Interested in depth:if strcmp(ds.Name , 'depth ')depth_sz = ds.Dataspace.Size;endend%% load data in chunks:chunk_size = 1792;%30 frames per secondstart_frame = 18522;%end_frame = 18523;for x = start_frame:chunk_size:depth_sz (4) % thiswill miss some data at the end of the filechunk = h5read(fname , ... % filename'/depth ', ... % path inside the h5[1, 1, 1,start_frame], ... % start[inf , inf , inf , chunk_size] ... % end86Appendix A. MATLAB Code);% get rid of the extra dimension:chunk = squeeze(chunk);break % skip all the chunks so we can plot at theendend%%%% Visualizing the data% Method 1I = imagesc(chunk (:,:,1));% Method 2I = chunk (:,:,1);imshow(I)I_g = mat2gray(I);imshow(I_g)% Method 3imshow(I, [])%% Stacking Frames Togethermov =[];[a1,a2,a3]=size(chunk);cd(dr)figurefor i=1:a3I=chunk(:,:,i); %check before and after attributes(thresholding)ind=find(I >3000);I(ind)= 2500;I_ROI = I;I_ROIg = mat2gray(I_ROI);if isfloat(I_ROIg)I_ROIg = im2uint8(I_ROIg);end87Appendix A. MATLAB CodeColorImage = ind2rgb(I_ROIg ,jet);mov(i).cdata = ColorImage;endimshow(ColorImage)%% Selecting the ROI , using the colored image , butthe high dimensional data[num_manualROIs ,rect_BW_manual ]=manual_ROIselection_strategy(ColorImage);rect_BW1 = rect_BW_manual;for chunknum =1: chunk_sizeK=chunk(:,:, chunknum);I_ROI =K(floor(rect_BW1 (2)):floor(rect_BW1 (2))+floor(rect_BW1 (4)),floor(rect_BW1 (1)):floor(rect_BW1 (1))+floor(rect_BW1 (3)));%I_mask_ch =I(floor(rect_BW1 (2)):floor(rect_BW1(2))+floor(rect_BW1 (4)),floor(rect_BW1 (1)):floor(rect_BW1 (1))+floor(rect_BW1 (3)));Mov_pulse(chunknum)=sum(sum(I_ROI));endfigure;plot(Mov_pulse)xlabel('Frame(number)');ylabel('Amplitude(Intensity)');title('Original Data ');%% making video from frames[F_sav ,P_sav] = uiputfile('*.avi');cd(P_sav)v = VideoWriter(F_sav);v.FrameRate =30;writer.FrameRate =30;writer.MJ2BitDepth =8;open(v)for k = 1:size(mov ,2)% frame=mov(k).cdata;88Appendix A. MATLAB CodewriteVideo(v,im2uint8(mov(k).cdata));%mov(k).cdata=uint16(ColorImage);end% writeVideo(v,mov)I_g)close(v)%% Wavelet Transformation% xd = cmddenoise(Mov_pulse ,'db4 ',4);xd = wdenoise(Mov_pulse ,3);figure;plot(Mov_pulse ,'c','LineWidth ',1 )hold on;plot(xd,'Color ','b','LineWidth ' ,1)legend('Original Signal ','Denoised Signal ')axis tight;hold off;title('Wavelet Tranformation ')xlabel('Frame(Number)');ylabel('Amplitude(Intensity)');%% Automatic Peak Prominencex=1: size(xd ,2);[up,lo]= envelope(xd ,30,'peak ')plot(x,xd)hold onplot(x,up,'-',x,lo ,'-')hold offxlabel('Frame(Number)');ylabel('Amplitude(Intensity)');title('Denoised Data ');legend('Breat Pulse ' ,'Upper Envelope ', 'LowerEnvelope ');diff_amp=up-lo;MD=mean(diff_amp);MinPeakProminence =0.05* MD;89Appendix A. MATLAB Codeframe_rate = 30;total_points = size(mov ,2);time_interval = 1 / frame_rate;% Define the x values for the entire dataseta = (0: total_points -1) * time_interval;% Load your Mov_pulse data herey = xd;findpeaks(xd,a,'MinPeakProminence ',MinPeakProminence,'MinPeakDistance ' ,0.5);[peaks ,peak_indices] = findpeaks(xd,a,'MinPeakProminence ',MinPeakProminence ,'MinPeakDistance ' ,0.5)ylabel('Amplitude (Intensity)');title('Peak Detection ')xlabel('Time (seconds) ');text(peak_indices +.02,peaks ,num2str ((1: numel(peaks))'));%% Calculating Breaths per Minuteduration=size(mov ,2)/frame_rate;% video length insecs% RR2 =(60/ duration)*RR1% RRmeanfinal_Sm= mean(RBpm_total_smoothedppgi);RRmeanfinal_Wm3int=peaks;RRmeanfinal_Wm3int_BPM =(60/ duration)*RRmeanfinal_Wm3int;disp(RRmeanfinal_Wm3int_BPM);% RRmeanfinal_Wuniversal= mean(RBpm_total_universal);%% Plot the original data and detected peaksfigure;subplot(3, 1, 1);90Appendix A. MATLAB Codeplot(Mov_pulse);xlabel('Frame number ');ylabel('Amplitude(Intensity)');title('RV Signal Original Data ');grid on;subplot(3, 1, 2);plot(a, y);xlabel('Time (seconds)');ylabel('Amplitude(Intensity)');title('RV Signal Denoised Data ');grid on;subplot(3, 1, 3);plot(a, y, 'b', peak_indices , peaks , 'ro');xlabel('Time (seconds)');ylabel('Amplitude(Intensity)');title('Respiratory Rate Estimation ');legend('Original Data ', 'Detected Peaks ');grid on;%Manual ROI Strategy%function [num1 ,rect_BW ]=manual_ROIselection_strategy(I_g)function[num1 ,rect_BW ]= manual_ROIselection_strategy(ColorImage)%function [num1 ,rect_BW ]=manual_ROIselection_strategy(I)%% ROI selection%imshow(I_g)imshow(ColorImage)%% manual ROI selectionnum1 = input('Enter number of ROIs: ');% selectingnumber of ROIs by user%% step 1- Rect selectionrect_BW =[];for counter_Roi =1: num1counter_Roi ;91Appendix A. MATLAB Code%I_g=rgb2gray(I_c);uiwait(msgbox('please select a mask that coverschest area including movement '))%imshow(I_g)imshow(ColorImage)rect_BW1= getrect;rect_BW1=floor(rect_BW1);rect_BW (:,:, counter_Roi)=rect_BW1;%mask=false(size(I_g));mask=false(size(ColorImage));mask(rect_BW1 (2):rect_BW1 (2)+rect_BW1 (4),rect_BW1 (1):rect_BW1 (1)+rect_BW1 (3)) = true;Frame1_mask=I_g(floor(rect_BW1 (2)):floor(rect_BW1 (2))+floor(rect_BW1 (4)),floor(rect_BW1 (1)):floor(rect_BW1 (1))+floor(rect_BW1 (3)));if Frame1_mask ==1figure%imshow(I_g)imshow(ColorImage)hold onvisboundaries(mask ,'Color ','r');imshow(ColorImage)endend% Intraclass correlationfunction [r, LB, UB, F, df1 , df2 , p] = ICC(M, type ,alpha , r0)% [r, LB, UB, F, df1 , df2 , p] = ICC(M, type , alpha, r0)%% M is matrix of observations. Each row is an92Appendix A. MATLAB Codeobject of measurement and% each column is a judge or measurement.%% 'type ' is a string that can be one of the sixpossible codes for the desired% type of ICC:% '1-1': The degree of absolute agreementamong measurements made on% randomly seleted objects. It estimates thecorrelation of any two% measurements.% '1-k': The degree of absolute agreement ofmeasurements that are% averages of k independent measurements onrandomly selected% objects.% 'C-1': case 2: The degree of consistencyamong measurements. Also known% as norm -referenced reliability and asWiner 's adjustment for% anchor points. case 3: The degree ofconsistency among measurements maded under% the fixed levels of the column factor.This ICC estimates the% corrlation of any two measurements , butwhen interaction is% present , it underestimates reliability.% 'C-k': case 2: The degree of consistency formeasurements that are% averages of k independent measurements onrandomly selected% onbjectgs. Known as Cronbach 's alpha inpsychometrics. case 3:% The degree of consistency for averages ofk independent% measures made under the fixed levels ofcolumn factor.% 'A-1': case 2: The degree of absoluteagreement among measurements. Also% known as criterion -referenced reliability.93Appendix A. MATLAB Codecase 3: The absolute% agreement of measurements made under thefixed levels of the column factor.% 'A-k': case 2: The degree of absoluteagreement for measurements that are% averages of k independent measurements onrandomly selected objects.% case 3: he degree of absolute agreementfor measurements that are% based on k independent measurements madedunder the fixed levels% of the column factor.%% ICC is the estimated intraclass correlation.LB and UB are upper% and lower bounds of the ICC with alpha levelof significance.%% In addition to estimation of ICC , ahypothesis test is performed% with the null hypothesis that ICC = r0. TheF value , degrees of% freedom and the corresponding p-value of thethis test are% reported.%% (c) Arash Salarian , 2008%% Reference: McGraw , K. O., Wong , S. P., "Forming Inferences About% Some Intraclass Correlation Coefficients",Psychological Methods ,% Vol. 1, No. 1, pp. 30-46, 1996%if nargin < 3alpha = .05;endif nargin < 494Appendix A. MATLAB Coder0 = 0;end[n, k] = size(M);SStotal = var(M(:)) *(n*k - 1);MSR = var(mean(M, 2)) * k;MSW = sum(var(M,0, 2)) / n;MSC = var(mean(M, 1)) * n;MSE = (SStotal - MSR *(n - 1) - MSC * (k -1))/ ((n -1) * (k - 1));switch typecase '1-1'[r, LB , UB , F, df1 , df2 , p] = ICC_case_1_1(MSR , MSE , MSC , MSW , alpha , r0 , n, k);case '1-k'[r, LB , UB , F, df1 , df2 , p] = ICC_case_1_k(MSR , MSE , MSC , MSW , alpha , r0 , n, k);case 'C-1'[r, LB , UB , F, df1 , df2 , p] = ICC_case_C_1(MSR , MSE , MSC , MSW , alpha , r0 , n, k);case 'C-k'[r, LB , UB , F, df1 , df2 , p] = ICC_case_C_k(MSR , MSE , MSC , MSW , alpha , r0 , n, k);case 'A-1'[r, LB , UB , F, df1 , df2 , p] = ICC_case_A_1(MSR , MSE , MSC , MSW , alpha , r0 , n, k);case 'A-k'[r, LB , UB , F, df1 , df2 , p] = ICC_case_A_k(MSR , MSE , MSC , MSW , alpha , r0 , n, k);end%----------------------------------------function [r, LB, UB, F, df1 , df2 , p] = ICC_case_1_1(MSR , MSE , MSC , MSW , alpha , r0 , n, k)r = (MSR - MSW) / (MSR + (k-1)*MSW);F = (MSR/MSW) * (1-r0)/(1+(k-1)*r0);95Appendix A. MATLAB Codedf1 = n-1;df2 = n*(k-1);p = 1-fcdf(F, df1 , df2);FL = (MSR/MSW) / finv(1-alpha/2, n-1, n*(k-1));FU = (MSR/MSW) * finv(1-alpha/2, n*(k-1), n-1);LB = (FL - 1) / (FL + (k-1));UB = (FU - 1) / (FU + (k-1));%----------------------------------------function [r, LB, UB, F, df1 , df2 , p] = ICC_case_1_k(MSR , MSE , MSC , MSW , alpha , r0 , n, k)r = (MSR - MSW) / MSR;F = (MSR/MSW) * (1-r0);df1 = n-1;df2 = n*(k-1);p = 1-fcdf(F, df1 , df2);FL = (MSR/MSW) / finv(1-alpha/2, n-1, n*(k-1));FU = (MSR/MSW) * finv(1-alpha/2, n*(k-1), n-1);LB = 1 - 1 / FL;UB = 1 - 1 / FU;%----------------------------------------function [r, LB, UB, F, df1 , df2 , p] = ICC_case_C_1(MSR , MSE , MSC , MSW , alpha , r0 , n, k)r = (MSR - MSE) / (MSR + (k-1)*MSE);F = (MSR/MSE) * (1-r0)/(1+(k-1)*r0);df1 = n - 1;df2 = (n-1)*(k-1);p = 1-fcdf(F, df1 , df2);FL = (MSR/MSE) / finv(1-alpha/2, n-1, (n-1)*(k-1));FU = (MSR/MSE) * finv(1-alpha/2, (n-1)*(k-1), n-1);LB = (FL - 1) / (FL + (k-1));96Appendix A. MATLAB CodeUB = (FU - 1) / (FU + (k-1));%----------------------------------------function [r, LB, UB, F, df1 , df2 , p] = ICC_case_C_k(MSR , MSE , MSC , MSW , alpha , r0 , n, k)r = (MSR - MSE) / MSR;F = (MSR/MSE) * (1-r0);df1 = n - 1;df2 = (n-1)*(k-1);p = 1-fcdf(F, df1 , df2);FL = (MSR/MSE) / finv(1-alpha/2, n-1, (n-1)*(k-1));FU = (MSR/MSE) * finv(1-alpha/2, (n-1)*(k-1), n-1);LB = 1 - 1 / FL;UB = 1 - 1 / FU;%----------------------------------------function [r, LB, UB, F, df1 , df2 , p] = ICC_case_A_1(MSR , MSE , MSC , MSW , alpha , r0 , n, k)r = (MSR - MSE) / (MSR + (k-1)*MSE + k*(MSC -MSE)/n);a = (k*r0) / (n*(1-r0));b = 1 + (k*r0*(n-1))/(n*(1-r0));F = MSR / (a*MSC + b*MSE);%df2 = (a*MSC + b*MSE)^2/((a*MSC)^2/(k-1) + (b*MSE)^2/((n-1)*(k-1)));a = k*r/(n*(1-r));b = 1+k*r*(n-1)/(n*(1-r));v = (a*MSC + b*MSE)^2/((a*MSC)^2/(k-1) + (b*MSE)^2/((n-1)*(k-1)));df1 = n - 1;df2 = v;p = 1-fcdf(F, df1 , df2);Fs = finv(1-alpha/2, n-1, v);LB = n*(MSR - Fs*MSE)/(Fs*(k*MSC + (k*n - k - n)*MSE97Appendix A. MATLAB Code) + n*MSR);Fs = finv(1-alpha/2, v, n-1);UB = n*(Fs*MSR -MSE)/(k*MSC + (k*n - k - n)*MSE + n*Fs*MSR);%----------------------------------------function [r, LB, UB, F, df1 , df2 , p] = ICC_case_A_k(MSR , MSE , MSC , MSW , alpha , r0 , n, k)r = (MSR - MSE) / (MSR + (MSC -MSE)/n);c = r0/(n*(1-r0));d = 1 + (r0*(n-1))/(n*(1-r0));F = MSR / (c*MSC + d*MSE);%df2 = (c*MSC + d*MSE)^2/((c*MSC)^2/(k-1) + (d*MSE)^2/((n-1)*(k-1)));a = k*r/(n*(1-r));b = 1+k*r*(n-1)/(n*(1-r));v = (a*MSC + b*MSE)^2/((a*MSC)^2/(k-1) + (b*MSE)^2/((n-1)*(k-1)));df1 = n - 1;df2 = v;p = 1-fcdf(F, df1 , df2);Fs = finv(1-alpha/2, n-1, v);LB = n*(MSR - Fs*MSE)/(Fs*(MSC -MSE) + n*MSR);Fs = finv(1-alpha/2, v, n-1);UB = n*(Fs*MSR - MSE)/(MSC - MSE + n*Fs*MSR);98
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Dumont, G. (Guy), 1951-
One breath at a time : a feasibility study
Text