Applied Science, Faculty of
Mechanical Engineering, Department of
DSpace
UBCV
Ranjbaran, Parinaz
Master of Applied Science - MASc
University of British Columbia
In orthopaedic trauma surgery, using the C-arm fluoroscope to identify the relative position of surgical tools and anatomical structures adds to surgical time and exposes the patient and surgical team to radiation. We hypothesize that overlaying the most recently acquired fluoroscopic image on a video stream showing surgical tool positions could significantly decrease radiation and potentially improve workflow efficiency. Our Depth Camera Augmented Fluoroscopy (DeCAF) system uses a 3D depth camera affixed to the image intensifier tube to create an augmented reality view that overlays a recently acquired X-ray image onto a live video feed to allow the surgeon to more intuitively understand the relationship between the patient’s surface anatomy, surgical tools in the operating field, and interior anatomical structures. To evaluate the potential value of DeCAF, we simulated proximal tibia fixation using 40 Synbone models embedded in a foam model representing surrounding soft tissues. Two surgeons inserted three screws under the tibial plateau in each model, in half of the cases using the conventional fluoroscopic method, and in the other half using the DeCAF system with no imaging taken after the initial shot, except to confirm final placement. We measured the number of X-rays taken in the conventional procedure, the time required for each step, and the depth of the exit points of the screws. The DeCAF overlay error (distance between corresponding points on the X-ray and video images) averaged 1.3±0.2mm within ±50 mm of the calibration plane. The average number of X-ray shots using the conventional method by each surgeon were 39.2 and 21.5. Overall, screws inserted with DeCAF tended to exit ~2mm deeper than with the conventional method but were generally still within or close to the normally targeted depth zone. The overlay error is consistent with accuracies reported in related studies (Navab 2010, Habert 2015) and is likely acceptable for clinical applications. Despite acquiring only one X-ray of the proximal tibia, the surgeon was able to appropriately place the screws without breaching and in equal or less time compared to the conventional method. These results support plans to deploy and test DeCAF in live surgical scenarios.
REDUCING RADIATION DOSE IN ORTHOPAEDIC TRAUMA SURGERY USING DEPTH CAMERA AUGMENTED FLUOROSCOPY (DECAF) by Parinaz Ranjbaran B.Sc., Sharif University of Technology, 2020 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Mechanical Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) July 2024 © Parinaz Ranjbaran 2024 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled: ______________________________________________________________________________ submitted by in partial fulfilment of the requirements for the degree of in Examining Committee: Supervisor Co-supervisor Supervisory Committee Member Supervisory Committee Member Reducing radiation dose in orthopaedic trauma surgery using Depth Camera AugmentedFluoroscopy (DeCAF)Parinaz RanjbaranMaster of Applied ScienceMechanical EngineeringAntony Hodgson, Professor, Mechanical Engineering, UBCPierre Guy, Professor, Orthopaedics, UBCIlker Hacihaliloglu, Assistant Professor, Radiology, UBCThomas Oxland, Professor, Mechanical Engineering, UBCiii Abstract In orthopaedic trauma surgery, using the C-arm fluoroscope to identify the relative position of surgical tools and anatomical structures adds to surgical time and exposes the patient and surgical team to radiation. We hypothesize that overlaying the most recently acquired fluoroscopic image on a video stream showing surgical tool positions could significantly decrease radiation and potentially improve workflow efficiency. Our Depth Camera Augmented Fluoroscopy (DeCAF) system uses a 3D depth camera affixed to the image intensifier tube to create an augmented reality view that overlays a recently acquired X-ray image onto a live video feed to allow the surgeon to more intuitively understand the relationship between the patient’s surface anatomy, surgical tools in the operating field, and interior anatomical structures. To evaluate the potential value of DeCAF, we simulated proximal tibia fixation using 40 Synbone models embedded in a foam model representing surrounding soft tissues. Two surgeons inserted three screws under the tibial plateau in each model, in half of the cases using the conventional fluoroscopic method, and in the other half using the DeCAF system with no imaging taken after the initial shot, except to confirm final placement. We measured the number of X-rays taken in the conventional procedure, the time required for each step, and the depth of the exit points of the screws. The DeCAF overlay error (distance between corresponding points on the X-ray and video images) averaged 1.3±0.2mm within ±50 mm of the calibration plane. The average number of X-ray shots using the conventional method by each surgeon were 39.2 and 21.5. Overall, screws inserted with DeCAF tended to exit ~2mm deeper than with the conventional method but were generally still within or close to the normally targeted depth zone. iv The overlay error is consistent with accuracies reported in related studies (Navab 2010, Habert 2015) and is likely acceptable for clinical applications. Despite acquiring only one X-ray of the proximal tibia, the surgeon was able to appropriately place the screws without breaching and in equal or less time compared to the conventional method. These results support plans to deploy and test DeCAF in live surgical scenarios. v Lay Summary Mobile C-arm X-ray machines are commonly used in orthopaedic trauma surgeries to visualize internal anatomy and surgical tools. However, the use of scouting images to aid surgical positioning during these procedures can prolong operating time and increase radiation exposure. Our Depth Camera Augmented Fluoroscopy (DeCAF) device is designed to reduce the use of fluoroscopy by overlaying the X-ray images onto a live video of the patient’s surface anatomy. In this study, we present the hardware and software development process and demonstrate in a simulated operating room (OR) environment that the DeCAF system has clinically acceptable overlay error (1.3 ± 0.2 mm) and allowed surgeons to substantially reduse of X-rays while fixing proximal tibial plates in acceptable positions without significantly changing the time required (p = 0.16). These results justify proceeding to live clinical evaluations. vi Preface This thesis is an original piece of work by me, Parinaz Ranjbaran. Hardware and algorithm development and experimental evaluations were carried out by me with guidance from my co-supervisors, Dr. Antony Hodgson and my clinical co-supervisor, Dr. Pierre Guy. Dr. Pierre Guy provided guidance regarding clinical applications and experimental scenarios. Chapter 1: The original concept of using a depth camera augmented fluoroscopy for creating X-ray overlays was initially proposed by Matthew Hickey as a project published in CAOS 2019 under the supervision of Dr. Antony Hodgson and Dr. Pierre Guy. I was responsible for creating the flowcharts and diagrams describing the previous workflow. All figures in this chapter that were obtained from copyrighted sources have been reproduced in accordance with “Fair Use” provisions. Chapter 2: I redesigned the software implementing the computational and operational pipeline, designed a user interface, and the refined the components including those used to mount the camera and the pointer tool. The mounting parts were fabricated by me using the 3D printer available in the Centre for Aging Smart (CASMART). I performed the technical study to measure overlay error at the surgical suite at CASMART. In order to get approval for performing studies in the operating room, I navigated the operational approval process from Vancouver Coastal Health Research Institute (VCHRI). I successfully obtained approvals from the Infection Prevention and Control (IPAC) and an inspection from the Biomedical Engineering department for the safety of electrical components and camera mount. Additionally, I prepared the necessary documentation on potential risks as requested by the radiology department and created a user manual for the system. The vii operational approval is pending ethics approval, and we initially submitted an ethics application (H19-02861) on June 16, 2023 to request permission to perform a clinical study in future. However, the work anticipated in this application does not form part of this thesis. Chapter 3: The simulated lab study protocol preparation and coordination were done by me in consultation with Dr. Antony Hodgson and Dr. Pierre Guy and was conducted by two participating surgeons, Dr. Pierre Guy and Dr. David Stockton at the Surgical suite at CASMART. I performed all analysis and data collection. This part of the study has been presented in the following conferences (links unavailable): • Canadian Orthopaedic Association (COA) 2024 Annual Meeting • Computer Assisted Orthopaedic Surgery (CAOS) 2024 • Orthopaedic Trauma Association (OTA) 2024 Annual Meeting In addition to these conferences, I presented this research at the Designing for People Showcase in 2023 (poster), the Orthopaedic Research Day at UBC in 2023 (podium presentation) and 2024 (poster), and the Ideas With Impact event at UBC in 2024 (poster). The following awards were received for this research: • DFP CREATE Design Showcase 2023 Best Poster: Runner Up • Emerging Researchers Award – CAOS 2024 Best Technical Poster Presentation (First Place) viii Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ....................................................................................................................... viii List of Tables ................................................................................................................................xv List of Figures ............................................................................................................................. xvi List of Abbreviations ............................................................................................................... xxiv Acknowledgements .................................................................................................................. xxvi Chapter 1: Motivation and Background .....................................................................................1 1.1 Motivating Problem ........................................................................................................ 1 1.2 Previous Efforts .............................................................................................................. 5 1.2.1 Camera-Augmented C-arm (CamC) system ........................................................... 6 1.2.2 Mirror-based RGBD X-ray imaging system ........................................................... 7 1.2.3 Integrated C-arm with an RGB-D sensor ................................................................ 9 1.3 Previous Work: DeCAF (Depth Camera-Augmented Fluoroscopy) ............................ 11 1.3.1 Principles of Operation ......................................................................................... 11 1.3.2 Limitations ............................................................................................................ 17 ix 1.3.2.1 Hardware Components...................................................................................... 17 1.3.2.2 Intra-operative Use of the Calibration Disk ...................................................... 17 1.3.2.3 Simplified Transform Formulation ................................................................... 19 1.3.2.4 Position Estimation Using Four Independent ArUco Markers ........................ 21 1.3.2.5 Video Distortion and Quality ............................................................................ 21 1.3.2.6 Limited Control in the User Interface ............................................................... 22 1.4 Specific Research Objectives ........................................................................................ 24 1.5 Thesis Overview ........................................................................................................... 25 Chapter 2: Design and Development of depth camera augmented fluoroscopy (DeCAF) ...26 2.1 Components .................................................................................................................. 26 2.1.1 Hardware Design .................................................................................................. 27 2.1.1.1 Camera Assembly ............................................................................................. 28 2.1.1.1.1 Depth Camera ............................................................................................. 28 2.1.1.1.2 Camera Mount Assembly ........................................................................... 29 2.1.1.2 Pointer Tool ..................................................................................................... 29 2.1.1.3 Electrical Connections ...................................................................................... 31 2.2 Operation....................................................................................................................... 32 2.3 Operational Pipeline ...................................................................................................... 35 x 2.3.1 Overview of Operational Pipeline ........................................................................ 36 2.3.1.1 Formulate Transforms ....................................................................................... 39 2.3.1.2 X-ray Overlay Scale .......................................................................................... 41 2.3.1.3 X-ray Image Processing and Blending ............................................................. 44 2.3.1.4 Adjustment Model for Laser Aimer Misalignment .......................................... 45 2.3.1.5 Qualitative Demonstration of Overlay Functionality ....................................... 48 2.4 Pre- and Intra-operative Software Design ..................................................................... 50 2.4.1 Main Pipeline ........................................................................................................ 51 2.4.1.1 Locate Virtual Camera ...................................................................................... 57 2.4.1.2 Select Overlay Depth ........................................................................................ 57 2.4.1.3 Generate X-ray Overlay .................................................................................... 58 2.4.1.3.1 Generate Point-cloud from Virtual Camera View ...................................... 58 2.5 Pre-Deployment Calibration ......................................................................................... 64 2.5.1 Calibrate Camera Intrinsic Parameters ................................................................. 65 2.5.2 Adjust for Laser Aimer Misalignment .................................................................. 67 2.6 Experimental Evaluations ............................................................................................. 71 2.6.1 Requirements for X-ray Overlay .......................................................................... 72 2.6.1.1 Evaluating Clinically Acceptable Deviation Levels ......................................... 72 xi 2.6.1.2 Mesh Analysis for Assessing Overlay Error ..................................................... 74 2.6.2 Overlay Error ........................................................................................................ 75 2.6.3 Usability in Clinical Use ....................................................................................... 79 2.6.3.1 Safety in Clinical Use ....................................................................................... 79 2.6.3.2 Sterile Field Cover Compatibility ..................................................................... 81 2.7 Summary ....................................................................................................................... 83 Chapter 3: Performance in Simulated Target Procedures ......................................................84 3.1 Selecting Surgical Procedure for Assessing DeCAF’s Potential Benefits.................... 85 3.2 User Study ..................................................................................................................... 88 3.2.1 Study Design ......................................................................................................... 88 3.2.2 Anatomic Models .................................................................................................. 88 3.2.3 Experimental Setup ............................................................................................... 90 3.2.4 Data Acquisition and Analysis.............................................................................. 97 3.3 Results ........................................................................................................................... 99 3.3.1 Breach Count and Depth of Screw Placement ...................................................... 99 3.3.2 Radiation ............................................................................................................. 103 3.3.3 Procedure time .................................................................................................... 106 3.3.3.1 Times for Major Steps .................................................................................... 106 xii 3.3.3.2 Detailed Timing .............................................................................................. 107 3.3.3.2.1 Overall Procedure Time ............................................................................ 110 3.3.3.2.2 Monitoring X-rays vs Checking Overlay .................................................. 112 3.3.3.2.3 Time Differences for Other Surgical Actions ........................................... 113 3.4 Summary ..................................................................................................................... 113 Chapter 4: Discussion and Conclusions ...................................................................................115 4.1 Key Findings ............................................................................................................... 115 4.1.1 Hardware Components........................................................................................ 115 4.1.2 Technical Verification: Overlay Error ................................................................ 116 4.1.3 Simulated Lab Study ........................................................................................... 117 4.1.3.1 Radiation ......................................................................................................... 117 4.1.3.2 Breach Count and Depth of Screw Placement ................................................ 118 4.1.3.3 Procedure time ................................................................................................ 119 4.2 Limitations .................................................................................................................. 121 4.2.1 Hardware Limitations ......................................................................................... 121 4.2.1.1 Laser Aimer Misalignment ............................................................................. 121 4.2.1.2 Use of the RGBD Camera ............................................................................... 122 4.2.1.3 Design for a Specific Mobile C-arm ............................................................... 122 xiii 4.2.2 Software Limitations ........................................................................................... 123 4.2.2.1 Overlay Error .................................................................................................. 123 4.2.2.1.1 Post-hoc Calibration Procedure Laser Aimer Misalignment .................... 124 4.2.2.1.2 Non-idealities in C-arm Structure ............................................................. 124 4.2.2.1.3 Distortions in the X-ray Image ................................................................. 125 4.2.2.1.4 Position Change of the Anatomy ............................................................. 125 4.2.2.2 Computational Time ....................................................................................... 125 4.2.2.3 Lack of Error Handling for Hardware Disconnections ................................... 126 4.2.2.4 Dependence on Lighting Conditions .............................................................. 126 4.2.3 Simulated Lab Study Limitations ....................................................................... 128 4.2.3.1 Use of Single Initial X-ray When Using DeCAF ........................................... 128 4.2.3.2 Potential Learning Effect ................................................................................ 128 4.2.3.3 Limited Number of Participants...................................................................... 129 4.2.3.4 Specific Simulated Surgical Application ........................................................ 129 4.3 Recommendations for Future Work ............................................................................ 130 4.3.1 Improvements to DeCAF .................................................................................... 130 4.3.2 Combination with Base Tracking System ........................................................... 131 4.4 Summary of Thesis Contributions .............................................................................. 131 xiv 4.5 Conclusions ................................................................................................................. 132 References ...................................................................................................................................133 Appendices ..................................................................................................................................142 Appendix A Calibration of the Virtual Camera’s Location in the Earlier Implementation of DeCAF .................................................................................................................................... 142 Appendix B Point-cloud Reprojection Pipeline in the Earlier Implementation of DeCAF .... 144 Appendix C ArUco Board Position Estimation ...................................................................... 145 Appendix D chhm_trackers .................................................................................................... 148 Appendix E display_pc ........................................................................................................... 151 Appendix F Open Reduction Internal Fixation (ORIF) .......................................................... 154 xv List of Tables Table 2.1 Technical specifications of Intel RealSense Depth Camera D435i (© Intel Corporation)....................................................................................................................................................... 28 Table 2.2 Format for storing machine calibration parameters for various C-arm models and different orientations of the camera around the image intensifier ................................................ 70 Table 3.1 Frequency and type of trauma operations requiring fluoroscopy use including median dose area product, screening time, and number of fluoroscopy images taken for trauma cases where n > 9. ................................................................................................................................. 85 Table 3.2 Effects of Surgeon, Method, and Screw Type on Exit Hole Depth in Three-Way ANOVA Analysis (significant findings bolded and starred) ...................................................... 101 Table 3.3 Effects of Surgeon and Screw Type on number of X-rays per step in Two-Way ANOVA Analysis ....................................................................................................................... 105 Table 3.4 Effects of Surgeon and Method on the Total Time in Two-Way ANOVA Analysis (significant findings bolded and starred) .................................................................................... 111 Table 3.5 Two-Way ANOVA Analysis examining the effects of Surgeon and Method on Combined X-ray or Overlay Monitoring Time (significant findings bolded and starred) ......... 112 xvi List of Figures Figure 1.1 Siemens Arcadis Orbic 3D C-Arm ................................................................................ 5 Figure 1.2 Video camera and the double mirror construction (Navab et al. 2010) ........................ 7 Figure 1.3 Setup of RGBDX with a Asus Xtion Live Pro and a mirror mounted on a C-arm (Habert, Gardiazabal, et al. 2015) ................................................................................................... 8 Figure 1.4 Setup with 2 Kinects attached on the C-arm gantry (Habert, Meng, et al. 2015) ......... 9 Figure 1.5 Overlay process on reconstructed model of the scene: A) 3D reconstructed model of the scene, B) The corresponding X-ray image, and C) The X-ray overlay (Hosseinian et al. 2020)....................................................................................................................................................... 10 Figure 1.6 A) Overview of DeCAF system and coordinate frames of the depth camera (D) , virtual camera (V), and calibration disk (C), B) camera setup when used in a mock surgical scene (Hickey et al. 2019) ....................................................................................................................... 12 Figure 1.7 A) Calibration paper with four ArUCo markers, where the cross sign represents the average coordinate position of all markers and is aligned with the laser aimer crosshair, B) Calibration disk used to identify the X-ray warp transform step, featuring colored markers with underlying bearing balls, and two slots allowing for alignment with the laser aimer, C) Camera mount including two clipped casings and a mount – The assembly is installed onto the C-arm with a silicon band ........................................................................................................................ 14 Figure 1.8 Three steps in the DeCAF pipeline: A) Calibrate virtual camera using the calibration paper, B) Generate the virtual camera view; capture X-ray; and select the corresponding points xvii on the X-ray and the video, C) Generate X-ray overlay by warping the X-ray image onto the video .............................................................................................................................................. 16 Figure 1.9 Image dimensions of AOI on the X-ray and video at A) close depths, and B) far depths relative to the image intensifier. In this representation 𝒘𝒂 is the AOI size, f’ is the distance between virtual camera and image plane, d and D are the distance from the target plane and X-ray tube to the image intensifier, respectively and images created on the video and X-ray are of 𝑷𝒂𝒗 and 𝑷𝒂𝒙 pixels in the 𝒚𝒄 direction, respectively - Note that this is because the X-ray emitter and the depth camera are positioned on opposite sides of the patient. ............................. 18 Figure 1.10 A) Nominal position of the camera assembly relative to the image intensifier; alignment of the C-arm axis and virtual camera axis, B) Real position of the camera assembly relative to the image intensifier; offset between the center of the virtual camera view and the laser aimer crosshair (C-arm axis) ................................................................................................ 20 Figure 1.11 Initial pipeline of the DeCAF system and main user controls that navigate the steps in the pipeline including A) Calibrate virtual camera using the calibration paper, B) Generate the virtual camera view; capture X-ray; and select the corresponding points on the X-ray and the video, C) Generate X-ray overlay ................................................................................................ 23 Figure 2.1 Overview of DeCAF system main components: A) depth camera assembly, B) pointer tool, C) computer and monitor ...................................................................................................... 27 Figure 2.2 Camera Assembly Components: A) Camera mount, B) Upper Casing (connects to the lower camera with snap-fit joint); C) Lower casing (where the depth camera is placed) ............ 29 xviii Figure 2.3 Calibration tool: A) Principal axes B) Pointer tool alignment with C-arm laser aimer....................................................................................................................................................... 30 Figure 2.4 Overview of system components and electrical connections ...................................... 32 Figure 2.5 Overview of system operation including three steps: electrical setup, mechanical setup, and software setup .............................................................................................................. 33 Figure 2.6. A) Camera mount assembly with plastic flattened around the lenses using a disposeable rubber band, B) Expected use of the system in the OR when the C-arm is draped .. 34 Figure 2.7 Instructions for cleaning, disinfection, and the products to be used for reprocessing the device between patient uses .......................................................................................................... 35 Figure 2.8 Software workflow for pre-deployment calibration (top) and pre- and intra-operative processes (bottom) – Note that, intra-operatively you can go back and forth between the two steps in the intra-operative use. ..................................................................................................... 37 Figure 2.9 The software carries out three main steps: A) Calibrate virtual camera B) Find indicated anatomical distance to image intensifier C) Project the point-cloud from C-arm view, capture X-ray and generate overlay. ............................................................................................. 38 Figure 2.10 A) Transforms between depth camera, pointer tool and virtual camera, B) 3D frames of the depth camera (D), pointer tool (C), and virtual camera (V) ............................................... 39 Figure 2.11 Overview of the transforms and steps to calculate each transform ........................... 40 Figure 2.12 Illustration of scaling relationships for X-ray and camera images: 𝒘𝒂: AOI size, f’: distance between virtual camera and image plane, and s(d) is the scaling factor ......................... 42 xix Figure 2.13 Overview of the overlay process – The captured X-ray image is scaled by s(d), masked, and padded with zeroes to match with the dimensions of the video image before alpha blending......................................................................................................................................... 45 Figure 2.14 Misalignment between image center and corresponding X-ray of a reference object: A) Sample object with centre aligned with the laser crosshair, B) Corresponding X-ray, C) misalignment from nominal cross point (emitter location indicated by small black dot), D) misalignment observed from another view ................................................................................... 46 Figure 2.15 Laser cross hair forming and the misalignment observed in: A) Physical world, and B) X-ray image of the sample object which is centred with the laser crosshair - Note that the misalignment between the X-ray image center and the centre point of the sample object in the X-ray, denoted as 𝒄𝒙 and 𝒄𝒚, varies depending on the distance between the plane of interest and the image intensifier ............................................................................................................................ 47 Figure 2.16 Examples of the X-ray overlays obtained in anatomical models: A) knee in a rolled lateral position, B) knee in an anteroposterior (AP) position, C) spine in a prone position, and D) ankle in a lateral position .............................................................................................................. 49 Figure 2.17 X-ray overlay of the proximal tibia with a plate secured by a K-wire ...................... 50 Figure 2.18 Software packages including main libraries and software development kits ............ 51 Figure 2.19 DeCAF Software Structure: A) App: user interface class which runs in the main thread, B) chhm_trackers for Marker detection which is called during virtual camera calibration and depth measurement step, C) display_pc which generates virtual camera frame and overlays and scales the X-ray, D) main functions in App ........................................................................... 54 xx Figure 2.20 DeCAF GUI which walks users through the three main pre- and intra-operative pipeline .......................................................................................................................................... 56 Figure 2.21 A) Pipeline for generating point-cloud and reprojecting in virtual camera view, B) Schematic of views and point-cloud reprojection ......................................................................... 59 Figure 2.22 Main operations in display_pc class based on RealSense SDK: A) inputs, B) generate point-cloud, C) reproject to virtual camera view, D) find texture coordinates, E) perform uv mapping, and F) output. Note: formulas derived from (Dorodnicov et al. 2023) .................... 60 Figure 2.23 ChArUco Board used for camera intrinsic calibration .............................................. 66 Figure 2.24 Laser aimer calibration protocol ................................................................................ 68 Figure 2.25 DeCAF GUI displaying initial system settings, including C-arm model selection, camera positioning, activation of developer's mode, and the choice to upload calibration data. . 71 Figure 2.26 Varying Deviations between X-ray and the RGB image: A) Horizontal offset, B) Angle offset ................................................................................................................................... 73 Figure 2.27 Test object with visible markers in video and X-ray to evaluate X-ray overlay ....... 74 Figure 2.28 X-ray overlay of the test object at calibration depth: A) Projected image from top view, B) X-ray overlay ................................................................................................................. 75 Figure 2.29 Vector field plot for distortions of visible marker points in X-ray image at the: A) closer <400 mm (363 mm), B) calibration plane (443 mm), and C) farther >500mm (517mm) distances. ....................................................................................................................................... 77 xxi Figure 2.30 Overlay Error as a function of distance to the virtual camera. The virtual camera calibration was done at a distance of ~440 mm. The dotted red lines represent the 95% confidence interval bounds around the mean value (green dashed line) across the range from 400 to 500 mm. .................................................................................................................................... 78 Figure 2.31. Use of the C-arm equipped with DeCAF at different orientations – The likelihood of the camera hitting the table increases with the C-arm's orbital angle. .......................................... 80 Figure 2.32 Comparison of the video and point-cloud quality in the absence (left) and presence (right) of the transparent surgical drape. Note some minor blurriness in the draped images. The undetected marker is marked with a red circle in the detected markers image with the drape present. The bottom right image shows one section (upper left; white arrow) where the overlay position is significantly in error. ................................................................................................... 82 Figure 2.33 Pointer tool edges in the virtual camera view, shown in white (absence of transparent surgical drape) and green (presence of transparent surgical drape). .......................... 83 Figure 3.1 Surgical Steps in Open Reduction Internal Fixation (ORIF) of the Proximal Tibia ... 87 Figure 3.2 A) AxSOS proximal tibial plate (Stryker) and B) Tibia Model: Cortical / hard cancellous bone, 8 mm Diameter Canal, and 387 mm Overall Length (Synbone) ...................... 89 Figure 3.3 Foam model used to wrap the tibial models used in the study and associated anatomical aspects of the tibia ...................................................................................................... 90 Figure 3.4 Surgical Tools used for simulated proximal tibia plate fixation procedure including power tool (A) used for drilling and screwing, K-wire (B), drill bit (C), sleeve (D), and screwdriver attachments (E), cortical screws (G), fixation plate (H) . ........................................ 91 xxii Figure 3.5 Soft Tissue Placement and Alignment with the C-arm’s axis on the Surgical Table .. 92 Figure 3.6 Workflow in plate insertion and fixation step in proximal tibia plate fixation: A) K-wire insertion, B) Drilling, and C) Screwing. Note that drilling and screwing process is repeated three times – once each for the middle, anterior, and posterior screws. ....................................... 93 Figure 3.7 Overlay shown to surgeon while positioning the sleeve and the drill ......................... 95 Figure 3.8 Overview of key steps in the procedure ...................................................................... 96 Figure 3.9 Hole locations labeled as M (Middle), A (Anterior), and P (Posterior) on the entry side (lateral condyle) and the exit side (medial condyle) of a synthetic bone specimen. ............. 98 Figure 3.10 Exit hole depths using conventional (left) and DeCAF (right) approaches done by the MES and LES. Points are sorted by the order of completed cases from left to right. ................ 100 Figure 3.11 X-Ray usage throughout procedure stages using the conventional method. Note that the surgeon places the middle screw first. .................................................................................. 104 Figure 3.12 Number of X-rays required for each screw positioning step when performing conventional cases ...................................................................................................................... 104 Figure 3.13 Time required for each major step in proximal tibia plate fixation using the conventional approach (left bar within each pairing) and the DeCAF approach (right bar) for the Most and Least Experienced Surgeons (left and right subplots, respectively). .......................... 107 Figure 3.14 Sequential Timeline of Actions in the Workflow using the Conventional or DeCAF Methods and performed by the MES or LES - K-wire Insertion (K), Middle Screw Insertion (M), Anterior Screw Insertion (A), and Posterior Screw Insertion (P) ....................................... 108 xxiii Figure 3.15 Aggregated time data per category for Conventional method (shown on top) and DeCAF method performed by the MES and LES ...................................................................... 109 Figure 4.1 A) Misalignment of X-ray, B) Misalignment of laser aimer, C) Table not parallel to image intensifier plane ................................................................................................................ 123 Figure 4.2 Alignment issue in rendering – A) In the absence of an object underneath the tool, B) When the hand is underneath the tool. ........................................................................................ 127 xxiv List of Abbreviations ALARA As Low as Reasonably Achievable AOI Anatomical Object of Interest ANOVA Analysis of Variance AP Anteroposterior ArUco Augmented Reality University of Cordoba CamC Camera-Augmented C-arm CAOS Computer-assisted Orthopaedic Surgery CASMART Centre for Aging Smart COA Canadian Orthopaedic Association CPU Central Processing Unit DA THA Direct Anterior Total Hip Arthroplasty DeCAF Depth Camera Augmented Fluoroscopy DOF Degree of Freedom DAP Dose-Area Product IPAC Infection Prevention and Control GPU Graphical Processing Unit GUI Graphical User Interface xxv Gy Gray LES Less Experienced Surgeon MES More Experienced Surgeon OR Operating Room ORIF Open Reduction Internal Fixation OTA Orthopaedic Trauma Association PLA Polylactic Acid RadTech Radiology Technologist RGB Red, Green, and Blue RGBD Red, Green, Blue, and Depth TOST Two One-Sided t-Test TKA Total knee arthroplasty USB Universal serial bus xxvi Acknowledgements I would like to start by thanking my supervisor, Dr. Antony Hodgson. Your guidance has taught me how to approach research effectively, always framing a problem with a research question while keeping the global objective in mind. I am deeply grateful for everything you have taught me throughout my degree. This includes not only conducting research and guiding me through the design stages but also how to be a professional in the medical device industry and present my work effectively. I also want to express my sincere appreciation to my co-supervisor, Dr. Pierre Guy. Your clinical insights and genuine interest have significantly helped me become a better engineer and researcher. You made this journey exciting, and I am grateful for all the support you have given me. Special thanks to Dr. David Stockton for his collaboration and the time he dedicated to this project. I deeply appreciate all the clinical insights and feedback you provided and your participation in my tests. I look forward to the day when DeCAF is used by both of you! Thank you to my peers in the Surgical Technologies Lab for their feedback and support throughout my degree. To my family, Zari, Hadi, Parisa, and Pouria, and my friends, thank you for your constant motivation and support in all aspects of my life and research throughout these years. I would like to acknowledge UBC Mechanical Engineering, and UBC Department of Orthopaedics for providing me with a supportive environment for my research. Special thanks to the Centre for Aging SMART at Vancouver Coastal Health, the Institute for Computing, Information and Cognitive Systems (ICICS), and the Designing for People Research Cluster for their support and resources throughout this journey. 1 Chapter 1: Motivation and Background 1.1 Motivating Problem Mobile C-arm X-ray machines are commonly used in orthopaedic surgeries to visualize internal anatomy (see Figure 1.1). These machines are especially indispensable in procedures like Intramedullary Femur Nailing, where they are frequently employed to assess the precise placement of intramedullary nails within the femur. However, many images are typically acquired for identifying the relative position of surgical tools and anatomy, as well as for monitoring changes in that relative position. Standard C-arms do not display acquired X-ray images in accurate spatial positions, making it challenging for surgeons to interpret due to variability in positioning and anatomy of patients, limited visibility of internal anatomy, and influence of surgical instruments or implants obscuring the view (De Silva et al. 2018; Kausch et al. 2021). As reported in (Weldon et al. 2023), there is the potential for inaccuracies in limb alignment assessment due to fluoroscopic parallax, stemming from non-centered imaging. In a study by Carlson (2021), using intra-operative fluoroscopic pelvic images, an average combined distortion of 10.0mm (range 2.0-20.0mm) was observed in 74 cases of direct anterior total hip arthroplasty (DA THA). The "combined distortion" in this work refers to the sum of the vertical distortion measured at the left and right peripheries of a fluoroscopic image, where the lesser trochanters are typically visible when assessing leg lengths. This metric quantifies the potential error in leg length assessment if peripheral distortion is not accounted for. This significant distortion underscores the risk of error in comparing limb lengths. Furthermore, the use of scout images to assist in C-arm positioning during these procedures can lead to prolonged operating times, ranging from 44 to 53 2 seconds per scout image, and increased radiation exposure ranging from of 3 to 12 scout images (Matthews et al. 2007). In a study focused on radiation exposure in trauma operating rooms spanning 18 months (Rashid et al. 2018), researchers established reference values for common trauma procedures involving fluoroscopy. Surgical procedures such as intramedullary nailing require capturing more than 200 images, and even routine procedures like neck of the femur surgeries require approximately 65 images, resulting in significant radiation exposure for patients and the surgical team. In particular, due to the significant use of fluoroscopy in trauma procedures, orthopaedic trauma surgeons are exposed to significantly higher doses of radiation compared with other orthopaedic subspecialties (Gausden et al. 2017). This heightened exposure is a concerning issue, as it has been linked to various adverse health outcomes within the orthopaedic community. While use of protective equipment (e.g., lead aprons) generally keeps radiation exposure within acceptable limits, there is evidence of orthopaedic surgeons experiencing significantly higher cancer rates than the general population. For instance, studies have reported a five-fold increase in the odds ratio of cancer development for orthopedic surgeons working in environments where best radiation safety practices were not consistently followed (Mastrangelo et al. 2005). Additionally, there have been reports of radiation-related damage to the fingers of Japanese orthopedic surgeons (Fujibayashi et al. 2021), as well as significantly decreased male offspring sex ratios amongst Japanese orthopaedic surgeons (Hijikata et al. 2021). And female orthopaedic surgeons in the US have cancer rates nearly twice as high and breast cancer rates nearly three times as high as those in the general population (Chou et al. 2022). These concerning findings support adopting the 3 precautionary approach of keeping radiation exposure As Low As Reasonably Achievable (ALARA) (Kaplan et al. 2016). According to (Salvia et al. 2011), the average fluoroscopy time across 80 orthopedic surgeries was 61 seconds per case. Specific procedures, such as bilateral proximal femoral epiphysiodesis (5.1 minutes) and femoral shaft fracture fixation using a locked intramedullary nail (3.3 minutes), required the highest fluoroscopy times. Furthermore, positioning the C-arm to a desired view required an average of 41 seconds, as reported in (De Silva et al. 2018). The complexity of the procedure can lead to additional time requirements, including the need for multiple images for precise positioning, adjusting the C-arm for optimal visualization, and ensuring that the images offer necessary guidance for the surgical team. These collective steps often contribute to prolonged surgery durations, thereby increasing overall operating time (Mahesh 2001) and cost. (Smith et al. 2022) found wide variability in estimated operating room costs per minute, ranging from $14 to $131, with an average cost of $46 per minute across the studies analyzed considering adjustment for inflation to 2022 dollars1. The large variability was due to variable inclusion of expenses like anesthesia fees, surgeon fees, and implant costs across different studies. For context, (Shippert 2005) reported average operating room charges of $62 per minute including OR fees and anesthesia. More recently, (Moody et al. 2020) found total per-minute costs of $16.21, excluding professional fees for anesthesiology and surgeons, as well as costs for materials and implants. Despite the variations in these estimated costs, using C-arm machines more efficiently can potentially yield both time and cost benefits in healthcare. For instance, saving approximately 5 1 Values in US Dollars 4 minutes of surgery time per procedure could translate to saving around 310 million dollars in North America, assuming a total of 1.5 million orthopaedic trauma fixation procedures per year (GobalData 2023), a cost of $46 per minute (Smith et al. 2022), and 90% of these procedures involving the use of a C-arm. Furthermore, even modest time savings in each procedure could potentially prevent the cancellation of the last surgery scheduled for the day. As reported in the literature (Caesar et al. 2014; Denis et al. 2022), approximately 40% of orthopaedic procedures were affected by cancellations, mainly due to the prioritization of more urgent orthopedic or non-orthopaedic surgical cases. These cancellations result in wasted costs associated with preparing the patient for surgery, only to have the procedure cancelled or postponed, which, according to (Chua et al. 2023), could range $1500-3390 per case excluding surgical set-up costs, and where between 25% and 86% of these are potentially avoidable (as reported in various settings). Therefore, implementing measures to avoid cancellations could result in savings on the order of 200 million dollars per year, assuming a 40% cancellation rate (assuming that the cancellation rate of orthopaedic trauma proedures is similar to that of all orthopaedic procedures), with 25% of those being avoidable, and a savings of $1500 per procedure. Given these findings, there is a clear need for research into optimizing orthopaedic procedures involving C-arms and reducing occupational exposure risks for medical staff. There are approximately 6,000 orthopaedic surgeons in the US who treat trauma cases (extrapolating from 5,083 orthopaedic trauma surgeons in 2012 and an increase rate of 1.5% as reported in (Sielatycki et al. 2016)), likely around 400 orthopaedic trauma surgeons in Canada (extrapolating from a total of 1,693 orthopaedic surgeons in Canada in 2019 (CMA Foundation 2019) and considering the 5 same ratio of orthopaedic trauma surgeons, ~6000, to total orthopaedic surgeons, ~27,000 (Definitive Healthcare 2024), as in the US) and some portion of ~221,000 radiology technologists in the US (US Bureau of Labor Statistics 2024) and ~27,000 radiology technologists in Canada (including major provinces and excluding certain territories) (Job Bank Canada 2024) who could potentially benefit from exposure-reducing technologies for C-arms. Figure 1.1 Siemens Arcadis Orbic 3D C-Arm 1.2 Previous Efforts Efforts have previously been made to address the challenge of connecting the information in X-ray images of hidden anatomy with the surgical scene. One such idea involves using X-ray overlays, which superimpose X-ray imagery onto the surgical field in order to provide better visualization and allow the surgeon to see the relative position of surgical tools in relation to the 6 underlying anatomy. In the following sections, we will describe the X-ray overlay systems developed over time. 1.2.1 Camera-Augmented C-arm (CamC) system The first example of such an overlay system is known as CamC (Navab et al. 1999) which uses a camera and double mirror structure placed at the source side to match the viewpoint of a camera and the X-ray as shown in Figure 1.2. The system calibration involves distortion correction, alignment of X-ray source and camera optical centre using a visual transparent bi-planar calibration phantom, and homography estimation for image overlay. This system offers a submillimetric overlay error (Navab et al. 2010), and a clinical trial demonstrated a 46% reduction in the number of X-ray shots compared to standard C-arms across 10 workflow steps in orthopaedic trauma procedures such as instrument guidance, control of progress of drilling, fracture realignment, etc (von der Heide et al. 2018). However, CamC has some important limitations. The device is used in an inverted configuration, leading to 60-65% more radiation scatter than the normal configuration (Tremains et al. 2001), and it significantly restricts the surgeon's work area due to the camera and mirror system intruding approximately 30-40 cm into the space in front of the X-ray emitter, according to an image in (von der Heide et al. 2018). 7 Figure 1.2 Video camera and the double mirror construction (Navab et al. 2010) 1.2.2 Mirror-based RGBD X-ray imaging system A subsequent iteration of CamC used a mirror-based RGBD (depth camera) X-ray imaging system, referred to as RGBDX (Habert, Gardiazabal, et al. 2015). In order to merge the RGBD camera's 3D visual data with the X-ray imaging data from the C-arm, the RGBDX system uses an inclined mirror mounted to the C-arm beneath the X-ray source as shown in Figure 1.3. By superimposing the X-ray image to the 3D reconstruction of the surgical scene, the system creates a 2D/3D augmented reality view on a computer display to enhance the surgeon's understanding of the surgical field. The calibration process is similar to the approach described in CamC (Navab et al. 2010). They achieved an average accuracy of 1.6mm when comparing the projected marker positions to their actual positions in X-ray images of spine, bone, and thorax models. However, the device is still used in an inverted configuration, and the mounting and mirror construction still intrude 8 significantly into the space around the X-ray source, which also requires hardware modification to the X-ray source side of the C-arm equipment. Figure 1.3 Setup of RGBDX with a Asus Xtion Live Pro and a mirror mounted on a C-arm (Habert, Gardiazabal, et al. 2015) Subsequently, (Habert, Meng, et al. 2015) introduced a less obtrusive system that avoids the need to install a mirror in the X-ray beam. It involves attaching two RGBD sensors with overlapping fields of views on either side of the X-ray source of a mobile C-arm (see Figure 1.4). In order to merge the video streams and create a projective color image from the viewpoint of the X-ray source, they used the depth information from the two cameras. They calibrated the cameras to the X-ray source using a 3D calibration object. The calibration involves imaging a marked grid object at various heights above the C-arm intensifier, capturing X-ray, color, and depth data, and using the 2D X-ray and 3D depth positions of the markers to calibrate the X-ray source and RGBD camera setup. 9 They found an average overlay error of 1.7mm when X-ray markers were placed on a hand patient and a spine model. However, this system still requires using the C-arm in the inverted configuration, which increases radiation exposure to the surgical team, and the setup still limits surgical workspace as shown in see Figure 1.4. Furthermore, the system has not yet been validated in actual surgical procedures. Figure 1.4 Setup with 2 Kinects attached on the C-arm gantry (Habert, Meng, et al. 2015) 1.2.3 Integrated C-arm with an RGB-D sensor Most recently (Hosseinian et al. 2020) proposed addressing the limitations of Habert's system (2015), principally the need for using the device in the inverted configuration, by mounting the depth camera directly on the C-arm detector side. They introduced a new calibration procedure involving both the C-arm intrinsic parameters and the 3D-2D transformation between the RGB-D camera and the C-arm. To improve reliability of this transform and overlay error, they proposed a pre-operative calibration step to determine pose-10 dependent C-arm intrinsic parameters using a novel 3D multi-modal Lego phantom with known reference points visible in both the X-ray images and depth data. The phantom is imaged by the C-arm and RGB-D camera from multiple viewpoints, and analyzing the phantom images allows for determining the spatial relationship and calibrating the RGB-D camera with respect to the C-arm source. Once calibrated, this precomputed spatial mapping is used intra-operatively to overlay the real-time X-ray images from the C-arm onto the 3D reconstruction from the RGB-D sensor as shown in Figure 1.5. In order to estimate the current C-arm pose and use proper C-arm intrinsic calibration parameters during intra-operative use, they used 3D-3D registration algorithms to align the live point cloud to the initial reference and reported a mean error of 1.03 mm when the C-arm had no rotation relative to its initial calibrated position. However, the article did not explicitly describe how the overlay scale factor or the absolute distance from the C-arm to the object is measured. Though they presented these preliminary calibration results, no further validation or clinical studies were reported afterwards. Figure 1.5 Overlay process on reconstructed model of the scene: A) 3D reconstructed model of the scene, B) The corresponding X-ray image, and C) The X-ray overlay (Hosseinian et al. 2020) 11 1.3 Previous Work: DeCAF (Depth Camera-Augmented Fluoroscopy) Simultaneously with, but independently of Hosseinian’s work, our group (Surgical Technologies Lab, UBC) came up with the insight that using the depth camera on the detector side, enabling use in the standard (non-inverted) configuration, might yield similar benefits to Habert’s system (2015) without significantly changing the normal surgical workflow and avoiding the adverse effect of increased radiation scatter. As with Hosseinian’s system, DeCAF uses a 3D RGB+depth (RGB-D) camera mounted to the image intensifier tube of the C-arm to generate an augmented reality view for the surgeon in which the most recently acquired X-ray image is overlaid on a live video view that allows the surgeon to easily and intuitively understand the relationship between the patient’s surface anatomy, surgical tools in the operating field, and anatomical and surgical structures within the patient (Hickey et al. 2019). We hypothesize that this improved perception of the relative positions of surgical tools and the patient’s anatomy will enable the surgeon to avoid taking multiple X-ray images to verify the relative position of their tools and the patient’s anatomy and potentially improve the efficiency of performing selected procedures. Our main objectives in this thesis are to develop our initial implementation of DeCAF system so that it can be deployed in the OR and to evaluate DeCAF’s utility in a simulated lab study emulating a likely clinical application of the system. 1.3.1 Principles of Operation A preliminary demonstration of the initial version of DeCAF was presented in (Hickey et al. 2019) This work introduced the physical components and principal pipeline for pre-operative and intra-operative steps to generate X-ray overlay using the RGB-D camera mounted to the image intensifier and included samples of X-ray overlays on anatomical models. 12 Hickey designed a 3D-printed camera assembly to secure the RGB-D or depth camera onto the image intensifier. This assembly features a groove for installation onto the image intensifier using a silicon band, as shown in Figure 1.7 C. The casing consists of two pieces clipped together using a snap-fit joint. Additionally, two flexible metal pieces are glued to the casing and can be used to adjust the orientation of the camera casing relative to the mount. In order to reproject the 3D point-cloud generated using the depth camera to the C-arm’s viewing perspective, a virtual camera is positioned nominally in line with the axis of the image intensifier tube, as shown in Figure 1.6. This point-cloud reprojection to the virtual camera view requires determining the transforms between the depth camera frame (D) and the virtual camera view (V), or 𝑇𝑑𝑣. Figure 1.6 A) Overview of DeCAF system and coordinate frames of the depth camera (D) , virtual camera (V), and calibration disk (C), B) camera setup when used in a mock surgical scene (Hickey et al. 2019) 13 To reproject the point-cloud from the virtual camera view: 1) The transform between the calibration plane (C) and the depth camera (D), 𝑇𝑐𝑑 is found pre-operatively using a calibration paper with four ArUco markers aligned with the laser aimer cross hair, as shown in Figure 1.7 A. The locations of the markers are hard coded into the script, and the pipeline uses available marker detection functions in OpenCV to detect the markers positions. As shown in the user interface in Figure 1.8 A, the detected markers are highlighted with green borders. Once all four markers are detected, the user can save the positions by pressing a specific keyboard button. The software averages the coordinates to find the centre position of the calibration paper and uses least-squares sitting of two 3-D point sets by (Arun et al. 1987) to find the optimal transform (rotation and translation) to register two sets of corresponding 3D point data, the corners of the four ArUco Markers with the average position of markers serving as the origin (cross sign shown in ) in the world coordinate system and corresponding points observed using the depth camera. For more details on pipeline used during the calibration of the virtual camera’s location, please refer to Appendix A . 2) The transform between the depth camera (D) and the virtual camera (V) is estimated using rotation of 𝛾 radians around the 𝑥𝑑 axis, where 𝛾 is extracted from the 𝑇𝑐𝑑 determined in the earlier step, as shown in Figure 1.6 A, and a translation along the 𝑦𝑑 axis, which is found using 𝛾 and 𝑑𝑖𝑠𝑡𝑧, the median depth captured by the camera, as follows: Translation along 𝑦𝑑= sin 𝛾 𝑑𝑖𝑠𝑡𝑧 14 Figure 1.7 A) Calibration paper with four ArUCo markers, where the cross sign represents the average coordinate position of all markers and is aligned with the laser aimer crosshair, B) Calibration disk used to identify the X-ray warp transform step, featuring colored markers with underlying bearing balls, and two slots allowing for alignment with the laser aimer, C) Camera mount including two clipped casings and a mount – The assembly is installed onto the C-arm with a silicon band Once the calibration is done, to overlay the X-ray on the reconstructed point-cloud, the following steps are performed: 1) Reproject point-cloud from virtual camera view: The point cloud is reprojected from the virtual camera perspective by applying the transform comprising the rotation around the 𝑥𝑑 axis and translation along the 𝑦𝑑 axis found in the earlier step. For more details on pipeline used during the point-cloud reprojection, please refer to Appendix B . 2) Align the calibration disk with the laser aimer and capture the X-ray: To overlay the X-ray on the reconstructed view from the virtual camera considering the need to perform scaling, Hickey introduced a calibration disk with visible elements on both the X-ray and video featuring slots indicating the object's perpendicularity to the laser aimer for calibration, as shown in Figure 1.7 B. This ensures that the X-ray view aligns closely with the vertical direction. After aligning 15 the object with the laser aimer, the user captures the X-ray of the object. By pressing the specific key, the user transfers the recent X-ray image to the system. The X-ray image is then cropped to a square shape, as shown in Figure 1.8 B. 3) Select corresponding points on the X-ray and virtual camera view: The user selects points on the X-ray image and the corresponding coloured points on the virtual camera view, as demonstrated in Figure 1.8 B. 4) Generate X-ray overlay by warping the video to match with the size of the video: Using the points identified in the previous step, the software calculates a transform to perform a 2D-2D warp of the X-ray onto the virtual camera view and generates the X-ray overlay (see Figure 1.8 C). At this stage, the user can remove the calibration disk and position an anatomical object of interest (AOI) at the same height where the calibration disk was initially placed. This allows for capturing another X-ray and generating the overlay. 16 Figure 1.8 Three steps in the DeCAF pipeline: A) Calibrate virtual camera using the calibration paper, B) Generate the virtual camera view; capture X-ray; and select the corresponding points on the X-ray and the video, C) Generate X-ray overlay by warping the X-ray image onto the video 17 1.3.2 Limitations Hickey made important simplifications in his implementation that enabled a rapid early proof-of-concept, but ultimately made it more challenging to translate the concept into the operating room. These were: 1) Hardware Components, 2) Intra-operative Use of a Calibration Disk, 3) Simplified Transform Formulation, 4) Position Estimation Using Four Independent ArUco Markers, 5) Video Distortion and Quality, and 6) Limited Control in the User Interface. 1.3.2.1 Hardware Components Hickey used 3D-printed components shown in Figure 1.7 C. This design facilitates clipping two casings and their connection to the mount. The mount can be easily installed onto the image intensifier using a silicone band, with the possibility for intra-operative rotation. However, some considerations are necessary for their use in the operating room, and further iterations are needed. Certain parts, such as the snap-fit join design used for clipping two casings and the two adjustment metals on each side, were found to be too fragile. Additionally, the metal pieces are glued to the 3D-printed mount, and their cleanability has not been demonstrated. Inspired by the concept of this design, our aim is to address these limitations in the next design iteration. 1.3.2.2 Intra-operative Use of the Calibration Disk A major challenge arises with the need to use the calibration disk during the intra-operative use of the system. The homography matrix determined during the point selection step in Figure 1.8 B is only valid for the specific distance at which the calibration disk was initially placed. If the depth or distance of the AOI relative to the image intensifier changes, as shown in Figure 1.9, the corresponding image dimensions on video and X-ray also change. Consequently, the user needs to repeat the step shown in Figure 1.8 B again to rescale the new X-ray image. Furthermore, this step 18 requires the user to perform mouse clicks on each medium, which causes inconvenience in practical operating room scenarios. We propose that separating the tasks of locating the position of the virtual camera and identifying the distance to the AOI would enable a simpler and more flexible intraoperative workflow. Figure 1.9 Image dimensions of AOI on the X-ray and video at A) close depths, and B) far depths relative to the image intensifier. In this representation 𝒘𝒂 is the AOI size, f’ is the distance between virtual camera and image plane, d and D are the distance from the target plane and X-ray tube to the image intensifier, respectively and images created on the video and X-ray are of 𝑷𝒂𝒗 and 𝑷𝒂𝒙 pixels in the 𝒚𝒄 direction, respectively - Note that this is because the X-ray emitter and the depth camera are positioned on opposite sides of the patient. 19 1.3.2.3 Simplified Transform Formulation As discussed in 1.3.1, the depth camera (D) to virtual camera (V) transform, which is required to reproject the point-cloud to the virtual camera view and generate 2D view from this view, comprises a rotation around the 𝑥𝑑 axis and a translation along the 𝑦𝑑 axis. During initial testing, we observed that the calibration paper's cross sign, as shown in Figure 1.7, initially aligned with the laser aimer, was not centered with the center of the reconstructed view from the virtual camera. This offset, of several centimeters, suggests an error in the transform formulation during calibration, due to simplification and reliance on planar transforms. Hickey assumed that the camera assembly, as depicted in Figure 1.7 C, is installed to the image intensifier at a position that precisely replicates the spatial relationship shown in Figure 1.6 A. However, in practice, we observed challenges in achieving this specific camera orientation as shown in Figure 1.10 B; For instance, even a slight angle deviation along 𝑦𝑑 axis of say 5 degrees, considering the 40cm distance between the camera and calibration plane, could result in approximately a 3-4 cm offset between the cross sign of the calibration paper and the center of the reconstructed view from the virtual camera. We propose that implementing a 6-degree-of-freedom (6-DOF) approach to determine the transform from the depth camera to the virtual camera would enhance the accuracy of reprojecting the point-cloud onto a direction perpendicular to the calibration plane. 20 Figure 1.10 A) Nominal position of the camera assembly relative to the image intensifier; alignment of the C-arm axis and virtual camera axis, B) Real position of the camera assembly relative to the image intensifier; offset between the center of the virtual camera view and the laser aimer crosshair (C-arm axis) 21 1.3.2.4 Position Estimation Using Four Independent ArUco Markers In Hickey’s original calibration method, he calculated the virtual camera location (characterized by the transform 𝑇𝑑𝑣 shown in Figure 1.6) by using a calibration paper displaying four ArUco markers whose positions can be read from the depth camera images as shown in Figure 1.7 A. As evidenced in Appendix A in the preliminary demonstration, we found that Hickey had to write custom code to scan individual markers and extract the origin of the board. In his method, the corner positions of the markers are hardcoded, and the average position of the four corners is used to estimate the position of the calibration paper relative to the camera (characterized by the transform 𝑇𝑐𝑑 shown in Figure 1.6). In contrast, by using an ArUco board, we can leverage predefined functions to simplify this process, while taking into account the marker configuration and their relative positions which is predefined. Furthermore, the software requires all markers to be visible during calibration, which can be challenging when lighting conditions affect one or more markers' visibility. We propose that using calibration boards that don’t require all markers to be visible could resolve this problem. In addition, the current calibration board is printed on paper, which is unsuitable for use in the OR due to its limited accuracy and inability to be sterilized. Therefore, there is a need to design a tool with markers that are printed, scannable, and can be integrated into use in the OR. 1.3.2.5 Video Distortion and Quality In order to correct lens distortions and accurately map 3D world coordinates to 2D image coordinates, intrinsic parameter calibration needs to be performed for Intel RealSense cameras. Hickey performed an intrinsic parameter calibration and used the obtained parameters during the marker detection step. However, he did not write the parameters into the camera’s memory. 22 Consequently, during the processing and reprojection of the point cloud from the virtual camera frame, the camera still uses the factory parameters, potentially resulting in distortions that affect overlay error. In the next iteration, we propose to write the intrinsic parameter calibration into the camera’s memory. In addition, Hickey used the default setting of 640×480 for both sensors, resulting in the relatively low-resolution overlay shown in Figure 1.8. These default settings are commonly used because they offer a balance between performance and image quality. To achieve the most precise calibration and overlay error, we aim to use the highest resolution available. 1.3.2.6 Limited Control in the User Interface The software structure of Hickey’s implementation is shown in Figure 1.11. As shown in Figure 1.8, in order to help the user understand the current step, text hints appear at each step on the lower part of the screen. The controls were also limited to keyboard buttons, and there was no option for recalibration or capturing the X-ray image again. In the next version, we aim to solve these gaps by preparing a user interface with enhanced user guidance, more intuitive controls, and capabilities for redoing any of the previous steps. 23 Figure 1.11 Initial pipeline of the DeCAF system and main user controls that navigate the steps in the pipeline including A) Calibrate virtual camera using the calibration paper, B) Generate the virtual camera view; capture X-ray; and select the corresponding points on the X-ray and the video, C) Generate X-ray overlay 24 1.4 Specific Research Objectives Given the various limitations mentioned above related to the first implementation of the DeCAF system, in order for this system to be used in the operating room, our priorities are to: • Refine the hardware to address operational concerns such as cleanability issues and mounting problems. • Improve the software by relaxing the simplifications mentioned, including the intra-operative use of the calibration disk and simplified transform formulation for position estimation using four independent markers. • Overcome the lack of options for redoing specific steps and the limitations of keyboard controls by developing a user interface with clear instructions and accessible control features. Therefore, our specific objectives in this thesis are: • To refine DeCAF so that it can be evaluated in a live operating room. • To verify the technical performance of DeCAF prior to user testing (in particular, is the overlay error within clinically usable limits below 2mm). • To test DeCAF on a simulated procedure designed to emulate a surgical situation in which DeCAF is predicted to provide a significant surgical benefit to gain more insight into whether or not DeCAF can likely deliver the anticipated benefits (in particular, can DeCAF reduce the number of X-ray images needed to perform a targeted surgical procedure) 25 1.5 Thesis Overview In Chapter 1, we reviewed the motivating problem and explored previous efforts to augment C-arms with X-ray overlays. We introduced the Depth Camera Augmented Fluoroscopy (DeCAF) system, initially demonstrated by Hickey, and discussed its limitations. In Chapter 2, we delve into the design and development of the DeCAF system, providing a technical evaluation of its components. Chapter 3 reviews the performance of DeCAF in simulated target procedures, assessing its effectiveness in surgical tool placement and reducing the need for repeated X-rays. Chapter 4 presents a discussion of the results from the previous chapter, summarizing the significance of the findings, offering recommendations for future work, and highlighting the thesis contributions to the field. 26 Chapter 2: Design and Development of depth camera augmented fluoroscopy (DeCAF) In the previous chapter, we presented the DeCAF concept and initial prototype, and identified limitations related to its functionality, accuracy, and usability. These limitations prevent it from being deployed to the operating room (OR) at its current stage of development. The objective of this chapter is to present refinements to the design of DeCAF (both its hardware and software) aimed at making it suitable for further evaluation and ultimately for live use in the OR. We describe the DeCAF system in the following order: 1) components, 2) operation, and 3) operational pipeline. We then discuss the pre- and intra-operative software design and pre-deployment calibration, present clinical samples of X-ray overlay on anatomical models, and evaluate the requirements and accuracy of the system for clinical use. 2.1 Components The components of the DeCAF system are as shown in Figure 2.1. The hardware setup comprises a depth camera assembly including the depth camera, a camera casing and mount, a silicon band to secure the camera to the image intensifier tube, and a camera connector cable, all of which are set up pre-operatively on a C-arm behind a transparent sterile drape so that these components remain outside the sterile field. Additionally, there is a pointing tool that is used within the sterile field during calibration and depth indication and a computer which runs the software and displays the real-time X-ray overlay on a monitor positioned beside the C-arm’s display. The components of the system are as shown in Figure 2.1. 27 Figure 2.1 Overview of DeCAF system main components: A) depth camera assembly, B) pointer tool, C) computer and monitor The device and all components are intended be used behind the plastic drape or inside a plastic bag so that there is no contact with the patient. Therefore, the components can be cleaned and disinfected after use and the system can be re-used on subsequent patients. 2.1.1 Hardware Design In this section, we describe the hardware components in more detail, including the camera assembly components, the pointer/calibration tool design, and electrical components and connections. C A B 28 2.1.1.1 Camera Assembly In this section, we describe the design of the camera assembly, including depth camera selection and design of the camera mounts. 2.1.1.1.1 Depth Camera To specify the depth camera's requirements, it is important to recognize that the source-to-skin distance for a mobile C-arm is normally not less than 30 cm (Lakhwani et al. 2019). Furthermore, the X-ray emitter to image intensifier distance in the Siemens Arcadis Orbic 3D C-arm is 98cm (BioteXray). Therefore, we selected the Intel RealSense D435i as our depth camera as it has a depth range of 30-300 cm (see Table 2.1 for more details on camera specifications) which corresponds well with the range of distances we expect to encounter in orthopaedic procedures. Moreover, the small size of the camera ensures it does not take up too much space in the surgeon's work area. Our assessment indicates it delivers output with acceptable quality, even when positioned behind the surgical drape, as will be discussed in 2.6.3.2. Table 2.1 Technical specifications of Intel RealSense Depth Camera D435i (© Intel Corporation) Ideal range .3 m to 3 m Depth Field of View (FOV) 87° × 58° Minimum depth distance (Min‑Z) at max resolution ~28 cm Depth output resolution Up to 1280 × 720 Depth frame rate Up to 90 fps RGB frame resolution 1920 × 1080 RGB frame rate 30 fps Length × Depth × Height: 90 mm × 25 mm × 25 mm 29 2.1.1.1.2 Camera Mount Assembly In order to secure the camera onto the image intensifier and adjust its angles, a camera mount assembly is designed which consists of a mount and two casings 3D-printed using polylactic acid (PLA) plastic, as shown in Figure 2.2 as A, B, and C, respectively. The lower casing and upper casing are clipped together using a snap-fit joint design, and the subassembly is rotated to different angular positions and held in place using a ball plunger. We describe the preparation process and draping of the assembly in 2.2. Figure 2.2 Camera Assembly Components: A) Camera mount, B) Upper Casing (connects to the lower camera with snap-fit joint); C) Lower casing (where the depth camera is placed) 2.1.1.2 Pointer Tool In order to pre-operatively calibrate the camera and reproject the live video from virtual camera view, a pointer tool is placed on the table manually with axes nominally aligned with the C-arm 30 before the patient is transferred to the table, as shown in Figure 2.3. The pointer tool consists of three design aspects: • Tip to point to the target anatomy • Handle to facilitate manipulation by surgeon and meet both calibration and depth measurement requirements • Marker pattern which is scanned by the camera to determine the required transforms in operational pipeline The marker pattern is an ArUco Board, which is generated using OpenCV (OpenCV 2022). Once printed, the marker can be scanned and recognized by a camera using OpenCV functions, provided the specifications of the board including type, dimensions, numbers, and configuration of markers are known. By default, the origin of the board is located at the bottom left, as shown in Figure 2.3 A. Figure 2.3 Calibration tool: A) Principal axes B) Pointer tool alignment with C-arm laser aimer x y A B 31 We used ArUco Boards in the pointer tool design as they offer pose estimation even with partial marker visibility or occlusions to solve the issue of marker occlusion observed in the earlier version of the DeCAF. Additionally, the availability of a higher number of point correspondences (marker corners) enhances the accuracy of the pose estimation. We evaluated various arrays based on criteria such as marker visibility, robustness to occlusion, and overall tracking accuracy, and determined that the 3×4 grid design with approximately 2 cm length markers on our ArUco board maintains a favorable balance between these factors. The pointer tool is printed on a rigid premium 1/8-inch Dibond (a rigid ultra flat sheet made of two sheets of pre-finished aluminum bonded to a polyethylene core) by (foamcoreprint 2024). Once the calibration is done, the object can be taken off the table. As stated in the previous chapter, a scale calculation needs to be performed to compute the overlay scale based on target depth (since, in contrast to the Cam-C system, with DeCAF, the X-ray emitter and the depth camera are positioned on opposite sides of the patient). In order to compute the target depth and determine the proper scale for the overlay, the same pointer tool is also used to indicate the target anatomy plane. Consequently, scanning the markers allows position detection and finding the vertical distance of the pointer tool to the depth camera. We will describe the explicit equations used during this process in 2.3.1.1 and 2.3.1.2. 2.1.1.3 Electrical Connections To obtain real-time X-ray images from the C-arm, we redirect the signal sent to the secondary monitor of the C-arm to our computer, where it is used within our software. As shown in Figure 2.4, the VGA output of the monitor is connected to an AGPtek HD video capture card (AGPtek) via a VGA to HDMI adapter, powered by the nearby computer. This adapter is then linked to the 32 video capture card using an HDMI cable, which in turn connects to the computer via USB 3.0. The Intel RealSense d435i depth camera, which is placed a camera casing, is connected to a computer through a USB 3.00 interface. Figure 2.4 Overview of system components and electrical connections 2.2 Operation To prepare the system for displaying overlays, we follow three steps: electrical setup, mechanical setup, and software setup as described in Figure 2.5. Additionally, post-procedure disassembly and cleaning is performed after each use of the system in the OR. 33 Figure 2.5 Overview of system operation including three steps: electrical setup, mechanical setup, and software setup For this work, X-ray images were obtained using a Siemens Arcadis Orbic 3D C-arm. In order to set up the camera assembly and secure it to the C-arm, the system operator (radiology technologist and/or a researcher) first places the depth camera into the lower casing (see Figure 2.2 B) and attaches it to the upper casing (see Figure 2.2 C) using the snap-fit joint. The system operator then attaches the camera casing to the camera mount (see Figure 2.2 A), where it can be rotated to different angular positions and held in place. To mount the assembly, which includes the camera, casings, and mount, to the image intensifier tube, as shown in Figure 2.1, the system operator carefully wraps a clean silicon band around the groove on the camera mount and the image intensifier. The system operator subsequently connects 34 a clean connector cable to the camera and connects that cable to a computer positioned outside the sterile field. When the system is used in OR, the C-arm is draped with a sterile sheet in the usual manner, and then a portion of the drape near the camera lenses across the camera face will be flattened and secured to the camera casing using a sterile rubber band to minimize optical distortions due to the presence of the drape in front of the camera, as illustrated in Figure 2.6. Figure 2.6. A) Camera mount assembly with plastic flattened around the lenses using a disposeable rubber band, B) Expected use of the system in the OR when the C-arm is draped 35 Finally, the pointer tool is placed inside a sterile bag. After each use, they must undergo thorough cleaning and disinfection following low-level protocols. Additionally, plastic bags are discarded after every use. The procedures of cleaning and disinfecting each of the above-mentioned components are described in Figure 2.7. Figure 2.7 Instructions for cleaning, disinfection, and the products to be used for reprocessing the device between patient uses 2.3 Operational Pipeline In this section, we describe an overview of the operational pipeline, highlighting the primary processes and mathematical principles involved in finding transforms and determining the X-ray scale. Subsequently, we will present a qualitative demonstration of overlay functionality by applying the system to several anatomical models. 36 2.3.1 Overview of Operational Pipeline The software follows the pipeline introduced in Figure 2.8, consisting of two main parts: pre-operative calibration and main pipeline. The one-time pre-deployment calibration is performed only once for each camera and C-arm model used and includes two steps. First, we perform the calibration of the depth camera to update intrinsic parameters for tracking accuracy, as will be explained in 2.5.1. Second, to adjust for laser aimer misalignment, we perform machine calibration, which will be explained in more detail in 2.5.2. 37 Figure 2.8 Software workflow for pre-deployment calibration (top) and pre- and intra-operative processes (bottom) – Note that, intra-operatively you can go back and forth between the two steps in the intra-operative use. 38 For the main pipeline, we execute three primary steps, as shown in Figure 2.9. • Locate Virtual Camera: First, we conduct pre-operative virtual camera calibration by aligning the pointer tool with the laser aimer and having the camera scan the pointer tool to determine the necessary transforms for reprojecting the point-cloud from a top view (see Figure 2.9 A). • Select Overlay Depth: Next, we perform a scan of the pointer tool, holding the pointer tool in line with the target anatomy to enable accurate scaling of the X-ray (see Figure 2.9 B). • Generate X-ray Overlay: Finally, the software projects the point-cloud from the C-arm view (referred to as virtual camera view as will be discussed in 2.3.1.1), using the transforms obtained earlier. This process includes capturing the X-ray image and generating the overlay (see Figure 2.9 C). To determine the X-ray scale, we apply the scale obtained in the previous step based on the measured depth. We then center the extracted X-ray image on the center of the obtained image and adjust the X-ray image position based on laser aimer alignment parameters. We will describe each step in more detail in the following subsections. Figure 2.9 The software carries out three main steps: A) Calibrate virtual camera B) Find indicated anatomical distance to image intensifier C) Project the point-cloud from C-arm view, capture X-ray and generate overlay. B A C 39 In the following two sections, we describe the mathematical formulations for calculating the key transforms and X-ray overlay scale that are calculated in the pre-operative calibration step and used in the intraoperative pipeline. 2.3.1.1 Formulate Transforms As shown in Figure 2.10 A, a virtual camera is positioned at the same height as the actual camera in line with X-ray axis and z axis of the pointer tool or calibration board frame (C). The objective is to determine the transform between the depth camera frame (D) and the X-ray view (V) to allow reproject the point-cloud from the X-ray view. The transformations are as depicted in Within select_tracking_method, the tracking method that the user would like to use is selected and important parameters of the pointer tool used could be initiated. As we use ArUco Board, where Tdc, Tcv and, Tdv are transformations from the depth camera frame (D) to pointer tool frame (C), the pointer tool frame (C) to virtual camera frame (V), and the depth camera frame (D) to the virtual camera frame (V), respectively. Figure 2.10 A) Transforms between depth camera, pointer tool and virtual camera, B) 3D frames of the depth camera (D), pointer tool (C), and virtual camera (V) 40 The key steps to calculate each transform are shown in Figure 2.11 and described below. Figure 2.11 Overview of the transforms and steps to calculate each transform • To calculate Tdc, we perform position estimation of the pointer tool relative to the depth camera the transformation between the depth camera and the virtual camera - ie., Tcd. Subsequently, we find Tdc using Tdc = Tcd−1. Tdc = [𝑟11 𝑟12 𝑟13 𝑡𝑥𝑟21 𝑟22 𝑟23 𝑡𝑦𝑟31 𝑟32 𝑟33 𝒕𝒛 ∗0 0 0 1] 41 • The frame (V) is nominally displaced purely vertically at distance of 𝑡𝑧 ∗ from frame (C) and reflected around 𝑥𝑐 axis. To calculate the transform Tcv, a vertical translation of 𝑡𝑧 ∗ along 𝑧𝑐 axis and a 180° rotation around 𝑥𝑐 axis are performed, where 𝑡𝑧 ∗ is the estimated vertical distance and is extracted from position (3,4) of Tdc as follows: Tcv = T𝑧𝑐,𝑡𝑧∗ R𝑥𝑐,180° = [1 0 0 00 1 0 00 0 1 𝒕𝒛 ∗0 0 0 1] [1 0 0 00 −1 0 00 0 −1 00 0 0 1] • Finally, we find the transform between the depth camera and the virtual camera, Tdv, using the rule of composition – i.e., Tdv = Tdc × Tcv. 2.3.1.2 X-ray Overlay Scale In this section, we describe how to calculate the scaling needed to overlay the X-ray image of an anatomical object of interest (AOI) positioned within the virtual camera view considering the opposing viewing directions of the X-ray and virtual camera. As demonstrated in Chapter 1 and as shown in Figure 2.12, if the AOI of width 𝑤𝑎 is placed at a target plane of depth d, it will project onto the image sensing planes of the video camera and X-ray, creating images of 𝑃𝑎𝑣 and 𝑃𝑎𝑥 pixels in the 𝑦𝑐 direction, respectively. Therefore, there is a mismatch in corresponding pixel sizes of X-ray and video image of the AOI. Furthermore, as the distance between the AOI and the image intensifier changes, the mismatch in the image sizes of the AOI will vary depending on the depth. To account for this mismatch and achieve an accurate X-ray overlay, we need to find the relationship between the image sizes and establish the X-ray overlay scale, s(d), which is a function of the vertical position, d, of the AOI. As shown in Figure 42 2.12, since the camera's field of view at the plane of the AOI is normally larger compared to the X-ray's, we normally only scale the X-ray image to match the corresponding image size of the AOI in the video, i.e., 𝑃𝑎𝑣 . Therefore, 𝑃𝑎𝑣(𝑑) = 𝑠(𝑑) 𝑃𝑎𝑥(𝑑). Figure 2.12 Illustration of scaling relationships for X-ray and camera images: 𝒘𝒂: AOI size, f’: distance between virtual camera and image plane, and s(d) is the scaling factor 43 In order to find s(d), we establish the relationship between image sizes from the virtual camera’s perspective and focal length and determine how 𝑃𝑎𝑣 and 𝑃𝑎𝑥 change over depth. Using the principles of the pin-hole camera model (Hartley and Zisserman 2004), we have: 𝑠𝑣𝑃𝑣 𝑤𝑣(d)=𝑠𝑣𝑃𝑎𝑣(d) 𝑤𝑎=𝑓′ 𝑑 𝑃𝑎𝑣(d) =𝑤𝑎𝑓′ 𝑠𝑣 𝑑 where 𝑠𝑣 is the scaling factor for the camera, converting pixels to mm. Likewise, considering the X-ray beam source location, additional relationships exist between the scaling factors for the X-ray image, illustrated below where D represents the distance from the X-ray source to the image intensifier: 𝑠𝑥 𝑃𝑥𝑤𝑥(d) =𝑠𝑥 𝑃𝑎𝑥(d) 𝑤𝑎 = 𝐷 𝐷−𝑑 𝑃𝑎𝑥(d) =𝑤𝑎 𝐷𝑠𝑥 (𝐷 − 𝑑) where 𝑠𝑥 is the scaling factor for the X-ray, converting pixels to mm. Finally, s(d) is found as: 𝑠(𝑑) =𝑃𝑎𝑣(𝑑)𝑃𝑎𝑥(𝑑)=𝑤𝑎𝑓′ 𝑠𝑣 𝑑 𝑤𝑎 𝐷𝑠𝑥(𝐷 − 𝑑)= (𝑠𝑥 𝑠𝑣)(𝑓′ 𝐷)𝐷 − 𝑑𝑑 44 where 𝑓′ is the focal length of the camera (available in the camera specifications), and D is the distance between the X-ray source and the image intensifier (not known). We combine all these constants (𝑠𝑥 𝑠𝑣)(𝑓′ 𝐷) into a parameter called c, and describe s(d) as follows: 𝑠(𝑑) = 𝑐𝐷 − 𝑑𝑑 where c and D are found in the machine calibration step. Note that in the formula for s(d), d is the target depth of the AOI and is found by calculating the distance between the pointer tool surface and the camera, along the 𝑧𝑐 axis. This parameter is equivalent to 𝑡𝑧 ∗ in the following transform and is extracted from position (3,4) of Tdc as follows: Tdc = [𝑟11 𝑟12 𝑟13 𝑡𝑥𝑟21 𝑟22 𝑟23 𝑡𝑦𝑟31 𝑟32 𝑟33 𝒕𝒛 ∗0 0 0 1] Having calculated s(d), we scale the X-ray image on the video stream based on the target depth of the AOI. 2.3.1.3 X-ray Image Processing and Blending The initial X-ray image captured using the video capture card encompasses the entire screen, including the imaging software environment. Therefore, we first extract the X-ray image using a pre-defined mask. We then scale the X-ray image based on the depth measured earlier, adjust its position based on machine calibration parameters and the depth measured, and create masks for overlaying it onto the projected point cloud. The resulting X-ray image is blended with the main output using alpha blending (Porter and Duff 1984). This blending process combines the X-ray 45 image with the main output using a weighted average based on alpha values using 𝛼 × 𝑖𝑚𝑎𝑔𝑒𝑣 +(1 − 𝛼) × 𝑖𝑚𝑎𝑔𝑒𝑥 where α is the alpha value, and 𝑖𝑚𝑎𝑔𝑒𝑣 and 𝑖𝑚𝑎𝑔𝑒𝑥 are the pixel values of the video and X-ray images being blended. This technique allows for user-defined transparency. To ensure compatibility with the output image size, the X-ray extracted image is padded with extra black pixels. Figure 2.13 illustrates an example of this process: Figure 2.13 Overview of the overlay process – The captured X-ray image is scaled by s(d), masked, and padded with zeroes to match with the dimensions of the video image before alpha blending 2.3.1.4 Adjustment Model for Laser Aimer Misalignment Misalignment between the laser aimer and the X-ray physical axis causes the center of the image to not align with the axis point of the X-ray which results in inaccuracies in overlay alignment. As shown in Figure 2.14, while the sample object appears to be perfectly aligned using the laser aimer, there is an offset between the center position of the X-ray image and the object. This issue becomes particularly relevant since the laser aimer may not always be accurately calibrated, a situation that may arise in other C-arms used in operating rooms. Our objective is to address this issue without physically calibrating or adjusting the laser aimer as this may not always be possible. 46 Figure 2.14 Misalignment between image center and corresponding X-ray of a reference object: A) Sample object with centre aligned with the laser crosshair, B) Corresponding X-ray, C) misalignment from nominal cross point (emitter location indicated by small black dot), D) misalignment observed from another view The crosshair is formed by the intersection of the rays emitted from two lasers mounted on the image intensifier, as illustrated in Figure 2.15. If there is laser aimer misalignment, the cross section point of the beams (shown as a green point in Figure 2.15 A) would be away from the 47 nominal axis of the C-arm (shown as a red point in Figure 2.15 A) both in the physical world. This phenomenon is also observable in the X-ray image as there is an offset between the centre of the image (shown as a red point in Figure 2.15 B) and the centre of the sample object which is aligned with the laser crosshair (shown as a green point in Figure 2.15 B). Figure 2.15 Laser cross hair forming and the misalignment observed in: A) Physical world, and B) X-ray image of the sample object which is centred with the laser crosshair - Note that the misalignment between the X-ray image center and the centre point of the sample object in the X-ray, denoted as 𝒄𝒙 and 𝒄𝒚, varies depending on the distance between the plane of interest and the image intensifier As the misalignment between the pointer tool and the lasers varies linearly with the distance from the image intensifier where the lasers are located, we seek to extend the existing overlay model to address these misalignments. If the misalignment between the X-ray image center and the center 48 point of the aligned sample object in the X-ray are denoted as 𝑐𝑥 and 𝑐𝑦 along with x and y of the coordinate system shown in Figure 2.15, to correct the misalignment along x and y, we adjust the centre of the X-ray image relative to the centre of the video image using the following equations: 𝑠(𝑑) = 𝑐𝐷 − 𝑑𝑑 𝑐𝑥 = 𝑐𝑥0𝑑, 𝑐𝑦 = 𝑐𝑦0𝑑 where s(d) is the X-ray image scaling factor as explained in 2.3.1.2, D represents the distance from the X-ray source to the image intensifier, d is the distance from the pointing tool to the image intensifier, and 𝑐𝑥0 and 𝑐𝑦0 are constant values describing the linear relationship between d and 𝑐𝑥 and 𝑐𝑦. Our primary goal is to determine the unknown parameters of 𝑐𝑥0, 𝑐𝑦0, 𝑐, and 𝐷 and establish a more accurate calibration approach without relying on laser aimer calibration. In section 2.5.2, we describe the software component that performs this adjustment for the laser aimer misalignment. 2.3.1.5 Qualitative Demonstration of Overlay Functionality In order to demonstrate the use of the system, we applied the system to several anatomical models featuring radiopaque elements mimicking real anatomy. Figure 2.16 shows examples of the X-ray overlays we obtained using anatomical models. Overall, the system generally shows the AOI at plausible positions with little visually perceptible error. The black or hidden region represents portions of the scene that are not visible from the camera perspective or are located outside the field of view of the camera. There appear to be some residual spots shown as black dots that are not fully reconstructed, but this generally doesn’t interfere with understanding the context. 49 Figure 2.16 Examples of the X-ray overlays obtained in anatomical models: A) knee in a rolled lateral position, B) knee in an anteroposterior (AP) position, C) spine in a prone position, and D) ankle in a lateral position 50 Note that, as discussed above, the scaling of the X-ray overlay depends on the depth of the AOI that the pointer tool indicates. Due to the parallax effect in the X-ray, the overlay is most accurate at the depth of the target plane shown. For example, Figure 2.17 illustrates the X-ray overlay of the proximal tibia with a plate held in place by a K-wire. The pointer tool is positioned approximately in line with the K-wire. As the tool is partially visible in both the video and the X-ray, the surgeon can determine the line of the tool being inserted based on the overlay. The overlay appears to be accurately positioned, as the line in the X-ray aligns with the K-wire. A more accurate test on overlay error will be conducted in 2.6.2 Figure 2.17 X-ray overlay of the proximal tibia with a plate secured by a K-wire 2.4 Pre- and Intra-operative Software Design To implement the operational pipeline, we designed software to carry out the pre-operative calibration and intraoperative use steps. The overall illustration of the main packages and toolkits used in the software development, is shown in Figure 2.18. 51 Figure 2.18 Software packages including main libraries and software development kits We used Python 3.8 as the programming language and OpenCV (OpenCV 2022) and Intel RealSense SDK 2.0 as the main software development kits. OpenCV provides a variety of tools and functions for marker detection including ArUCo markers as well as image processing tools in computer vision applications. Furthermore, we used Intel RealSense SDK 2.0, which is a software development kit provided by Intel for their RealSense depth cameras, and pyrealsense2, a Python wrapper for this tool kit which allows for depth sensing and post-processing, RGB image processing, and point-cloud visualization. 2.4.1 Main Pipeline In this section, we describe the main pipeline including classes and main steps in the user interface. The software, demonstrated in Figure 2.19, is fundamentally built upon three classes described below: 52 A) App: This class within the user interface initiates various operations based on button activations and displays the output frame from each main step in the pipeline using the update_frame_chhm_trackers function in this class. The key elements of the user interface are buttons that activate the three steps in the pipeline and transfer X-ray step. These elements are linked to functions shown as D in Figure 2.19 which interacts and initiates other operations based on the classes chhm_trackers and display_pc. Other elements, such as X-ray overlay controls, including transparency and zoom-in features, are not included in the structure in Figure 2.19 for simplicity. To enable simultaneous operation of the user interface and the camera, we employed the threading library, which allows for parallel execution of tasks. We allocated the main thread for the user interface, and for each step in the pipeline, a new thread is initiated and terminated upon start and completion of that step. B) chhm_tracker2: This class mainly relies on the cv2 and pyrealsense2 packages and provides various methods for camera initialization and marker detection. We used components of this class and made slight modifications concerning the transforms we are interested in and the method of displaying resulting frames within the user interface frame instead of an OpenCV window. Upon activation of the virtual camera calibration or depth measurement step, an instance of this function is called upon to conduct marker detection on the ArUco Board, where the output of the first and second steps are Tdv and 𝑡𝑧 ∗, respectively, as explained in 2.3.1.2. 2 This code uses elements of an earlier implementation of a marker detection package written by another member of Surgical Technologies Lab, Jakub Piwowarczyk. There is no reference to this work available as it is not published yet. 53 For more technical documentation on the structure of the chhm_trackers class and functions used, please refer to 0. C) display_pc: This class is based on cv2 and pyrealsense2 packages. The main processes supported by this class involve setting up and processing camera frames to create the point-cloud and reproject it to the virtual camera's perspective. It also has functions for capturing, scaling, and overlaying the X-ray image based on the measured depth of 𝑡𝑧 ∗ from the earlier step and pre-deployment adjustment parameters for laser aimer misalignment. The latter feature is not shown in Figure 2.19 but will be discussed in more detail in 2.5.2. For more technical documentation on the structure of the display_pc class and functions used, please refer to 0. 54 Figure 2.19 DeCAF Software Structure: A) App: user interface class which runs in the main thread, B) chhm_trackers for Marker detection which is called during virtual camera calibration and depth measurement step, C) display_pc which generates virtual camera frame and overlays and scales the X-ray, D) main functions in App To facilitate easy interaction during clinical use, the user interface (UI) guides users through a step-by-step process to use the DeCAF system. 55 As users initiate the interface, they are guided through the three key pre-and intraoperative steps described in 2.3.1 to locate virtual camera, select the overlay depth, and generate the X-ray overlay as shown in Figure 2.20. To enhance clarity and navigation through the process, we have designed each main step in the interface to activate only the relevant widgets for each step. 56 Figure 2.20 DeCAF GUI which walks users through the three main pre- and intra-operative pipeline 57 Upon setting the fundamental configurations, users can proceed with the following steps through the UI. 2.4.1.1 Locate Virtual Camera During this phase, the user interface calls perform_calibration function which starts running in a thread and to display the RGB video stream and calls pre-existing functions in OpenCV detect the markers in real-time and overlay the corresponding coordinate frame of the board and the borders of the detected markers with green colour, as shown in Figure 2.20 A. The user is instructed to place the C-arm in a vertical orientation located approximately 40-60cm from the operating table and to place the left bottom corner of the pointer tool in alignment with the laser aimer crosshair so that the perpendicular axis of the pointer tool is nominally aligned with the axis of the C-arm. If the ArUco Board is correctly identified, the visibility icon for the pointer tool illuminates prompting “Pointer Tool Visible”, and at the same time, Tcd is calculated in the loop. At this point, user clicks the “Calibrate Virtual Camera” button which terminates the marker detection process in chhm_trackers and saves the Tcd, the transform from the current pointer tool frame (C) to depth camera frame (D). As discussed in 2.3.1.1, the software calculates and stores Tdv for later use in reprojecting the point-cloud from the virtual camera’s perspective. Additionally, the interface features a help button that, when clicked, opens a window providing instructions on correct positioning of the pointer tool. For more details on the main steps in perform_calibration and ArUco Board detection, please refer to 0and 0. 2.4.1.2 Select Overlay Depth Upon completion of the first step, Calibrate Virtual Camera, the user is instructed to indicate the target anatomical plane (see Figure 2.20 B) using the pointer tool. Similar to the previous stage, 58 the live video feed is presented with the tracked marker and pointer tool coordinate frame highlighted on the video. If the ArUco board is correctly identified, the visibility icon for the pointer tool illuminates prompting “Pointer Tool Visible” while Tcd is calculated in the loop. At this point, user clicks the “Set Overlay Depth” button which terminates the marker detection process in chhm_trackers. This time instead of Tcd, 𝑡𝑧 ∗ is stored as described in 2.3.1.2. 2.4.1.3 Generate X-ray Overlay In the final step, the user is presented with a top-view point-cloud visualization (see Figure 2.20 C). The user clicks "Transfer X-ray", which allows the system to capture the X-ray image from the C-arm. Using the depth information obtained in the previous phase, the X-ray image is overlaid onto the corresponding anatomical region. The user adjusts the transparency of the X-ray using the slider to achieve their preferred level of visibility for overlaying onto the corresponding anatomical region. For subsequent overlays with the same depth target, the user clicks "Transfer X-ray" again. Alternatively, if adjustment to the target anatomy depth is necessary, the user reinitiates the process by clicking "Select Overlay Depth" tab and going through the process described above. 2.4.1.3.1 Generate Point-cloud from Virtual Camera View To generate the overlay, we reproject the point-cloud from the virtual camera view using Tdv based on the pipeline shown in Figure 2.21. 59 Figure 2.21 A) Pipeline for generating point-cloud and reprojecting in virtual camera view, B) Schematic of views and point-cloud reprojection To implement these steps shown in Figure 2.21, we used the RealSense SDK to develop the display_pc class. To process each camera frame, create the point-cloud, and display the reconstructed frame from the virtual camera's perspective, the software goes through the pipeline described in Figure 2.22 (formulas presented and used below are from (Dorodnicov et al. 2023). To transform the point-cloud from the depth camera view (D) to the virtual camera frame (V) using the estimated virtual camera position relative to the depth camera, Tdv, we incorporated the Tdv and an additional step in the main point-cloud generation pipeline in display_pc class (shown as green boxes in Figure 2.22). 60 Figure 2.22 Main operations in display_pc class based on RealSense SDK: A) inputs, B) generate point-cloud, C) reproject to virtual camera view, D) find texture coordinates, E) perform uv mapping, and F) output. Note: formulas derived from (Dorodnicov et al. 2023) 61 A) Inputs Inputs include the depth and color frames captured by the camera, and an RGB array with dimensions matching the depth image filled with zero values, and the Tdv. B) Generate Point-Cloud • Process the depth frame for potential noises: Apply filtering to the depth frame to reduce noise and fill holes. This process involves applying a series of filters to enhance the quality of a depth frame, including decimation, spatial, temporal, hole-filling, depth-to-disparity, and disparity-to-depth. Initially, the frame is downsampled to reduce its density. Then, the depth data is transformed to disparity data for more accurate processing. An edge-preserving filter smooths the image while maintaining important depth edges and filling small gaps. Temporal filtering reduces noise by averaging data over multiple frames. Finally, the data is converted back to depth form and further processed to fill any remaining holes. • De-project depth pixel to 3D point: De-project the processed depth pixels to obtain 3D points in the depth camera's coordinate system. By applying the inverse lens distortion function on each depth pixel (i,j,d), the 3D position (x,y,z) of the point is reconstructed as follows: (𝑥, 𝑦, 𝑧) = 𝐷𝑒𝑝𝑟𝑜𝑗(𝑖, 𝑗, 𝑑) = (𝑑. 𝑈𝑚𝑜𝑑𝑒𝑙 ((𝑖, 𝑗) − 𝑃𝐹) , 𝑑) where 𝑈𝑚𝑜𝑑𝑒𝑙: R2 → R2 represents the undo lens distortion function, 𝑃 = (𝑝𝑥, 𝑝𝑦) is the principal point, and 𝐹 = (𝑓𝑥, 𝑓𝑦) is the focal length. 62 C) Reproject in Virtual Camera View: • Transform point-cloud by 𝐓𝐝𝐯: To apply the extrinsic transformation matrix Tdv, we represent each 3D point as a homogeneous coordinate (𝑋𝑑 , 𝑌𝑑 , 𝑍𝑑 , 1)𝑇. To obtain homogeneous coordinate in the virtual camera view (𝑋𝑣, 𝑌𝑣, 𝑍𝑣, 1)𝑇, we use the following equation: [𝑋𝑣𝑌𝑣𝑍𝑣1] = Tdv [𝑋𝑑𝑌𝑑𝑍𝑑1] [𝑋𝑣𝑌𝑣𝑍𝑣1] = [𝑟11 𝑟12 𝑟13 𝑡𝑥𝑟21 𝑟22 𝑟23 𝑡𝑦𝑟31 𝑟32 𝑟33 𝑡𝑧0 0 0 1] [𝑋𝑑𝑌𝑑𝑍𝑑1] • Project 3D points on virtual camera view using perspective division: In order to project the transformed 3D points onto the virtual camera's image plane, we perform perspective division, dividing the 3D coordinates by the depth component (𝑍𝑣). These coordinates are then scaled and translated to fit within the dimensions of the output image, effectively mapping the camera's view frustum to the 2D image plane. • Mask invalid pixels in reprojection: The reprojection process may result in some points falling outside the image boundaries. To filter out these invalid pixel values based on the size of the output image, we obtain a subset of valid pixels. Note that in our case, the output image is (1280 × 720, 2) based on the resolution of the depth frame. To explain the process, we showed an example of a matrix containing the 2D coordinates (i, j) in the virtual camera's view below. The coordinates might include some NaN (not-a-number) 63 values if they correspond to a depth pixel with a NaN depth value or if they fall outside the image boundaries. As an example, (-400, 200) in the example below is invalid. 𝑝𝑟𝑜j =[ 𝑖 𝑗460𝑛𝑎𝑛325⋮−400600𝑛𝑎𝑛450⋮200⋮ ⋮ ] (1280×720,2) To filter out invalid points, we create a mask 𝑚 where each element is True if the corresponding point is valid and False otherwise: 𝑚 =[ 𝑇𝑟𝑢𝑒𝐹𝑎𝑙𝑠𝑒𝑇𝑟𝑢𝑒⋮𝐹𝑎𝑙𝑠𝑒⋮ ] (1280×720,1) Using this mask, we extract the valid 𝑖 and 𝑗 coordinates: 𝑖𝑣𝑎𝑙𝑖𝑑 = 𝑖[𝑚] = [460325⋮](𝑛,1) , 𝑗𝑣𝑎𝑙𝑖𝑑= 𝑗[𝑚] = [600450⋮](𝑛,1) The dimension of valid coordinates of i[m] and j[m] changes (n,1) where n corresponds to the number of True elements in the mask. D) Find Texture Coordinates • Align 3D point to the color frame: Align the 3D point in the depth coordinate frame with the color frame. This step ensures that the 3D points derived from the depth frame correspond to the correct positions in the color frame. 64 • 3D to 2D texture coordinates: Project each 3D point onto the 2D image plane to obtain normalized coordinates (𝑢,𝑣). • Match with image dimensions: Scale the normalized coordinates to match the resolution of the color image and add 0.5 for pixel center alignment. 𝑢𝑣_𝑚𝑎𝑝 = [𝑢𝑝𝑖𝑥𝑒𝑙 𝑣𝑝𝑖𝑥𝑒𝑙](1920,1080) = 𝑖𝑛𝑡[𝑢 × 𝑐𝑤 + 0.5 𝑣 × 𝑐ℎ + 0.5] where cw and ch are the width and height of the color frame, respectively. E) UV mapping: In order to add textures to the image, map the 2D projected coordinates to texture coordinates. Use these texture coordinates to fetch and apply the corresponding color values from the color frame to the projected points. Points that are outside the boundaries or have no corresponding color information due to occlusions will appear as black regions in the reprojected view. 𝑜𝑢𝑡[𝑖[𝑚], 𝑗[𝑚]] = 𝑐𝑜𝑙𝑜𝑟[𝑢[𝑚], 𝑣[𝑚]] 𝑜𝑢𝑡[𝑖𝑣𝑎𝑙𝑖𝑑 , 𝑗𝑣𝑎𝑙𝑖𝑑] = 𝑐𝑜𝑙𝑜𝑟[𝑢𝑣𝑎𝑙𝑖𝑑 , 𝑣𝑣𝑎𝑙𝑖𝑑] where out is the initial input image, initially filled with 0 (black), and now being filled with the mapped color values, and color refers to the color image obtained from the camera. 2.5 Pre-Deployment Calibration Prior to using the system in a procedure, we perform two equipment characterization steps: (1) calibrating the depth camera’s intrinsic parameters, and (2) measuring laser aimer misalignment on the C-arm and making appropriate adjustments. 65 2.5.1 Calibrate Camera Intrinsic Parameters In order to ensure correct mapping of image data to real-world coordinates and precise marker tracking, we perform camera calibration to determine the intrinsic parameters for both the RGB and depth sensors. The process of camera intrinsic calibration corrects any distortions caused by the camera lens. These intrinsic parameters include the focal length, principal point, and distortion coefficients. Camera intrinsic calibration typically involves capturing images of a calibration pattern with known features from various viewpoints and distances. In our application, we employed a ChArUco Board as our calibration pattern as shown in Figure 2.23 due to its superior accuracy compared to individual or multiple ArUCO Markers. We used a program supplied by Intel (the maker of the RealSense camera) to perform this calibration. This program detects the ChArUco markers, extracts their corner locations and symbol IDs, and computes the intrinsic and extrinsic parameters of each sensor (Zhang 2000) where intrinsic parameters of a camera define its internal characteristics, such as focal length, optical center, and lens distortion, while extrinsic parameters describe its position and orientation in the external world. We then write the intrinsic calibration parameters to the camera using the Dynamic Calibration Tool (Intel 2023) to replace the default intrinsic parameters. 66 Figure 2.23 ChArUco Board used for camera intrinsic calibration Once both RGB and Depth sensors are individually calibrated, they can be combined to create a single coordinate system that accurately aligns the captured color and depth information. This alignment ensures that the spatial relationships between the color and depth information are accurately preserved in the resulting point cloud. To assess the accuracy of our camera calibration, we used the toolbox function cv2.projectPoints to calculate the average re-projection error for the corner points of the ArUco Board in the RGB frame, which was found to be below 0.1 pixel. Given that the transverse dimension of the field of view of the RGBD camera is approximately 69°, the typical calibration depth is ~450 mm, and the camera resolution (horizontal) is 1920 pixels, a 0.1 pixel re-projection error corresponds to a 67 dimensional error of ~0.03mm (i.e., ~450𝑚𝑚 × tan69°2× 2 × 0.1/1920), which is considered well under clinically acceptable levels for our typical applications. 2.5.2 Adjust for Laser Aimer Misalignment To mitigate the impact of misalignment of the laser crosshair projector on overlay error discussed in 2.3.1.4, we performed an ad-hoc correction during the overlaying process to shift and rescale the position of the scaled X-ray image by parameters 𝑐, 𝑐𝑥, and 𝑐𝑦. In order to find offset correction parameters, 𝑐𝑥, and 𝑐𝑦 along with the c and D parameters in the X-ray scale model described in 2.3.1.4, we introduced a calibration pipeline that eliminates the need for physical laser aimer calibration. We implemented two manual adjustment features in the user interface which shifts and rescales the X-ray image overlaid on the virtual camera video. These manual adjustments control: 1) horizontal and vertical translation which changes the values of 𝑐𝑥 and 𝑐𝑦 by small increments with each click of the corresponding button in the user interface, and 2) scaling, which changes the parameter of c by small intervals. In order to find the unknown parameters including 𝐷, 𝑐, 𝑐𝑥, and 𝑐𝑦, the user should perform the manual adjustments for two X-ray overlays corresponding to two different depths as shown in Figure 2.24. This step will be done at Overlay Step under a different tab (laser aimer calibration): • First, the user is prompted to select the first depth and apply manual adjustment. The user needs to redo the Select Overlay Depth step to update depth (𝑑1). Then the software navigates the user to the Overlay Step. Then to create the X-ray overlay at this depth, the user can take an X-ray of a reference object (we used the calibration disk developed by Mathew Hickey as the reference object during this study which has markers visible in both video and X-ray) and apply 68 scaling and shifting adjustments so that the corresponding points on the video and X-ray image of the object are properly overlaid. The user then saves 𝑐𝑥 = 𝑐𝑥0𝑑1, 𝑐𝑦 = 𝑐𝑦0𝑑1 and 𝑠(𝑑1). • Second, the user is prompted to select the second depth and apply manual adjustment to the X-ray image overlaid on the video. The user performs the similar step described in the previous step including Select Overlay Depth at the second depth, taking a new X-ray of the image at the new depth to create X-ray overlay at this depth and then performing manual adjustment so that the corresponding points on the video and X-ray image of the object are properly overlaid. The user then saves 𝑐𝑥 = 𝑐𝑥0𝑑2, 𝑐𝑦 = 𝑐𝑦0𝑑2 and 𝑠(𝑑2). • Using the parameters 𝑠(𝑑)., 𝑐𝑥, 𝑐𝑦, and d identified at two depths, we can now use the equations in 2.3.1.4 to find 𝐷, 𝑐, 𝑐𝑥, and 𝑐𝑦. Figure 2.24 Laser aimer calibration protocol 69 We assessed the repeatability of the laser misalignment correction protocol through a series of tests involving laser aimer calibration across twenty distinct pairs of depths with values between 300 to 550mm. The resulting data indicates that the values for 𝑐𝑥, 𝑐𝑦, 𝐷, and 𝑐 fall within the ranges of -0.019 (SD: 0.003), -0.016 (SD: 0.004), 976 (SD: 55 mm), and 0.308 (SD:0.029), respectively. It is unclear how much impact this level of variability might have on the overlay error; in order to estimate this impact, one may need to vary each parameter independently and in combination with others in the model, calculate the resulting overlay error for each scenario, and assess the significance of each parameter on the overlay error. Note that depending on the orthopaedic procedure and visibility of the anatomy within surgical area, the optimal position of the camera around the image intensifier may differ. As a result, we propose four positions around the C-arm, each spaced at nominally 90 degrees from one another. It’s suggested to perform machine calibration at each camera position or alternatively, perform a coordinate transform to adjust for where the camera is mounted. In the former case, to implement the software on various C-arm models at each camera position, we propose to have the parameters saved as a table as shown in Table 2.2 and import this data into the software. 70 Table 2.2 Format for storing machine calibration parameters for various C-arm models and different orientations of the camera around the image intensifier Model Camera Position s D 𝒄𝒙 𝒄𝒚 Model 1 Position 1 𝑠11 𝐷11 𝑐𝑥11 𝑐𝑦11 Model 1 Position 2 𝑠12 𝐷12 𝑐𝑥12 𝑐𝑦12 Model 1 Position 3 𝑠13 𝐷13 𝑐𝑥13 𝑐𝑦13 Model 1 Position 4 𝑠14 𝐷14 𝑐𝑥14 𝑐𝑦14 Model 2 Position 1 𝑠21 𝐷21 𝑐𝑥21 𝑐𝑦21 Model 2 Position 2 𝑠22 𝐷22 𝑐𝑥22 𝑐𝑦22 Model 2 Position 3 𝑠23 𝐷23 𝑐𝑥23 𝑐𝑦23 Model 2 Position 4 𝑠24 𝐷24 𝑐𝑥24 𝑐𝑦24 These laser malalignment adjustment parameters are loaded into the system in the first step of the program, where users are first prompted to define machine-specific parameters, including the C-arm model, camera position, and a choice between operating in developer mode and standard mode. The GUI at this step is shown in Figure 2.25. The laser aimer alignment parameters corresponding to the selected C-arm model and camera position are then loaded for later use. Also, the developer's mode provides access to additional capabilities to adjust for laser aimer misalignment, which are deliberately concealed in the standard method to maintain simplicity and safety when used in the operating room setting. 71 Figure 2.25 DeCAF GUI displaying initial system settings, including C-arm model selection, camera positioning, activation of developer's mode, and the choice to upload calibration data. 2.6 Experimental Evaluations In this section, we present an overview of the experimental evaluation methods employed to redesign the DeCAF system, with the primary objective of enhancing its suitability for use in the operating room (OR). We present experiments that: 1. Estimate the accuracy requirements for the X-ray overlay, 2. Evaluate the overlay error achieved using the calibrations and corrections described above, and 3. Evaluate the usability of the system when draped, as would be required in the operating room. 72 2.6.1 Requirements for X-ray Overlay The performance of the DeCAF system needs to align with the intended purpose to ensure its effective use in real-world medical scenarios. In the context of X-ray overlay applications, a key requirement is the alignment accuracy between the overlaid X-ray image and the video. Studies on X-ray overlays have typically aimed for millimeter-scale accuracy, although different applications could have significantly different requirements. For instance, the system presented by (Navab et al. 2010) achieved an accuracy of ~1 mm in cadaver studies for pedicle screw placement and interlocking of intramedullary nails. A related system presented by (Habert, Meng, et al. 2015) achieved 1.7 mm overlay error on a patient hand using their X-ray overlay system, which included two stereo-RGBD cameras attached to the C-arm gantry. In addition to the overlay error requirements, the system must function with the C-arm in acceptable positions relative to the patient, which we characterize in terms of the distance between the surgical table and the C-arm, as well as the distance between the image intensifier (II) and the patient. According to (Earle C. Gregg 1989; Lakhwani et al. 2019), for a mobile C-arm, the distance between the source and the skin should be greater than 30 cm. Given that the distance between the image intensifier and the X-ray source in mobile C-arms is approximately 100 cm, we estimate that the system should be designed to work effectively when the AOI is positioned within ~30 to 55 centimeters from the C-arm image intensifier. 2.6.1.1 Evaluating Clinically Acceptable Deviation Levels To establish the clinically acceptable range of overlay error, we presented our consulting surgeon, Dr. Pierre Guy, with X-ray images overlaid onto an image of the target anatomy while surgical tools were in use. We focused solely on a proximal tibia plate fixation scenario. By deliberately 73 introducing controlled deviations between the X-ray and video images during the tool overlay process (examples shown in Figure 2.26), we created a spectrum of overlays with varying degrees of deviation with 0.5mm and 1° increments. Subsequently, we presented the overlays to collaborating surgeons and sought their assessments regarding which levels of deviation would be considered acceptable for intraoperative use. Figure 2.26 Varying Deviations between X-ray and the RGB image: A) Horizontal offset, B) Angle offset Video Image X-ray Image x offset ΔA B Video Image X-ray Image Angle offset Δ=8° 74 Given the tool's size and the anticipated quality of the point cloud, the surgeons commented that an accuracy of less than 2mm and an angle offset of approximately 3° would likely be considered suitable for anticipated purposes. 2.6.1.2 Mesh Analysis for Assessing Overlay Error To qualitatively evaluate the accuracy of the X-ray overlay and the impact of X-ray distortion on the overlay error, we used an object with visible markers in videos alongside underlying radiopaque markers as a test object as shown in Figure 2.27. The object consists of small steel pins with round heads, placed inside a foam board marked with a grid. In order to compare the locations of the visible and radio-opaque markers on the overlaid images, we aligned the centre of the object with the laser aimer and created an X-ray overlay, as shown in Figure 2.28. We also placed a K-wire within the field of view to show the typical use case for the system. Note that we have cropped both images for better comparison. Figure 2.27 Test object with visible markers in video and X-ray to evaluate X-ray overlay 75 Overall, it appears that the corresponding points are properly overlaid onto each other. However, there seems to be larger deviations for the points along the top of the overlay. In addition, the K-wire seems to be well-aligned with the actual tool. Figure 2.28 X-ray overlay of the test object at calibration depth: A) Projected image from top view, B) X-ray overlay 2.6.2 Overlay Error Given that the qualitative results from the test described in the section above were positive, we then conducted a laboratory-based study to evaluate the accuracy of the overlay at varying distances to the camera using the object described in Figure 2.27. We performed the initial virtual camera location calibration at a consistent mid-range distance of ~44cm from the image intensifier, which is a typical distance for the primary intended uses. In order to evaluate how accurate the overlay was at distances other than the distance at which the system was initially calibrated, we positioned the grid described in the previous section at a series B A 76 of heights ranging from 300 to 550 mm, chosen to cover the range of distances at which the system is intended to be used. At each height, we adjusted the height of the table, selected the corresponding overlay depth by placing the pointer tool on the table and scanning the corresponding depth of the object, took an X-ray of the object, and created and saved the X-ray overlay. To maintain consistency, the virtual camera calibration parameters remained unchanged across all trials. We measured the overlay error at each depth by computing RMS values for the Euclidean distances between corresponding markers across the grid points in the object. Figure 2.29 illustrates the overlay at three distances: at the closer, calibration, and farther distances. Overall, the corresponding points and overlay error seems to be accurate at the calibration plane, but some discrepancies are visible when moving away from the calibration plane, whether nearer or further. Note that the shifts tend to be downward and to the right when closer (top) and upward and to the left when further away (bottom). 77 Figure 2.29 Vector field plot for distortions of visible marker points in X-ray image at the: A) closer <400 mm (363 mm), B) calibration plane (443 mm), and C) farther >500mm (517mm) distances. 78 Figure 2.30 shows the overlay error for different target distances to the virtual camera plane. The overlay error is higher near the calibration plane, but it gradually decreases as the distance from the calibration plane increases. The overlay error measured in the middle zone around the calibration plane is 1.3 ±0.2mm (95% confidence interval), but the errors increase significantly beyond this range, rising to upwards of 3 mm for distances ~75 mm away from the calibration plane. Figure 2.30 Overlay Error as a function of distance to the virtual camera. The virtual camera calibration was done at a distance of ~440 mm. The dotted red lines represent the 95% confidence interval bounds around the mean value (green dashed line) across the range from 400 to 500 mm. 79 2.6.3 Usability in Clinical Use The Radiology Department in Vancouver General Hospital described two requirements for the system that need to be validated prior to it being approved for use in clinical trials: 1- Safety: Ensure that the device is securely attached to the X-ray tube to minimize the risk of the system being dislodged during surgery when the C-arm is placed in positions typically used in surgery. 2- Sterile Field Cover Compatibility: Verify that the camera attachment to the tube allows for the placement of a plastic cover over the entire C-arm to maintain a sterile field and prevent contamination. Ensure that the cover does not obstruct the camera's view, considering potential transparency issues that may arise. In the following sections, we describe the assessments performed to address these two concerns. 2.6.3.1 Safety in Clinical Use To address the safety requirement, we simulated the system configuration for various potential poses of the C-arm including when the camera assembly approaches the edge of the OR table as shown in Figure 2.31. Overall, the device occupies little space around the C-arm, so there will likely be relatively few situations in which there would be significant risks of the camera being dislodged, though caution should be exercised in cases where the distal end of the image intensifier is brought into close proximity to the operating room table. 80 Figure 2.31. Use of the C-arm equipped with DeCAF at different orientations – The likelihood of the camera hitting the table increases with the C-arm's orbital angle. 81 2.6.3.2 Sterile Field Cover Compatibility In order to assess the requirement of sterile field cover compatibility, we evaluated whether or not the system could function when wrapped with a transparent surgical drape. As discussed earlier in this chapter, during clinical use, the portion of the drape near the camera lenses across the camera face will be flattened and secured to the camera casing using a sterile rubber band to minimize optical distortions due to the presence of the drape in front of the camera. Figure 2.6 illustrates the system when the C-arm is draped. In order to evaluate the impact of plastic drape coverings camera lenses on marker detection quality and point-cloud generation, we qualitatively compared video quality and point-cloud data with and without the drape, as shown in Figure 2.32. While the RGB video stream seems to be slightly blurred compared to the undraped situation, the markers remain visible and detectable. With the presence of the drape, 15 out of 16 markers in the ArUco board were correctly detected, and the board’s position is relatively similar to the scenario with no drape. Minor distortions are observable in the object’s point-cloud, and there is one instance of an erroneously located patch. However, Figure 2.33 displays an overlay of the pointer tool edges in the virtual camera view under two conditions: in the absence of the transparent surgical drape (white) and in the presence of the transparent surgical drape (green), and overall the positions of these edges are well-aligned with one another across most of the tool – particularly in the centre of the image. Overall, these assessments indicate that the system is likely to function acceptably when draped. 82 Figure 2.32 Comparison of the video and point-cloud quality in the absence (left) and presence (right) of the transparent surgical drape. Note some minor blurriness in the draped images. The undetected marker is marked with a red circle in the detected markers image with the drape present. The bottom right image shows one section (upper left; white arrow) where the overlay position is significantly in error. Without drape With drape RGB view Detected Markers Pointcloud 83 Figure 2.33 Pointer tool edges in the virtual camera view, shown in white (absence of transparent surgical drape) and green (presence of transparent surgical drape). 2.7 Summary In this chapter, we presented an overview of all system components, and we demonstrated that the system works as intended, with an overlay error averaging 1.3 mm (+/-) within 50 mm of the calibration plane. In the next chapter, we evaluate the potential utility of the system in a simulated surgical procedure typical of one of the key intended applications. 84 Chapter 3: Performance in Simulated Target Procedures In the previous chapter, we demonstrated that the updated DeCAF system is able to create X-ray/video overlays with accuracy we believe to be appropriate for clinical use, but we do not yet know how useful orthopaedic surgeons will find this image overlay capability to be in various orthopaedic trauma procedures. The purpose of the study in this chapter, therefore, is to enable our collaborating orthopaedic trauma surgeons, Drs. Pierre Guy and David Stockton, to use the system in a selected simulated scenario where they feel the augmented visualization capability may provide a significant clinical benefit in order to obtain preliminary qualitative and quantitative feedback regarding the potential clinical utility of the system. The results of this study are intended to guide design of future clinical studies aimed at evaluating whether there would be benefits found in live operating room scenarios, which would be needed to justify future efforts aimed at translating this technology into routine usage by a broader range of surgeons. To assess DeCAF’s potential to show value in the clinical setting, we started by identifying common surgical scenarios through literature analysis and discussions with experienced surgeons (details presented below). We hypothesized that, by using the DeCAF image overlay system in a simulation of a promising clinical scenario, surgeons will be able to reduce the number of X-ray images acquired and potentially reduce surgical time required for the fluoroscopically guided portions of these procedures. In the following sections, we present the methods employed for this study, elaborate on the simulation setup, and present comparative results obtained through both traditional C-arm. guidance and the DeCAF-assisted guidance. 85 3.1 Selecting Surgical Procedure for Assessing DeCAF’s Potential Benefits To identify surgical procedures suitable for our simulation study, we began by assessing which orthopaedic trauma procedures involve the greatest use of radiation. Rashid (Rashid et al. 2018) reports radiation exposure values measured in a series of 808 orthopaedic trauma procedures reported in a large tertiary care hospital over a period of 18 months. Table 3.1 summarizes the frequency and median DAP (Dose-Area Product) for five surgeries with relatively higher number of fluoroscopy images adapted from (Rashid et al. 2018). Table 3.1 Frequency and type of trauma operations requiring fluoroscopy use including median dose area product, screening time, and number of fluoroscopy images taken for trauma cases where n > 9. n = Percentage (of 808 procedures performed) Median DAP (mGy.cm2) Median screening time (mm:ss) Median no. of fluoroscopy images Neck of femur with dynamic hip screw (DHS) 190 23.5% 667.5 00:36 65 Proximal femoral nail (Short) 75 9.3% 1040 00:49 109 Intramedullary femur nail (long) 39 4.8% 1720 02:36 243 Neck of femur (cannulated screws) 35 4.3% 881 00:56 113 Tibia (ORIF) 19 2.4% 185 00:43 84 Intramedullary tibia nail 14 1.7% 359.5 01:44 202 We also reviewed the number of X‐ray shots and operation time reported in (von der Heide et al. 2018) across various orthopedic procedures involving incision, implant control and for a number 86 of orthopaedic surgeries performed either with the CamC system (described in Chapter 1) or traditional C-arms. After consulting with one of our partnering surgeons (Dr. Pierre Guy), we chose to base our initial simulation study on the Open Reduction and Internal Fixation (ORIF) of the proximal tibia, which, according to (Rashid et al. 2018), requires a large number of X-ray images (median of 84) and has a high radiation dosage (median DAP of 185 mGy.cm2). ORIF contributes to 2.4% of trauma operations described in (Rashid et al. 2018). Compared to other high-radiation procedures described in Table 3.1, ORIF’s use of the AP (anteroposterior) position ensures continuous fluoroscopy without frequent C-arm repositioning and facilitates a clear evaluation of the system's effectiveness in minimizing radiation exposure. Additionally, ORIF of the tibia involves fewer procedural steps than complex orthopedic surgeries, enabling a focused assessment of DeCAF's impact on radiation reduction without added procedural complexities. The workflow in this procedure is illustrated in Figure 3.1. We specifically narrowed our focus to the Plate Insertion and Fixation steps, as they are, in the opinion of our consulting surgeon, the ones that contribute most significantly to overall X-ray usage. 87 Figure 3.1 Surgical Steps in Open Reduction Internal Fixation (ORIF) of the Proximal Tibia During the plate fixation step in this procedure (see Figure 3.1), which includes fixing the plate with screws, several substeps are involved (Hanse and Pesántez 2010). While the C-arm is used in an AP orientation, the surgeon adjusts the plate, ensuring its proper alignment with the bone. Subsequently, each screw is inserted, involving drilling and screwing. In the drilling and screwing phase for plate fixation, X-ray shots are commonly captured to ensure the proper alignment of the tool within the joint. Furthermore, as the drilling or screwing process advances, X-rays are taken periodically to ensure that the trajectory avoids contact with the adjacent joint and to monitor the depth and length of the tool's progression. 88 3.2 User Study Overview: In order to evaluate the utility of the DeCAF system, we carried out a user study in a simulated surgical setting involving two collaborating orthopaedic trauma surgeons who performed proximal tibial plate insertions on anatomic models using both the conventional approach and the DeCAF system. We evaluated relative performance based on the number of X-rays obtained, screw placement descriptors, and time used. In the following sections, we present details of the study design, data acquisition protocols, and data analysis processes. 3.2.1 Study Design In this user study, the two collaborating surgeons (one a More Experienced Surgeon (MES) and the other a Less Experienced Surgeon (LES)3) each performed a total of 20 proximal tibia plate insertions, each involving one K-wire insertion and three screw insertions. Ten of the plate insertions were performed with the conventional fluoroscopically guided approach and ten with the DeCAF system, in alternating order. Each insertion took approximately 10 minutes, and the complete set of insertions was performed over multiple sessions as their clinical schedules allowed (3 sessions for the LES, 6 for the MES). 3.2.2 Anatomic Models After consulting with Dr. Pierre Guy, we decided to use intact rather than fractured bone models because our focus is on the steps after reduction, where the fractured bones are held tightly together, typically using K-wires or other similar temporary fixation devices. We used a surgical 3 MES: Pierre Guy, 26 years as an attending surgeon; LES: David Stockton, 1 year as an attending surgeon 89 bone model of the tibia from Synbone to represent the anatomical target. We selected a right tibia that measures ~39 cm in length and features a foam cortical shell that is claimed to exhibit more realistic tactile experience during cutting and drilling compared to plastic models. It incorporates denser foam to emulate the inner cancellous material and has a 12 mm diameter intramedullary canal, as shown in Figure 3.2, which emulates to a reasonable extent the typical appearance on X-ray images. Figure 3.2 A) AxSOS proximal tibial plate (Stryker) and B) Tibia Model: Cortical / hard cancellous bone, 8 mm Diameter Canal, and 387 mm Overall Length (Synbone) To present an external appearance that emulated the fact that the bone is not visible during surgery due to the presence of overlying soft tissues, we covered the bone with a piece of foam, sculpted to approximate the proximal tibia’s soft tissue envelope, and intentionally removed a section of soft tissue close to the tibial plateau to emulate the surgical exposure used to enable plate insertion 90 and fixation (see Figure 3.3). We also created space on the lateral side to allow for inserting and removing the synthetic bone specimens to allow re-use of the soft tissue cover for the different specimens. Figure 3.3 Foam model used to wrap the tibial models used in the study and associated anatomical aspects of the tibia 3.2.3 Experimental Setup To emulate the operating room setup, we used a simulated surgical suite that included a C-arm, surgical table, tibia model, and soft tissue model with an incision at the desired region as shown in Figure 3.4. 91 Figure 3.4 Surgical Tools used for simulated proximal tibia plate fixation procedure including power tool (A) used for drilling and screwing, K-wire (B), drill bit (C), sleeve (D), and screwdriver attachments (E), cortical screws (G), fixation plate (H) 4. Specimen Setup and Positioning: For each trial, we inserted the bone model within the soft tissue cover and placed the model on the surgical table ensuring that it was nominally aligned with the C-arm image intensifier. We used tape and blocks of wood fixed to the table with a vise to secure the foam model in place as shown in Figure 3.5. In all tests, we maintained a distance of approximately 45cm from the C-arm to the model in an anterior-posterior (AP) position, which allowed us to visualize the relevant portions of the proximal tibia in the video overlay. Furthermore, since all screws were relatively close to each other, we performed the depth measurement step only once for the middle screw, which is 4.5 cm away from the calibration plane 4 AxSOS proximal tibial plate and screws along with instruments provided in-kind by Stryker Canada A B C D E F G H 92 (operating room table). According to the findings in Chapter 2, this distance is within ±5 cm of the calibration plane. Figure 3.5 Soft Tissue Placement and Alignment with the C-arm’s axis on the Surgical Table The procedure for performing the plate insertion and fixation procedure using conventional approach is illustrated in Figure 3.6: 93 Figure 3.6 Workflow in plate insertion and fixation step in proximal tibia plate fixation: A) K-wire insertion, B) Drilling, and C) Screwing. Note that drilling and screwing process is repeated three times – once each for the middle, anterior, and posterior screws. System Initialization: Before starting the experiment, we ensured the C-arm was turned on and functioning. For each test, we started a new C-arm case to "zero" the exposure dose reading. We captured an anteroposterior (AP) view X-ray of the proximal tibia to confirm appropriate C-arm position and made any adjustments as needed. We performed calibration to locate the virtual 94 camera as discussed in section 2.4.1.1 for cases using DeCAF once before starting each case, and we generated an overlay on a test image to check that the system was functioning properly. We then set up the video camera to record the entire procedure for subsequent analysis. The video camera was placed on a stand so that the C-arm monitor and surgical scene were both visible and started recording before the surgeon started the procedure. We noted the number of X-rays and cumulative radiation dose during this step. This data was used to exclude these X-rays from those measured during the main steps of the procedure. K-wire Insertion: As shown in step A in Figure 3.6, while conducting the experiment, the surgeon inserted the plate at the desired position on the bone model, confirming placement using the C-arm. They fixed the plate with a K-wire, captured an X-ray after the K-wire insertion, and noted the radiation dose and the number of exposures. Our study protocol discouraged reliance on the position of the initial K-wire for guidance, instead encouraging the surgeons to rely on the X-rays or X-ray overlays to guide the screw positioning. Therefore, we encouraged surgeons to refrain from aligning the K-wire to a target, thus eliminating the need for X-rays in this step. The surgeon initially used the conventional approach to insert screws for the first specimen and subsequently employed the DeCAF system for the next specimen, alternating thereafter for the remaining specimens. Screw Placement: For each screw position, the surgeon performed the following steps: 1. Drilling: A brief overview of steps involved in the drilling phase is shown in Figure 3.6 B. The surgeon inserted the drill bit into the power tool and positioned the drill in the desired hole at the proximal portion of the plate, aiming for a safe (sub-articular) drilling trajectory. In the conventional approach, the surgeon captured X-rays during drill advancement to ensure 95 accurate trajectory tracking and avoid incursion into the joint space; during this process, we noted the radiation dose and recorded the number of exposures. In the DeCAF approach, the surgeon relied solely on the real-time overlay based on the initial X-ray captured after K-wire insertion as shown in Figure 3.7 and did not obtain additional X-ray images during insertion. In order to track where the tool is heading, surgeons relied on the visible portion of the tool in the overlay and visually extrapolated the tool's trajectory within the bone. Figure 3.7 Overlay shown to surgeon while positioning the sleeve and the drill 2. Screw Insertion: As shown in Figure 3.6 C, after completing the drilling, the surgeon removed the drill bit and attached the screwdriver attachment. In the conventional approach, when positioning the screw after inserting it into the power tool, the surgeon acquired an X-ray to confirm its position. We took note of the radiation dose and recorded the number of exposures. 96 In the screw insertion using the DeCAF approach, similar to the drilling procedure, the surgeon relied solely on the X-ray overlay. The surgeon repeated the described steps for a total of three screw insertions on the plate at the proximal tibia (middle, anterior, and posterior, in order) as shown in Figure 3.8. During this process, we recorded a video of the procedure and noted the radiation dose and number of exposures after each screw placement. After all screws were inserted, we removed the specimen and examined it for any visible breaches, noting any that occurred. Figure 3.8 Overview of key steps in the procedure The main actions performed in the various steps described in Figure 3.8 were categorized as follows: • Adjusting Plate: This category pertains to the surgeon's actions in placing the plate inside the foam model and ensuring its proper alignment with the bone. • Setting Tool: This group encompasses the steps involved in preparing the surgical tool, including the time taken to assemble components such as the K-wire, drilling bit, screwdriver, 97 and the screw itself, for use in the procedure. Note that Setting Tool step for Inserting K-wire has been excluded for all cases. This exclusion stems from instances where the K-wire was pre-inserted into the tool at the end of a completed case, eliminating the necessity for tool preparation during the next case. • Aligning Tool: This group includes actions that align the tip of the tool to the target hole occurring either immediately after tool preparation or during the drilling or screwing stages before monitoring X-ray or overlay. • Monitoring X-ray: This group accounts for the time required to take an X-ray shot and monitor the resulting image. • Checking Overlay: This category includes the time spent examining the overlay between positioning steps. • Inserting the K-wire: Actions specifically focused on the insertion of the K-wire inside the power tool. • Drilling: Actions related to active use of the surgical power tool for drilling. • Placing Screw: Actions related to the active use of the surgical power tool for screw driving. 3.2.4 Data Acquisition and Analysis We evaluated performance for each approach based on screw positioning descriptors, the number of X-rays obtained for each step in the procedure, and time used. Screw positioning descriptors: To evaluate the screw placement in the proximal tibia, we assessed the hole positions on the opposite (medial) side of the bone from which the screw was inserted, as shown in Figure 3.9. For each hole, we measured the proximo-distal distance from the 98 top part of each hole to the edge of the tibial plateau on the medial side. As described above, we also assessed the specimen for breaches and detectable differences in breach counts. Figure 3.9 Hole locations labeled as M (Middle), A (Anterior), and P (Posterior) on the entry side (lateral condyle) and the exit side (medial condyle) of a synthetic bone specimen. Number of X-rays: The X-ray count was determined post-procedure by reviewing the saved images on the computer. We then recorded the number of X-rays for the four primary steps in the procedure including K-Wire Insertion, Middle Screw Insertion, Anterior Screw Insertion, and Posterior Screw Insertion as illustrated in Figure 3.8. We did not include the number of X-rays used for the initial C-arm positioning in the count as this step is not affected by the use of DeCAF. Time: We analyzed the video recording of each procedure and identified the time required for each of the four primary steps in the procedure, removing any pauses or gaps that we judged to be unrelated to the procedure, e.g. when surgeons paused the procedure for discussions or explanations. Given the relatively close vertical proximity of the screws to one another, the 99 surgeon performed the depth measurement step to scale the X-ray image overlay once per specimen before starting the procedure and not for each screw placed. We excluded the pre-operative system setup times, including the pre-operative calibration process and depth measurement step using the pointer tool, as we performed both steps pre-operatively in this specific study. The calibration process and depth measurement step took about 10 seconds each. Note that this time does not include the pre-deployment calibration for adjusting the laser aimer as this step was already performed before the simulated study. 3.3 Results In the following sections, we compare performance with the conventional and DeCAF methods in terms of exit hole depth, radiation used, and time taken for each surgeon. 3.3.1 Breach Count and Depth of Screw Placement Figure 3.10 shows the exit hole depths for the three screw positions. The grey zone illustrates the typical range of depths beneath the tibial plateau targeted in the conventional approach and is determined by identifying the minimum and maximum exit hole depths in conventional cases performed by the MES. Only one tibial plateau breach occurred (in the posterior location in the first case using DeCAF performed by the LES – see bottom right image in Figure 3.10). 100 Figure 3.10 Exit hole depths using conventional (left) and DeCAF (right) approaches done by the MES and LES. Points are sorted by the order of completed cases from left to right. No breaches occurred with either technique in the cases performed by the MES, and only one breach occurred in the first specimen performed by the LES using the DeCAF approach. On average, the MES and LES inserted the screws ~3-4 mm deeper when using DeCAF (2.9 ± 1.8 mm and 3.8 ± 2.3 mm (p= 0.37 using Student’s t-test), respectively). MES tended to place the screws more deeply than the LES (1.7 ± 2.1 mm and 0.9 ± 3.2 mm deeper than the LES when using the conventional and DeCAF approaches (p= 0.40 using Student’s t-test), respectively). Conventional DeCAF MES LES 101 The MES and LES placed the anterior screws 3.2 ± 1.8 mm and 4.0 ± 2.6 mm deeper, respectively (p=0.43) when using the DeCAF. For middle screws, the MES and LES placed the screws of 3.1 ± 1.5 mm and 4.8 ± 2.2 mm deeper, respectively (p=0.049) when using DeCAF. Similarly, the MES and LES placed posterior screws 2.5 ± 2.1 mm and 2.4 ± 1.2 mm deeper, respectively (p=0.96) when using the DeCAF. In all cases, the anterior screw was the deepest, and the posterior screw the shallowest. To determine if any of these observed trends were statistically significant, we performed a three-way ANOVA examining the effects of the following three main factors: surgeon (MES vs LES), approach (conventional vs DeCAF), and screw position (anterior, middle, or posterior) on the depths of the exit holes; the results are shown in Table 3.2. Table 3.2 Effects of Surgeon, Method, and Screw Type on Exit Hole Depth in Three-Way ANOVA Analysis (significant findings bolded and starred) Effects Sum of squares df F Prob > F Surgeon 51.4 1 12.2 6.81E-04 * Approach 332.8 1 79.2 1.47E-14 * Screw 193.3 2 23.0 4.77E-09 * Surgeon * Approach 5.27 1 1.26 0.27 Surgeon * Screw 26.2 2 3.12 0.048 * Approach* Screw 12.7 2 1.52 0.22 Surgeon * Approach * Screw 4.18 2 0.50 0.61 102 The Surgeon factor demonstrated a significant main effect on exit hole depths (p < 0.001), supporting our observation that the MES placed the screws more deeply than the LES. Similarly, the Method factor showed a significant main effect (p < 0.001), confirming that screws placed using DeCAF were deeper than those placed conventionally. There was also a statistically detectable influence of screw position on depth (p < 0.001), again confirming our observation that the anterior screw was the most deeply placed. Using Tukey's Honestly Significant Difference (HSD) (Tukey 1949) as a post-hoc analysis method to evaluate pairwise differences between group means after ANOVA, we found that Anterior screws exhibited a statistically significant shorter depth compared to both Middle and Posterior screws, with p-values of p=0.0208 and p<0.001, respectively. Similarly, Posterior screws displayed a statistically significant shorter depth compared to Middle screws, with a p-value of p=0.055, which is close to the conventional significance level of 0.05. We observed an interaction effect between surgeon and screw (p = 0.048), indicating that the relative depth of placement of the various screws differed slightly between the surgeons, though the absolute size of these differences (< 1mm) was likely negligible in practice. None of the other interaction effects, including surgeon and method (p = 0.26), method and screw type (p = 0.22), along with the three-way interaction (p = 0.61), were statistically significant. In addition to evaluating whether there were effects of the various factors on the mean screw depths, we used Levene's test (Levene 1960) (less sensitive than the Bartlett test to departures from normality (Bartlett 1937)) to determine if there were differences in the variances of the exit hole depths between the four possible combinations of conventional vs DeCAF approaches and performed by the MES vs the LES for each of the three screw positions. None of these tests proved 103 to be significant (p values of 0.18 (anterior), 0.22 (middle) and 0.10 (posterior), so we conclude that there is insufficient evidence to conclude that the variances differ across the various test conditions. 3.3.2 Radiation The average number of x-ray shots taken when using the conventional method was 39.2 ± 5.1 (vs 2 with DeCAF, p<0.05) and 21.5 ± 8.3 (vs 2 with DeCAF, p < 0.05) for the MES and LES, respectively. With the settings used (0.56s of fluoroscopy exposure per shot), this corresponds to 22 and 12 s of exposure for the MES and LES, respectively. Figure 3.11 presents a detailed breakdown of X-ray usage per case for the MES and LES, highlighting the number of X-rays used in each of the four major steps of the procedure: K-wire insertion, and middle, anterior, and posterior screw insertions, while Figure 3.12 compares the average number of X-rays used in each screw placement in conventional cases done by the MES and LES. Overall, it appears that the number of X-ray images acquired by the MES is somewhat greater than the number acquired by the LES, and the variability in number of images appears to be greater for the LES than the MES. The three cases where the LES took the greatest number of images occurred in the first four cases. These impressions are confirmed in Figure 3.12, which shows that the MES took approximately the same number of images for each screw, while the LES appears to have taken a smaller number of images for the anterior screw. 104 Figure 3.11 X-Ray usage throughout procedure stages using the conventional method. Note that the surgeon places the middle screw first. Figure 3.12 Number of X-rays required for each screw positioning step when performing conventional cases A two-way analysis of variance (ANOVA) to test the main effects of surgeon and screw positions on the number of X-rays required for each screw when using the conventional approach is summarized in Table 3.3. Consistent with the impressions described above, the analysis shows a detectable difference in the number of X-rays used by the MES and LES (p < 0.05). However, no MES LES LES MES 105 detectable difference was found in numbers of X-rays used among the three screw positions (p-value: 0.17), nor was there any detectable interaction between surgeon and screw positions (p-value: 0.71). Table 3.3 Effects of Surgeon and Screw Type on number of X-rays per step in Two-Way ANOVA Analysis Effects Sum of squares df F Prob > F Surgeon 582.8 1 70.90 2.1E-11 Screw 30.4 2 1.85 0.17 Surgeon * Screw 5.7 2 0.35 0.71 Residual 443.9 54 NaN NaN To further evaluate whether there were any clinically relevant differences between the number of X-rays taken for the different screw positions, we ran three pairwise Two One-Sided t-Tests or TOSTs (Middle vs Anterior, Middle vs Posterior, and Anterior vs Posterior), with an alpha level of 0.05. As the total number of X-rays for each surgeon differs (MES ~39 and LES ~21), the average number of X-rays per screw position was 13 and 7 X-ray shots, respectively. For the MES, the TOST showed that the number of X-ray shots could be considered equivalent within a range of ±22% (2.8 shots), while for the LES the range would be ±70% (4.9 shots). Overall, it seems that the number of X-rays across different screws for both surgeons are not equivalent the defined ranges considering an equivalence interval of 10% of mean number of shots per screw position for each surgeon. 106 3.3.3 Procedure time The videos were processed via manual examination over a period of three months. To ensure that the labelling process was consistent throughout that period, we performed a repeated assessment of the second case performed by the MES. We found only 1 second of difference in duration of two consecutive labels, one being 1 second longer and one being 1 second shorter. Considering the whole procedure to be around 500 seconds, we considered this to have minimal error. 3.3.3.1 Times for Major Steps Figure 3.13 shows the times required for each of the four main steps in the procedure for both the conventional and DeCAF approaches. Two-sided t-tests showed no detectable difference in almost all of the major steps including K-wire insertion (MES: p = 0.77, LES: p= 0.18), middle screw placement (MES: p = 0.19, LES: p= 0.19), and posterior screw placement (MES: p = 0.49, LES: p=0.21). For the anterior screw placement, no detectable difference in the time required between the conventional and DeCAF approach was found for the MES (p= 0.59), but a detectable difference was observed for the LES (p = 0.04), though the difference was relatively modest (~21s against a baseline of about 98s). 107 Figure 3.13 Time required for each major step in proximal tibia plate fixation using the conventional approach (left bar within each pairing) and the DeCAF approach (right bar) for the Most and Least Experienced Surgeons (left and right subplots, respectively). 3.3.3.2 Detailed Timing To gain further insight into how the two surgeons performed the task using each of the conventional and DeCAF approaches, we extracted detailed timing diagrams for each case from the video recordings (see Figure 3.14). Overall, these diagrams illustrate the iterative nature of the procedure, regardless of whether the conventional or DeCAF procedure is being used, with frequent alternations between tool adjustment steps and position monitoring actions. Despite the underlying anatomical model being identical in all cases, there is a reasonable amount of variability across the different executions of the tasks, and one can see that the LES generally moved through the processes more quickly and with fewer iterations than the MES. LES MES 108 Figure 3.14 Sequential Timeline of Actions in the Workflow using the Conventional or DeCAF Methods and performed by the MES or LES - K-wire Insertion (K), Middle Screw Insertion (M), Anterior Screw Insertion (A), and Posterior Screw Insertion (P) Conventional DeCAF MES LES 109 To better understand the relative amounts of time taken for each surgical action between the two approaches (Conventional and DeCAF), as well as the total procedure time, we aggregated the times from the video records shown above – the results are shown in Figure 3.15. Figure 3.15 Aggregated time data per category for Conventional method (shown on top) and DeCAF method performed by the MES and LES MES LES 110 Using DeCAF is expected to substantially replace the surgical action of Monitoring the X-ray with Checking Overlay. It is possible that that checking the overlay may take less time than monitoring the X-ray. However, in this early prototype, there is a processing delay that limits the frame rate of the overlay. This means the surgeon needs to wait a perceptible length of time for the overlay to update every time they reposition the tool (~1s). Therefore, we investigated whether the total procedure times for monitoring the X-ray and checking the overlay are equivalent, considering the anticipated delay in the DeCAF system's response time. We did this by evaluating (1) whether overall procedure time differed between approaches, (2) whether monitoring the X-rays when doing the conventional procedure took more time than the combination of monitoring the x-rays and checking the overlay when using DeCAF, and (3) whether the mean times taken for the other surgical actions were essentially equivalent. 3.3.3.2.1 Overall Procedure Time In order to compare the overall procedure time between the two methods, we added the depth measurement step, which took an average of 12 seconds, to the total procedure time of all cases performed using the DeCAF approach. Although we performed this step preoperatively in all trials by scanning the pointer tool indicating the middle screw, it could be done intraoperatively as part of the whole procedure. This adjustment allows us to accurately compare the total time for the intraoperative steps in the conventional and DeCAF approach. We did not find a detectable difference in total procedure time for the MES and LES using the conventional and DeCAF approach (MES: 498.8±57.2 vs 487.2±83.2s, p = 0.72 and LES: 370.4±102.2 vs 312.2±50.4s, p = 0.13). The two-way ANOVA examining the effects of Surgeon and Approach on Total Time is summarized in 111 Table 3.4, which indicates that the factor 'Surgeon' has a detectable effect on the total procedure time (p < 0.0001). However, the factor Approach (p = 0.16) and the interaction between Surgeon and Method (p = 0.34) do not have detectable effects on the total procedure time. Table 3.4 Effects of Surgeon and Method on the Total Time in Two-Way ANOVA Analysis (significant findings bolded and starred) Effects Sum of squares df F Prob > F Surgeon 230,128.9 1 39.69 2.77e-07 * Approach 12,180.1 1 3.79 0.16 Surgeon * Approach 5428.9 1 0.94 0.34 208,727.2 36.0 NaN NaN To assess the equivalence of the total procedure times using each approach, we conducted a TOST using an alpha level of 0.05 and an equivalence margin of ~1 minute, which is a value we feel surgeons would likely consider to be roughly equivalent in terms of impact on clinical practice (for a task that has a baseline duration of ~5 minutes in this simulated scenario). The results indicate that neither the MES and LES demonstrated equivalent total procedure times between the conventional and DeCAF approaches within one minute (MES: p(lower)=0.07, p(upper)=0.02), LES: p(lower)=0.48, p(upper)=0.002)5; however, the equivalence tests did show that the MES demonstrated equivalence within a range of 68s and the LES within 121s. 5 p(lower) and p(upper) refer to the p-values for the lower and upper bounds of the equivalence interval, respectively. Both p-values must be below the alpha level to demonstrate equivalence. 112 3.3.3.2.2 Monitoring X-rays vs Checking Overlay The two-way ANOVA results examining the effects of Surgeon and Approach on the combined X-ray or overlay monitoring time are summarized in Table 3.5, which suggest that both Surgeon (p<0.001) and the interaction between Surgeon and Approach (p ~ 0.01) have detectable effects on the combined X-ray and overlay monitoring time, while the Approach alone does not show a detectable impact (p=0.63). Table 3.5 Two-Way ANOVA Analysis examining the effects of Surgeon and Method on Combined X-ray or Overlay Monitoring Time (significant findings bolded and starred) Effects Sum of squares df F Prob > F Surgeon 40576.9 1 107.55 2.33e-012 * Approach 90.0 1 3.79 0.63 Surgeon * Approach 5428.9 1 7.48 9.62e-03* 13582.6 36.0 NaN NaN In addition, to assess the equivalence of the combined monitoring time using each approach for each surgeon, i.e., Monitoring X-ray in the conventional approach vs Monitoring X-ray and Checking the Overlay in the DeCAF approach (MES: 106.0 ± 13.7 vs. 125.8 ± 29.2, LES: 59.1 ± 14.9 vs. 45.3 ± 9.6), we conducted a TOST using an alpha level of 0.05 and an equivalence margin of 10% of the total baseline (~10s and 6s for the MES and LES). The results indicate that neither MES nor LES took equivalent X-ray or overlay monitoring time within the defined margin when using the conventional and DeCAF approaches. 113 3.3.3.2.3 Time Differences for Other Surgical Actions We compared the aggregated times across different categories between the conventional and DeCAF methods for each surgeon using two-sided t-tests. In cases performed by the MES, for the categories of Adjusting Plate (p-value = 0.75), Setting Tool (p-value = 0.56), Aligning Tool (p-value = 0.60), Inserting K-wire (p-value = 0.62) and Drilling (p-value = 0.35) no statistically detectable differences were observed between the two methods (p > 0.05). However, detectable differences were found in the category of Placing Screw (p-value = 0.04), indicating that the time taken for this activity was somewhat faster with DeCAF than with the conventional approach (~28s relative to a baseline of ~106s, p=0.04). In cases performed by the LES, for the categories of Adjusting Plate (p-value = 0.47), Setting Tool (p-value = 0.55), and Inserting K-wire (p-value = 0.51), and Placing crew (p-value = 0.45) no statistically detectable differences were observed between the two methods (p > 0.05). However, detectable differences were found in the categories of Aligning Tool (~15s relative to a baseline of ~55s, p =0.01), Drilling (~22s relative to a baseline of ~54s, p =0.01) between the conventional and DeCAF approaches (p < 0.05). 3.4 Summary In this chapter, we demonstrated the potential clinical utility of the DeCAF system in a simulated proximal tibia plate fixation. We showed that the system could reduce the number of X-ray images from an average of 39.2 for the MES and 21.5 for the LES, down to 2-3 X-rays with largely equivalent overall times (~16% faster in the case of the LES) and similar screw placement performance (slightly deeper placement on average with DeCAF, but similar variability between insertions). These insights will be used to guide future studies in live operating room scenarios to 114 evaluate potential clinical benefits of using DeCAF. In the next chapter, we discuss the main findings of this study and provide recommendations for future work. 115 Chapter 4: Discussion and Conclusions In this chapter, we discuss the results of the system development reported in Chapter 2 and the user study presented in the previous chapter and then summarize the significance of these findings, make recommendations for future work, and identify key thesis contributions. 4.1 Key Findings The overall global objective of this thesis is to evaluate whether DeCAF can potentially reduce radiation use in orthopaedic procedures in the operating room. As stated in Chapter 1, our study aims to: • Refine DeCAF’s hardware design to be used in operating rooms • Assess DeCAF's overlay error. • Evaluate DeCAF effectively in a simulated operating room environment and determine if DeCAF can decrease the dependence on X-ray images during targeted surgical procedures. Overall, we successfully refined and improved the key hardware and software components, demonstrated acceptable technical accuracy, and showed in a simulated proximal tibia plate fixation procedure the feasibility of using DeCAF to reduce radiation exposure in a relevant surgical procedure. In the following sections, we describe the main findings for each of these objectives. 4.1.1 Hardware Components In order to comply with operating room standards, we refined the hardware design to address usability, sterile field requirement, reprocessing protocol, and electrical safety requirements 116 • We designed a camera assembly that can easily be assembled and mounted on the image intensifier. The device was tested by the biomedical engineering department, where they observed the process of mounting the hardware onto the image intensifier and tested the rotation mechanism of the camera mount. • To address the sterile field requirements, the system is used behind the plastic drape where the portion facing the camera lens is flattened using a rubber band. We tested the system under the surgical drape and demonstrated acceptable performance in marker detection during the procedure, along with acceptable point-cloud rendering. • We developed a reprocessing protocol to sterilize the system between uses. This process was approved by the Infection Prevention and Control (IPAC) department in Vancouver Coastal Health Research Institute. • In terms of electrical power, all components requiring power operate on safe 5V power supplies, in accordance with biomedical engineering protocols. Thus, the department approved the electrical safety of the system. 4.1.2 Technical Verification: Overlay Error In a study involving the use an object with radiopaque markers and measuring overlay at 32 depths, we showed accuracy that would likely be acceptable for clinical use (1.3±0.2mm with 95% confidence interval) within ± 5cm of the calibration plane (44cm away from virtual camera plane). This value is comparable with values reported in the literature including overlay error of 1.7mm reported in (Habert, Meng, et al. 2015) which we judged to be sufficient for our simulated lab study. 117 4.1.3 Simulated Lab Study In order to evaluate the clinical utility of the DeCAF system, we carried out a user study in a simulated surgical setting involving two collaborating orthopaedic trauma surgeons who performed proximal tibial plate insertions on anatomic models using both the conventional approach and the DeCAF system. In this section, we describe the main results in the following order: radiation usage, breach count and depth of screw placement, and procedure time. 4.1.3.1 Radiation With DeCAF, 2-3 X-ray shots were acquired after initial placement of the plate and for final monitoring in procedures performed by both surgeons. Compared with an average of 39.2 ± 5.1 and 21.5 ± 8.3 shots taken by the Most and Least Experienced Surgeons (MES and LES), respectively, in the conventional approach, this substantial reduction in the number of X-rays translates directly to minimized radiation exposure for both patients and surgical staff. We have not shown tight equivalence in the number of X-rays taken for the different screw positions when using the conventional approach for either surgeon. The smallest equivalence bounds found in the TOST were 2.8 and 4.9 X-ray shots for the MES and LES, respectively, suggesting that the current study is likely underpowered to demonstrate any differences that might exist in the challenge involved in placing screws in different locations. However, the large bound found for the LES might be due to some outliers, such as case 4 done by the LES as shown in Figure 3.14. 118 4.1.3.2 Breach Count and Depth of Screw Placement No breaches occurred with either technique in the cases performed by the MES, and only one breach occurred in the first specimen performed by the LES using the DeCAF approach, but they generally remained within or in close proximity to the normally targeted depth zone indicating an absence of discernible patterns or trends over time. Overall, screws inserted with DeCAF exited 2.91 and 3.75 mm deeper than with the conventional method for the MES and LES. In this regard, the MES pointed out that they proceeded cautiously while depending on the overlay as the line of tool was not visible. Therefore, it seems that the lack of ongoing feedback during insertion could potentially explain why surgeons erred towards placing screws a bit deeper when using DeCAF. Levene’s test showed that there was no detectable difference between the variances of exit hole depth per screw position achieved by the MES and LES using either the conventional approach or the DeCAF approach. Both MES and LES seem to insert the anterior screw deeper than other screws, regardless of the chosen method (p<0.05). This variation might be related to the shape of the proximal tibia and its positioning under the X-ray. Different positioning angles could potentially improve depth consistency among different screws in future tests. Another potential reason for the deeper positioning of the anterior screw is that we performed the depth measurement step at a fixed distance of approximately 45 mm from the table and did not update it for each screw. A three-way ANOVA analysis showed detectable differences in the exit hole depths achieved by each method (DeCAF or conventional), with screws placed using DeCAF being slightly deeper. However, both surgeons were still able to place the screws at safe distances to the joint by relying on the X-ray. Also, the analysis exhibited an interaction between surgeon and screw position, highlighting surgeon-specific outcomes. This interaction suggests that the influence of screw type 119 on DeCAF response varies depending on the surgeon involved and emphasizes the potential impact of individual surgical practices on the effectiveness of the DeCAF. 4.1.3.3 Procedure time We did not show equivalence in total procedure time within one minute between the conventional and DeCAF approaches for either surgeon, which we considered to be a difference that surgeons might consider clinically meaningful. However, the equivalence bounds found in the TOST were 68 and 121 seconds for the MES and LES, respectively, suggesting that it is unlikely the mean procedure time differences are much more than a minute for the MES and two minutes for the LES, which implies that DeCAF is highly unlikely to extend the overall surgical time (indeed, the mean overall procedure times actually favoured the use of DeCAF for both surgeons, though this difference did not achieve statistical significance). As shown in a sequential timeline of actions using the same method, there was some variation in the time taken in some steps in the procedure in some cases. For example, in the first case performed by the MES using the conventional approach as shown in Figure 3.14 A, the screwing time in the posterior screw took longer compared to other cases. Also, in the fourth case performed by the LES using the conventional approach as shown in Figure 3.14 C, the procedure duration was prolonged, likely due to the surgeon conducting tests after a period of inactivity between sessions, resulting in efforts to reacquaint themselves with the experimental setup. This is evident in the extended durations for steps such as “adjusting plate” and “drilling” compared to the usual timings for these tasks. Following this trial, the overall time appears to have remained relatively consistent. Also, in the first three conventional trials and the first three DeCAF trials, LES positioned the K-wire on a trajectory that was relatively parallel to the intended screw trajectory. 120 This K-wire placement may have influenced subsequent steps and overall outcomes, as the LES pointed out he was trying to follow the trajectory of the K-wire when positioning the screws, which affected the reliance on the overlay. Furthermore, the aggregated time data per category for both methods indicate that, for the MES, the procedural times for tasks such as drilling, plate adjustment, K-wire insertion, etc. are not significantly dependent on the choice of the procedure type (DeCAF or C-arm). However, a detectable difference was observed in the screw insertion process, where the duration tended to be shorter in DeCAF cases compared to C-arm cases. This shows that, for the MES, most tasks took about the same time to be completed regardless of approach (though inserting the screw took a detectably shorter time when using DeCAF). For the LES, the procedural times for tasks such as plate adjustment, tool setting, K-wire insertion, and screw placement are not significantly dependent on the choice of the procedure type (DeCAF or C-arm). However, a detectable difference has been observed in the tool alignment and drilling processes, where the durations tend to be shorter in DeCAF cases compared to C-arm cases. This suggests that for the LES, most categories took about the same time to be completed, with the notable exception of tool alignment and drilling, which showed a significant reduction in time for DeCAF cases (p-values of 0.01 for both categories). We have not shown equivalence in the combined monitoring time using each approach for either surgeon within an equivalence margin of 10% of the total baseline (106.0s for the MES and 59.1 for the LES). Specifically, the results indicate that neither MES nor LES achieved equivalent X-ray or overlay monitoring time within the defined margin when using the conventional and DeCAF approaches. The smallest equivalence margins found were ±39s for the MES and ±25s for the LES, 121 respectively, which represents approximately 37% and 42% of the baselines, respectively. Furthermore, it appears that there exists an interaction between the surgeon and the method. The mean monitoring time using DeCAF was larger than that of the conventional approach for the MES by 18%, while the pattern differed for the LES, where the conventional approach showed a higher mean monitoring time by 23%. This suggests that the impact of the DeCAF approach may vary depending on the surgeon. 4.2 Limitations In the following section, we outline the hardware- and software-related limitations of the updated DeCAF system, as well as those associated with the simulation study. 4.2.1 Hardware Limitations Our DeCAF implementation poses several hardware-based limitations, particularly those related to laser aimer misalignment, use of the RGBD camera, and design for a specific mobile C-arm. 4.2.1.1 Laser Aimer Misalignment One hardware limitation was related to the laser aimer, which should show the X-ray axis position. However, we discovered that the crosshairs were not accurately aligned with the true X-ray axis. To potentially fix this issue, recalibrating the laser aimers is necessary. Yet, this recalibration process is time-consuming and costly. Therefore, we introduced a post-hoc calibration procedure that can be done beforehand for each C-arm device to correct any laser aimer misalignment. Nonetheless, this approach also has its limitations, which will be discussed in Section 4.2.2.1.1. 122 4.2.1.2 Use of the RGBD Camera The current DeCAF system relies on only a single RGBD camera mounted in a single position on the C-arm. As a result, some portions of the reconstructed surgical scene are not visible, resulting in black areas in the display. Furthermore, there are some situations where the normal position for the C-arm would result in the video view being blocked. Both of these issues could be addressed through future hardware improvements. For example, additional cameras could be used (similar to the approach reported in (Habert, Meng, et al. 2015)), which would enable us to capture views of the surgical scene from differing viewpoints which could be merged to more fully reconstruct the scene. Adding one or more additional cameras could also address the situation where the C-arm’s position takes the existing camera out of view. An additional issue is that the camera’s minimum focusing depth is approximately 28 cm, which limits how close the image intensifier can be placed to the patient (although this is a reasonably good match to the normal minimum distance from patient to image intensifier of ~30 cm). Different depth cameras could offer shorter focusing distances, which could possibly further minimize this issue. 4.2.1.3 Design for a Specific Mobile C-arm The current mounting hardware is designed for mobile C-arms with a 12-inch image intensifier and would have to be modified to make the system usable with other C-arms, including those with flat panel detectors and mini C-arms. Overall, however, none of these hardware-based limitations pose particularly significant difficulties and all are anticipated to be relatively easily addressed. 123 4.2.2 Software Limitations There are also several software-based limitations of the current system, including overlay error, computational time, lack of error handling for hardware disconnections, dependence on lighting conditions, as outlined below. 4.2.2.1 Overlay Error The current overlay error was reasonable (~1.3mm) within a zone of ±50mm away from the calibration plane, but this may not be acceptable for all applications. The current algorithms are based on several key assumptions, most notably that the C-arm is aligned in the vertical direction and that all key axes including the X-ray axis and the laser aimer axis (the line forming from both beams intersecting) are nominally aligned with one another. Deviations from these assumptions will cause some errors in overlay error (three sources of such deviations are shown in Figure 4.1). Figure 4.1 A) Misalignment of X-ray, B) Misalignment of laser aimer, C) Table not parallel to image intensifier plane 124 In the following subsections, we elaborate on several limitations that contribute to overlay errors in practical use. These include issues such as post-hoc calibration procedures, misalignment of laser aimers, non-idealities in C-arm structure, distortions in X-ray images, and changes in anatomical position. 4.2.2.1.1 Post-hoc Calibration Procedure Laser Aimer Misalignment One significant source of error was misalignment in the laser aimer, which is supposed to indicate the position of the X-ray axis, but we found that the cross-hairs were offset from the true X-ray axis. As described earlier, we implemented a post-hoc calibration procedure which handles the horizontal and vertical offsets at each plane of interest and can be performed in advance of use for every C-arm device to compensate any misalignment of laser aimer. We demonstrated good consistency in performing this calibration process, and estimate that the residual errors are on the order of below 1mm at the calibration plane. However, the laser aimer could also be misaligned in terms of the angulation of the planes. Therefore, addressing the misalignment merely by shifting the origin may not be sufficient. Examining the intersection of the crosshairs can provide insight into the non-idealities discussed in Figure 4.1. 4.2.2.1.2 Non-idealities in C-arm Structure We only studied our system at AP position. In addition to the non-idealities described in 4.2.2.1, small deformations in the C-arm at different orientations can affect its intrinsic parameters, as mentioned in related studies in the literature (e.g., (Hosseinian et al. 2020)). Based on this work, we estimate that these effects would be on the order of 0-3mm when angular rotation is between 0-45°. 125 4.2.2.1.3 Distortions in the X-ray Image We didn't address distortions in the X-ray image, which could be causing the increasing misalignment observed between corresponding points on the X-ray and video overlays, especially towards the edges of the X-ray image. To mitigate this, implementing calibration procedures to correct geometric distortions could enhance image alignment and accuracy. 4.2.2.1.4 Position Change of the Anatomy In the current implementation, we did not address potential position changes of the anatomy following X-ray acquisition that may occur in the live operating room. In the simulated study, we mitigated this limitation by securing the anatomy to the table using tapes and wooden blocks. However, in future implementations, we plan to develop methods for dynamically tracking and compensating for anatomical movement to ensure accurate and consistent visualization (one approach for dealing with this was presented in (Navab et al. 2012), where the researchers placed a marker on the skin that was visible in both the X-ray and video images; discrepancies indicated that anatomical motion had occurred). 4.2.2.2 Computational Time The current rendering algorithm generates overlays at ~1 FPS on the CPU compared to ~7 FPS reported in (Hosseinian et al. 2020) using a GPU. This causes a slight, but noticeable, delay between live events and the overlay image; however, this delay is deemed manageable by MES. We anticipate that the computational process could be significantly sped up by implementing the core computations on a GPU. 126 4.2.2.3 Lack of Error Handling for Hardware Disconnections During use, there are occasions when USB or other hardware connections fail to function properly during intra-operative operations. In such instances, we temporarily paused the trials to fix the connections. It is advisable to add error handling mechanisms so that these errors are promptly communicated to the user and so that corrective actions can be taken. 4.2.2.4 Dependence on Lighting Conditions The quality of the point-cloud varied with different lighting conditions, affecting noise levels and depth accuracy. With Intel RealSense cameras, this can be managed by adjusting parameters such as exposure, gain, and laser power. However, we faced challenges in consistently achieving high quality. In our simulated lab study, we strategically placed the room lighting to ensure optimal quality. However, in real scenarios, OR lighting depends on surgical requirements, which can affect point-cloud quality consistency. Additionally, we observed that the point-cloud sensitivity to environmental changes, such as variations in lighting and tool type, can lead to shifts in the representation of the tool by a few millimeters. For instance, as depicted in Figure 4.2, positioning our hand underneath the screwdriver caused slight shifts in its position. This misalignment is likely attributable to calibration inaccuracies and is influenced by the object and scene specifics, as well as sensitivity to specular effects affecting depth perception. We speculate that measuring the depth map and observing differences between two images (with hand or a close flat surface underneath the tool and without it) could help identify the source of misalignment. Additionally, using similar objects of the same dimension but different materials could help determine if specular reflections are the cause. Furthermore, we speculate that 127 optimizing camera settings or integrating multiple cameras from different angles could potentially enhance the quality and reliability of the point-cloud data. Figure 4.2 Alignment issue in rendering – A) In the absence of an object underneath the tool, B) When the hand is underneath the tool. 128 4.2.3 Simulated Lab Study Limitations Finally, our simulated lab study poses a number of limitations including the restriction to using a single initial X-ray when using DeCAF, potential learning effects, limited number of participants, and a specific surgical application. 4.2.3.1 Use of Single Initial X-ray When Using DeCAF In the simulation study using the DeCAF system, the participating surgeons were instructed to rely solely on the overlays during the procedure to determine if it were feasible to entirely replace use of the C-arm during the K-wire drilling procedure. In real surgeries, surgeons may opt to take additional x-rays to monitor their progress and provide themselves with additional assurance. As such, the degree of reduction in radiation usage may be understated, and the time needed may be overstated. In future, simulation studies could remove this instruction and evaluate the surgeons’ performance with their preferred level of reliance on x-rays and the DeCAF overlay. 4.2.3.2 Potential Learning Effect The results of the simulated lab study could perhaps indicate a learning effect, as evidenced by some of the longer procedural times shown in Figure 3.14 occurring in the earlier cases, though we did not explicitly evaluate this statistically. It is actually a little surprising that the MES showed shorter procedure times in later cases when using the conventional approach, which they are presumably highly experienced in. To better quantify the ultimate level of performance DeCAF might deliver, future studies could investigate surgeon performance both early in a surgeon’s experience of working with the system and after they have gained more familiarity with it. 129 4.2.3.3 Limited Number of Participants The simulation study involved only two surgeons, so the results should not be interpreted as reflecting the likely levels of performance of all orthopaedic surgeons. As previously mentioned, the study showed that some performance measures varied by surgeon. A study with a higher number of surgeons with broader ranges of experience could guide further modifications or adaptations of the system across clinical contexts and would help us better understand how broadly the findings could be generalized. 4.2.3.4 Specific Simulated Surgical Application The study was also limited to a single initial simulated surgical application, proximal tibia plate fixation procedure, and so was not able to demonstrate potential benefits for other surgical procedures amenable to use of DeCAF, either in simulation or in the live operating room particularly those identified as promising candidates at the beginning of Chapter 3. More specifically, the main focus was on applications where C-arm is used in the AP position. Once the technical performance of the system at other orientations is verified, we would plan to explore other applications where different C-arm orientations are used in order to evaluate the degree to which DeCAF could help reduce radiation and time in these scenarios. In future, we hope to assess performance and potential benefits in other simulated scenarios and in live surgical applications in order to better identify situations where DeCAF can provide the greatest benefits. This approach aligns with the methodology used by (von der Heide et al. 2018) where they evaluated CamC in a range of real orthopedic trauma surgeries, including plate, screw, and nail osteosynthesis, as well as implant removals. 130 4.3 Recommendations for Future Work In addition to the suggestions for future work addressing the specific limitations described above, the system could be further improved in a number of ways, including both improvements to the DeCAF system itself and in integrating DeCAF with other systems. 4.3.1 Improvements to DeCAF Mathematical Model to Overlay the Xray: First, introducing a new mathematical model that directly models a broader range of non-idealities and characterizing the associated parameters could improve the system's accuracy under a wider range of conditions (e.g., at farther distances relative to the calibration plane). Improvements to Machine Calibration: Additionally, improving the machine calibration step could be approached in two ways: (1) representing more general transformations (i.e., adding rotation adjustments to the existing translational adjustments), and (2) incorporating new lasers directly into the hardware design, rather than relying on the C-arm laser aimer, and implementing a protocol to ensure their accurate calibration with respect to the C-arm axis could offer an alternative solution to addressing the current misalignment issues with the C-arm’s crosshairs. Display Tool Path and Tip: To alleviate the issue with portions of the tool interior to the body not being visible in the video feed, both surgeons who participated in the simulated lab study recommended incorporating a feature to display the extension of the tool in the overlay (e.g., by displaying the projection of the drill or K-wire on the X-ray overlay as the tool advances). This feature would allow the surgeon to observe the estimated progress of the drilling or screwing operation between X-ray shots. Implementing this feature requires implementing another tracking component or object detection algorithm applied to the video feed which is able to track or identify 131 the tool and predict the location of the tool’s axis (an example include the drill attachment including spherical markers developed by (Diotte et al. 2015)). Address Position Change of the Anatomy: The current system’s overlay mechanism assumes that the patient does not move following X-ray acquisition, but it is common for minor shifts of the patient to occur due to interactions with the surgeon and their tools. In principle, we would like to track such shifts and, if possible, to compensate for them. The CamC system uses a reference object placed on the patient that incorporates both optically-visible and radio-opaque markers so that discrepancies between the markers are immediately apparent to the user if the patient moves after the X-ray has been acquired (Navab et al. 2012; von der Heide et al. 2018). 4.3.2 Combination with Base Tracking System The DeCAF system has the potential to be combined with a C-arm base-tracking system called OPTIX, which has been developed as a separate project in our lab, to provide new quantitative measurement capabilities. For example, by combining DeCAF with OPTIX, we could generate intra-operative panoramic X-ray images to address the issue of limited field of view in long bone fracture fixations. The CamC system has been used for this purpose (Heining et al. 2009), but required a planar marker pattern to be placed in the operative field and overlapping images to be taken and assembled. By directly tracking the base movement with OPTIX, we could eliminate the need for using a separate marker pattern during the surgical task and enable concatenation of images at greater distances (e.g., to measure leg length during femoral or tibial fracture repairs). 4.4 Summary of Thesis Contributions In this thesis, I made the following key contributions: 132 • redesigned the DeCAF camera mount to address adjustability to different angles while being aligned with clinical standards including electrical and reprocessing. • reimplemented the overlay process through implementing new calibration and depth measurement steps and showed that it has clinically acceptable accuracies. • performed a simulated lab comparing two approaches: conventional and DeCAF approach in terms of the number of X-ray images and time required to complete the procedure, breach count, and depth of screw placement 4.5 Conclusions In summary, we have implemented a significant upgrade to the DeCAF system which has passed key evaluations required to enable it to be tested in live surgeries, and have demonstrated in a reasonably high fidelity surgical simulation of proximal tibial plate fixation that it has clinically acceptable overlay accuracies and can significantly reduce the number of X-ray images required to carry out this procedure, which is expected to be a common application for this system, without significantly changing the time required for the procedure. This finding justifies moving towards testing this system in live surgeries. 133 References © Intel Corporation. Intel® RealSenseTM Depth Camera D435i. [accessed 2023 Sep 26]. https://www.intelrealsense.com/depth-camera-d435i/. AGPtek. AGPTEK USB 3.0 HDMI HD Video Capture Card 1080P 60FPS Game Recorder Box Device Live Streaming for Windows Linux Os X System. [accessed 2024 Jan 24]. http://www.agptek.com/AGPTEK-USB-3-0-HDMI-HD-Video-Capture-1089-212-1.html. Arun KS, Huang TS, Blostein SD. 1987. Least-Squares Fitting of Two 3-D Point Sets. IEEE Trans Pattern Anal Mach Intell. PAMI-9(5):698–700. doi:10.1109/TPAMI.1987.4767965. [accessed 2023 Aug 4]. https://ieeexplore.ieee.org/document/4767965. Bartlett MS. 1937. Properties of sufficiency and statistical tests. Proc R Soc Lond A Math Phys Sci. 160(901):268–282. doi:10.1098/rspa.1937.0109. https://royalsocietypublishing.org/doi/10.1098/rspa.1937.0109. Caesar U, Karlsson J, Olsson LE, Samuelsson K, Hansson-Olofsson E. 2014. Incidence and root causes of cancellations for elective orthopaedic procedures: A single center experience of 17,625 consecutive cases. Patient Saf Surg. 8(1). doi:10.1186/1754-9493-8-24. Carlson VR, Elliott IS, DeKeyser GJ, Pelt CE, Anderson LA, Gililland JM. 2021. Are We Being Fooled by Fluoroscopy? Distortion May Affect Limb-Length Measurements in Direct Anterior Total Hip Arthroplasty. Journal of Arthroplasty. 36(5):1799–1803. doi:10.1016/j.arth.2020.11.021. Chou LB, Johnson B, Shapiro LM, Pun S, Cannada LK, Chen AF, Valone LC, Van Nortwick SS, Ladd AL, Finlay AK. 2022. Increased Prevalence of Breast and All-cause Cancer in Female 134 Orthopaedic Surgeons. J Am Acad Orthop Surg Glob Res Rev. 6(5). doi:10.5435/JAAOSGlobal-D-22-00031. Chua I, Duff J, Munday J. 2023. Elective day of surgery cancellations: A retrospective observational study. Collegian. 30(5):721–726. doi:10.1016/j.colegn.2023.03.013. CMA Foundation. 2019. Number of physicians by specialty and age, Canada, 2019. [accessed 2024 Jul 16]. https://www.cma.ca/sites/default/files/2019-11/2019-02-physicians-by-specialty-age-e.pdf. Definitive Healthcare. 2024. How many orthopedic surgeons are in the U.S.? [accessed 2024 Jul 16]. https://www.definitivehc.com/blog/how-many-orthopedic-surgeons-in-us. Denis A, Montreuil J, Harvey EJ, Berry GK, Reindl R, Bernstein M. 2022. Cancellations and delays of emergent orthopedic operations at a Canadian level 1 trauma centre. Canadian Journal of Surgery. 65(3):E382–E387. doi:10.1503/cjs.009420. Diotte B, Fallavollita P, Wang L, Weidert S, Euler E, Thaller P, Navab N. 2015. Multi-modal intra-operative navigation during distal locking of intramedullary nails. IEEE Trans Med Imaging. 34(2):487–495. doi:10.1109/TMI.2014.2361155. Dorodnicov S, Grunnet-Jepsen A, Wen G. 2023. Projection, Texture-Mapping and Occlusion with Intel® RealSenseTM Depth Cameras. [accessed 2024 Jun 11]. https://dev.intelrealsense.com/docs/projection-texture-mapping-and-occlusion-with-intel-realsense-depth-cameras. Earle C. Gregg C 1977-1983 JSKC 1983-1989 RDAHLKSHLRJNWSPGUVRLSRGW. 1989. Report No. 102. 135 foamcoreprint. 2024. Calibration Targets. [accessed 2024 Jan 25]. https://www.foamcoreprint.com/calibration-targets. Fujibayashi S, Murata K, Shimizu T, Otsuki B, Masamoto K, Shimizu Y, Matsuda S. 2021. An observational survey of nail and skin of spine surgeons-possible damage by occupational ionizing radiation exposure. Spine Surg Relat Res. 5(6):359–364. doi:10.22603/SSRR.2020-0204. Gausden EB, Christ AB, Zeldin R, Lane JM, McCarthy MM. 2017. Tracking Cumulative Radiation Exposure in Orthopaedic Surgeons and Residents. Journal of Bone and Joint Surgery - American Volume. 99(15):1324–1329. doi:10.2106/JBJS.16.01557. GobalData. 2023. North America Trauma Fixation Procedures Count and Forecast to 2030. [accessed 2024 Jul 11]. https://www.globaldata.com/store/report/north-america-trauma-fixation-procedures-analysis/. Habert S, Gardiazabal J, Fallavollita P, Navab N. 2015. RGBDX: First design and experimental validation of a mirror-based RGBD X-ray imaging system. In: Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2015. Institute of Electrical and Electronics Engineers Inc. p. 13–18. Habert S, Meng M, Kehl W, Wang X, Tombari F, Fallavollita P, Navab N. 2015. [POSTER] Augmenting mobile C-arm fluoroscopes via stereo-RGBD sensors for multimodal visualization. In: Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2015. Institute of Electrical and Electronics Engineers Inc. p. 72–75. 136 Hanse M, Pesántez R. 2010. ORIF - Plates without angular stability. [accessed 2024 Jun 9]. https://surgeryreference.aofoundation.org/orthopedic-trauma/adult-trauma/proximal-tibia/partial-articular-fracture-split-depression/orif-plates-without-angular-stability#reduction. Hartley R, Zisserman A. 2004. Multiple View Geometry in Computer Vision. Cambridge University Press. https://www.cambridge.org/core/product/identifier/9780511811685/type/book. von der Heide AM, Fallavollita P, Wang L, Sandner P, Navab N, Weidert S, Euler E. 2018. Camera-augmented mobile C-arm (CamC): A feasibility study of augmented reality imaging in the operating room. International Journal of Medical Robotics and Computer Assisted Surgery. 14(2). doi:10.1002/rcs.1885. Heining SM, Wang L, Navab N. 2009. Clinical Application of CAMC - Long Bone X-Ray Image Stitching. p. 2323–2325. [accessed 2024 Jul 11]. https://link.springer.com/10.1007/978-3-642-03882-2_617. Hickey M, Guy P, Hodgson AJ. 2019. Depth Camera Augmented Fluoroscopy with Video Overlay. p. 173–168. https://easychair.org/publications/paper/ZDBk. Hijikata Y, Kamitani T, Yamamoto Y, Itaya T, Kogame T, Funao H, Miyagi M, Morimoto T, Kanno H, Suzuki A, et al. 2021. Association of occupational direct radiation exposure to the hands with longitudinal melanonychia and hand eczema in spine surgeons: a survey by the society for minimally invasive spinal treatment (MIST). European Spine Journal. 30(12):3702–3708. doi:10.1007/s00586-021-06973-3. 137 Hosseinian S, Arefi H, Navab N. 2020. Toward an end-to-end calibration for mobile C-arm in combination with a depth sensor for surgical augmented reality applications. Sensors (Switzerland). 20(1). doi:10.3390/s20010036. Intel. 2023. Intel® RealSenseTM D400 Series Dynamic Calibration Tool. [accessed 2023 Aug 29]. https://www.intel.com/content/www/us/en/download/645988/intel-realsense-d400-series-dynamic-calibration-tool.html. Job Bank Canada. 2024. Job prospects - X-ray (radiology) Technician in Canada. https://www.jobbank.gc.ca/marketreport/outlook-occupation/18292/ca. Kaplan DJ, Patel JN, Liporace FA, Yoon RS. 2016. Intraoperative radiation safety in orthopaedics: A review of the ALARA (As low as reasonably achievable) principle. Patient Saf Surg. 10(1). doi:10.1186/s13037-016-0115-8. Kausch L, Thomas S, Kunze H, Privalov M, Vetter S, Franke J, Mahnken AH, Maier-Hein L, Maier-Hein K. 2021. Correction to: Toward automatic C-arm positioning for standard projections in orthopedic surgery. Int J Comput Assist Radiol Surg. 16(9):1637–1637. doi:10.1007/s11548-021-02446-6. Koh A, Cribbie R. 2013. Robust tests of equivalence for k independent groups. British Journal of Mathematical and Statistical Psychology. 66(3):426–434. doi:10.1111/j.2044-8317.2012.02056.x. https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/j.2044-8317.2012.02056.x. Lakhwani OP, Dalal V, Jindal M, Nagala A. 2019. Radiation protection and standardization. J Clin Orthop Trauma. 10(4):738–743. doi:10.1016/j.jcot.2018.08.010. 138 Levene H. 1960. Robust Tests for Equality of Variances. In: Olkin I, editor. Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling. Stanford University Press. p. 278–292. Mahesh M. 2001. Fluoroscopy: Patient Radiation Exposure Issues. RadioGraphics. 21(4):1033–1045. doi:10.1148/radiographics.21.4.g01jl271033. http://pubs.rsna.org/doi/10.1148/radiographics.21.4.g01jl271033. Mastrangelo G, Fedeli U, Fadda E, Giovanazzi A, Scoizzato L, Saia B. 2005. Increased cancer risk among surgeons in an orthopaedic hospital. Occup Med (Chic Ill). 55(6):498–500. doi:10.1093/occmed/kqi048. Matthews F, Hoigne DJ, Weiser M, Wanner GA, Regazzoni P, Suhm N, Messmer P. 2007. Navigating the Fluoroscope’s C-Arm Back into Position: An Accurate and Practicable Solution to Cut Radiation and Optimize Intraoperative Workflow. J Orthop Trauma. 21(10):687–692. doi:10.1097/BOT.0b013e318158fd42. https://journals.lww.com/00005131-200711000-00004. Moody AE, Gurnea TP, Shul CP, Althausen PL. 2020. True Cost of Operating Room Time: Implications for an Orthopaedic Trauma Service. J Orthop Trauma. 34(5):271–275. doi:10.1097/BOT.0000000000001688. Navab N, Bani-Kashemi A, Mitschke M. 1999. Merging visible and invisible: Two Camera-Augmented Mobile C-arm (CAMC) applications. In: Proceedings - 2nd IEEE and ACM International Workshop on Augmented Reality, IWAR 1999. Institute of Electrical and Electronics Engineers Inc. p. 134–141. 139 Navab N, Blum T, Wang L, Okur A, Wendler T. 2012. First Deployments of Augmented Reality in Operating Rooms. Computer (Long Beach Calif). 45(7):48–55. doi:10.1109/MC.2012.75. http://ieeexplore.ieee.org/document/6165241/. Navab N, Heining SM, Traub J. 2010. Camera augmented mobile C-arm (CAMC): Calibration, accuracy study, and clinical applications. IEEE Trans Med Imaging. 29(7):1412–1423. doi:10.1109/TMI.2009.2021947. OpenCV. 2022. OpenCV-Python Tutorials. [accessed 2024 Jan 25]. https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html. OpenCV. 2024. Detection of ArUco Boards. [accessed 2024 Jan 29]. https://docs.opencv.org/3.4/db/da9/tutorial_aruco_board_detection.html. Porter T, Duff T. 1984. Compositing digital images. ACM SIGGRAPH Computer Graphics. 18(3):253–259. doi:10.1145/964965.808606. https://dl.acm.org/doi/10.1145/964965.808606. Rashid MS, Aziz S, Haydar S, Fleming SS, Datta A. 2018. Intra-operative fluoroscopic radiation exposure in orthopaedic trauma theatre. European Journal of Orthopaedic Surgery and Traumatology. 28(1):9–14. doi:10.1007/s00590-017-2020-y. Salvia JC La, de Moraes PR, Ammar TY, Schwartsmann CR. 2011. FLUOROSCOPY DURATION IN ORTHOPEDIC SURGERY. Revista Brasileira de Ortopedia (English Edition). 46(2):136–138. doi:10.1016/s2255-4971(15)30228-7. Shippert RD. 2005. A Study of Time-Dependent Operating Room Fees and How to Save $100 000 by Using Time-Saving Products. 140 Sielatycki JA, Sawyer JR, Mir HR. 2016. Supply and Demand Analysis of the Orthopaedic Trauma Surgeon Workforce in the United States. http://journals.lww.com/jorthotrauma. De Silva T, Punnoose J, Uneri A, Mahesh M, Goerres J, Jacobson M. 2018. Virtual fluoroscopy for intraoperative C-arm positioning and radiation dose reduction. Journal of Medical Imaging. 5(01):1. doi:10.1117/1.JMI.5.1.015005. Smith TS, Evans J, Moriel K, Tihista M, Bacak C, Dunn J, Rajani R, Childs B. 2022. Cost of Operating Room Time is $46.04 Dollars per Minute. Stryker. 2023. AxSOS 3 Ti Proximal Tibia. [accessed 2024 Jul 15]. https://www.stryker.com/us/en/trauma-and-extremities/products/axsos-3-proximal-tibia.html. Synbone. Tibia right. [accessed 2024 Jun 9]. https://www.synbone.com/product/ifb-040001. Tremains MR, Georgiadis GM, Dennis MJ. 2001. Radiation Exposure With Use Of the Inverted-C-Arm Technique in Upper-Extremity Surgery. http://journals.lww.com/jbjsjournal. Tukey JW. 1949. Comparing Individual Means in the Analysis of Variance. Biometrics. 5(2):99. doi:10.2307/3001913. https://www.jstor.org/stable/3001913?origin=crossref. US Bureau of Labor Statistics. 2022. Occupational Employment and Wages, May 2023 29-1242 Orthopedic Surgeons, Except Pediatric. https://www.bls.gov/oes/current/oes291242.htm. US Bureau of Labor Statistics. 2024. Occupational Employment and Wages, May 2023. [accessed 2024 Jul 16]. https://www.bls.gov/oes/current/oes292034.htm#nat. Weldon M, Arenas A, Abraham A, Haidar LA, Warth RJ, Mansour A. 2023. Attention to Detail: The Effect of Fluoroscopic Parallax on Limb Alignment Assessment During Corrective 141 Osteotomy. J Am Acad Orthop Surg Glob Res Rev. 7(2). doi:10.5435/JAAOSGlobal-D-22-00289. Zhang Z. 2000. A Flexible New Technique for Camera Calibration. 142 Appendices Appendix A Calibration of the Virtual Camera’s Location in the Earlier Implementation of DeCAF We described Hickey’s routine for calculating the transform between the depth camera and the calibration paper in Figure A.1. This approach does not use available functions in OpenCV to find the position of the markers relative to the board. Instead, it augments the position of the four markers by adding a Z-coordinate equal to the median of the depths acquired from the camera, 𝑑𝑖𝑠𝑡𝑧. It then calculates the centroid of each set of points and uses singular value decomposition to calculate the rotation and translation matrix describing the 𝑇𝑐𝑑 transform. 143 Figure A.1 Algorithm used by Hickey to find optimal rotation and translation between corresponding 3d points in order to transform the point-cloud to the virtual camera view 144 Appendix B Point-cloud Reprojection Pipeline in the Earlier Implementation of DeCAF The required pitch is given by γ_pitch = |γ|, where γ is the camera's pitch angle. The required translation along the y-axis is dy - sin(γ_pitch) × dist_z. Here, dy represents the y-translation of the camera image, γ_pitch is the absolute value of the camera's pitch angle, and dist_z is the median depth from the camera to the table. Figure B.1 Pipeline for point cloud rendering 145 Appendix C ArUco Board Position Estimation In order to estimate the position of the ArUco Board, we used a standard marker detection in OpenCV (OpenCV 2024). To detect markers in an image, the process starts by finding potential square shapes using adaptive thresholding and contour extraction. After filtering candidates based on size, the algorithm verifies markers by applying a perspective transformation, analyzing bit patterns through thresholding, and employing error correction techniques for accurate identification. After marker detection, the output includes information about the detected markers, such as their corner coordinates and corresponding IDs. Additionally, the markers can be visually highlighted on the input image using the provided data. Once the corner points on the board are detected, the next step involves using a specific function for estimating the board position by establishing correspondence between detected corners and predefined marker IDs. The process starts by validating that the total number of detected corners matches the total number of marker IDs. It then proceeds to obtain the corresponding object and image points required for subsequent processing. In the next step, we solve the perspective-n-point (PnP) problem to determine 𝑇𝑐𝑑, the camera pose in relation to the marker board given the object points and their corresponding image points, along with the camera matrix, distortion coefficients, and initial guesses for rotation and translation vectors. As shown in Figure C.1, points expressed in the pointer tool frame (c) are projected into the image plane (d) using the perspective projection model, Π, and the camera intrinsic parameters matrix K. Here, the intrinsic matrix 𝐾 captures the intrinsic parameters of the camera, including 146 focal lengths (𝑓𝑥, 𝑓𝑦) and principal point (𝑐𝑥, 𝑐𝑦)6. The matrix 𝑇𝑐𝑑 represents the camera pose, encompassing its rotation and translation in the world frame. The vector [𝑋𝑐; 𝑌𝑐; 𝑍𝑐; 1] denotes the 3D object point coordinates in homogeneous form. 𝑠 [𝑢𝑣1] = [𝑓𝑥 0 𝑐𝑥0 𝑓𝑦 𝑐𝑦0 0 1] [1 0 0 00 1 0 00 0 1 0] [𝑟11 𝑟12 𝑟13 𝑡𝑥𝑟21 𝑟22 𝑟23 𝑡𝑦𝑟31 𝑟32 𝑟33 𝑡𝑧0 0 0 1] [𝑋𝑐𝑌𝑐𝑍𝑐1] = 𝐾𝛱 𝑇𝑐𝑑 [𝑋𝑐𝑌𝑐𝑍𝑐1] 6 We will describe the process of calibrating intrinsic parameters of the camera in 2.2.4.1. 147 Figure C.1 Points expressed in the world frame Xc are projected into the image plane [u,v] using the perspective projection model, Π and the camera intrinsic parameters matrix K 148 Appendix D chhm_trackers Upon calling perform_calibration function, a tracker object is initialized. Note that after initializing the tracker object and initial camera and tracking setups, the tracking process is performed by track_UI which will be described in Figure D.1. Figure D.1 Overview of key steps in perform_calibration function which performs calibration of the location of virtual camera Here’s more details on methods described earlier: 149 • The select_frame_source method in this class handles the initial configuration of frame sources including activation of RGB and depth sensor and setting up exposure, format, frame rate, and resolution of each sensor. In our code, select_frame_source initiates a streaming thread that captures frames at a frame rate of 30 frames per second for both depth and color sensors. The camera configuration involves 1280×720 resolution in z16 format for the depth frames, alongside 1920×1080 resolution in z16 format for the colour frames. • Within select_tracking_method, the tracking method that the user would like to use is selected and important parameters of the pointer tool used could be initiated. As we use ArUco Board, parameters such as the marker layout, marker length, marker separation, and dictionary type are specified accordingly. Additionally, constants are set for corner refinement methods to enhance marker detection accuracy. Note that by default, OpenCV considers the coordinate axes to be defined at the bottom-left corner of ArUco Boards which is aligned with the C-arm laser aimer as shown in Figure 2.3 B. • After the initial setup, track_UI initiates the tracking process, wherein ArUco marker detection is performed for each frame. Detected boards are visually indicated by outlining their boundaries, providing users with a real-time view of the detection process. If the detection is satisfactory, users have the option to conclude the calibration process through the user interface. Upon completion, the software saves the camera position relative to the pointer tool as a homogenous matrix for future reference, ie Tcd. The process of detecting ArUco boards in each frame using Track Frames module is briefly discussed in Figure D.2. This process starts by capturing the most recent RGB frame from camera and converting it to grayscale. The dictionary specific to ArUco boards is selected and relevant detection parameters including 150 number and size of markers are configured. The algorithm then proceeds to identify the ArUco board in the grayscale image, and the detected board is visually indicated by outlining their boundaries. The algorithm estimates the pose of the detected board and calculates the board’s orientation and position relative to the camera. The 3D coordinate axes on the detected board are visualized as well. This estimation allows for accurate alignment of virtual camera with the C-arm view. The annotated image is displayed in real-time, and the process can be employed across multiple frames in video streams. Figure D.2 Track Frames using the track_UI function which performs AruCo Board tracking method 151 Appendix E display_pc Additionally, we wrote functions for capturing, processing, and overlaying X-ray onto the point cloud based on key functions offered by OpenCV. Figure E.1 Overview of key steps in perform_overlay function which point-cloud reprojection and displays the X-ray overlay • In order to streamline the overlay process, an object of the display_pc class is initiated with attributes and configurations. These include resolutions for depth view and transformation matrices for scaling and translation, machine calibration parameters (as explained in section 2.5.2). 152 • The setup_camera function configures and initializes the RealSense camera pipeline, enabling color and depth streams with specific settings such as resolution, format, and frame rate. It also checks for the availability of the RGB camera and sets up processing blocks for creating a 3D point cloud and colorizing depth frames. The function returns essential components like the pipeline, color and depth stream profiles, alignment object, point cloud processor, and colorizer, which are crucial for subsequent camera operations and 3D data visualization. • The setup_filters function configures a set of RealSense filters, including a decimation filter for reducing point cloud density, a spatial filter for domain-transform edge-preserving smoothing, a temporal filter for reducing temporal noise, and a hole-filling filter for depth interpolation. Additional options are adjusted for the spatial filter to emphasize its effects, and the temporal filter is set to operate on consecutive frames. The function also includes filters for transforming between depth and disparity representations. The configured filters are crucial for enhancing the quality and visual appeal of the captured 3D data by mitigating noise, reducing point density, and filling in depth discontinuities. • The show_overlay_frames Function, which is presented in Figure E.2 in more details, continuously displays the point cloud overlay from the virtual camera perspective feed within a loop. The displayed overlay includes the processed point cloud. Upon requesting the user for an X-ray image, the X-ray image would be captured and overlaid on the point cloud. There are additional user controls available, including the ability to adjust transparency, save the overlay, and manually manipulate X-ray parameters (rotation, translation – used in machine calibration). These controls are not depicted in Figure E.2 for simplicity. 153 Figure E.2 Overview of key steps in show_overlay_frames function which displays the X-ray overlay frame 154 Appendix F Open Reduction Internal Fixation (ORIF) ORIF, or Open Reduction Internal Fixation (Hanse and Pesántez 2010), is a surgical procedure commonly used to treat fractures in bones. The goal of ORIF is to stabilize and align broken bone fragments by using internal implants such as plates, screws, rods, or pins. This technique is particularly beneficial for fractures that are complex, displaced, or involve joint surfaces where precise realignment is crucial for proper healing and functional recovery. The ORIF procedure typically involves the following steps as shown in Figure F. 1: • Incision: A surgical incision is made near the fracture site to access the broken bone fragments. • Reduction: The surgeon carefully repositions the bone fragments back into their correct alignment. This step is called reduction and may involve manipulating the bones manually or with the assistance of specialized instruments. • Fixation: Once the bone fragments are properly aligned, internal fixation devices (such as metal plates, screws, or nails) are used to hold the bones in place. These devices provide stability, allowing the bones to heal properly over time. • Closure: After the bone fragments are fixed in their proper position, the incision is closed using sutures or staples. 155 Figure F. 1 Overview of ORIF Workflow including incision, reduction, K-wire insertion, preliminary fixation, plate insertion, and plate Fixation
Thesis/Dissertation
eng
Mechanical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Hodgson, Antony J.
Guy, Pierre
Reducing radiation dose in orthopaedic trauma surgery using Depth Camera Augmented Fluoroscopy (DeCAF)
Text