Panorama Ultrasound for Navigation and Guidance of Epidural Anesthesia by Hedyeh Rafii-Tari B.A.Sc., The University of British Columbia, 2009 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in The Faculty of Graduate Studies (Electrical & Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) September 2011 c© Hedyeh Rafii-Tari 2011 Abstract Epidural anesthesia is a common but challenging procedure in obstetrics and surgery, especially for the obese patient, and can result in complica- tions such as dural puncture and nerve injury. Ultrasound has the potential to significantly improve epidural needle guidance, by being able to depict the spinal anatomy and the epidural space. An ultrasound guidance sys- tem is therefore proposed, using a transducer-mounted camera to create 3D panorama images of the spine relative to markings on the skin. Guidance will include depiction of the spinal anatomy, identification of individual ver- tebrae, and selection of a suitable puncture site, trajectory and depth of needle insertion. The camera tracks the transducer movement using a specialized strip of markers attached to the skin surface. This enables 6-DOF absolute posi- tion estimation of the transducer with respect to the patient over the full range of the spine. The 3D panorama image can then be resliced in various parasagittal planes to show either the target epidural spaces or the laminae. The overall accuracy of the panorama reconstruction is validated by measur- ing inter-feature distances of a phantom of steel beads against measurements obtained from an optical tracking system (Optotrak), resulting in an aver- age error of 0.64 mm between camera and Optotrak. The algorithm is then tested in vivo by creating panorama images from human subjects (n=20), obtaining measurements for depth of insertion to the epidural space, inter- vertebral spacings, and registration of interspinous gaps to the skin, and validating these against independent measurements by an experienced sono- grapher. The results showed an average error of 1.69 mm (4.23%) for the depth measurements, average error of 4.44 mm (15.2%) for the interspinous distance measurements, and an average error of 6.65 mm for registering the ii Abstract interspinous gaps to the skin (corresponding to 18.5% of the interspinous distances). Tracking of ultrasound images wtih respect to the marker is im- plemented in real time and visualized using the 3D Slicer software package. iii Preface This thesis was prepared under the supervision and guidance of Dr. Robert Rohling and Dr. Purang Abolmaesumi. They introduced the research topic of creating panorama ultrasound images of the spine and registering the panoramas to the skin, using a camera mounted on the transducer, and provided ongoing feedback as the research progressed. They also provided revisions of the manuscripts resulting from this research, including a confer- ence paper and this thesis. Acquiring data from subjects was facilitated by Victoria A. Lessoway (RDMS, Department of Ultrasound, British Columbia Womens Hospital and Health Center). Ethics approval for this study was obtained from the UBC Research Ethics Board (Certificate Number: H07-01691). Some of the work presented in this thesis has been published in Lecture Notes in Computer Science [71]. This material appears in some parts of Chapters 4 and 5 of this manuscript. The author assembled the apparatus, designed the marker, wrote the Matlab code for implementing the tracking algorithm, constructed the validation phantom, and performed data acqui- sition and processing of ultrasound and camera images. The conference paper was written by the author and revised by Dr. Robert Rohling and Dr. Purang Abolmaesumi. iv Table of Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . v List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 Difficulties with Epidural Anesthesia . . . . . . . . . 2 1.1.2 Ultrasound Guidance . . . . . . . . . . . . . . . . . . 5 1.2 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Contributions and Applications . . . . . . . . . . . . . . . . 9 1.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1 Ultrasound Panorama Generation . . . . . . . . . . . . . . . 12 2.2 Position Estimation of Ultrasound Images . . . . . . . . . . . 13 2.2.1 Sensorless Approaches . . . . . . . . . . . . . . . . . . 13 v Table of Contents 2.2.2 Explicit Tracking Techniques . . . . . . . . . . . . . . 14 2.3 Transducer-Mounted Cameras . . . . . . . . . . . . . . . . . 18 2.4 Marker Applications in Computer Vision . . . . . . . . . . . 18 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3 System Components . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1 Ultrasound Acquisition . . . . . . . . . . . . . . . . . . . . . 21 3.2 Camera System . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2.1 Single Camera Versus Stereo . . . . . . . . . . . . . . 23 3.3 Software and Graphical User Interface . . . . . . . . . . . . . 23 3.3.1 Stradwin . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3.2 3D Slicer . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4 Calibration and Tracking . . . . . . . . . . . . . . . . . . . . . 27 4.1 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . 28 4.2 Marker Design . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2.1 Cross ratio Invariant . . . . . . . . . . . . . . . . . . 30 4.3 Tracking Procedure . . . . . . . . . . . . . . . . . . . . . . . 31 4.3.1 Background . . . . . . . . . . . . . . . . . . . . . . . 32 4.3.2 Feature Detection . . . . . . . . . . . . . . . . . . . . 33 4.3.3 Collinearity Test . . . . . . . . . . . . . . . . . . . . . 34 4.3.4 Invariant Interval Test . . . . . . . . . . . . . . . . . 35 4.3.5 Pose Estimation . . . . . . . . . . . . . . . . . . . . . 35 4.4 Ultrasound Image Calibration . . . . . . . . . . . . . . . . . 37 4.4.1 Background . . . . . . . . . . . . . . . . . . . . . . . 37 4.4.2 Single-wall Technique . . . . . . . . . . . . . . . . . . 38 4.4.3 Comparison against Optotrak Tracking System . . . . 41 4.5 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.5.1 Camera Calibration Accuracy . . . . . . . . . . . . . 42 4.5.2 Feature Detection and Pose Estimation Accuracy . . 42 4.5.3 Ultrasound Calibration Accuracy . . . . . . . . . . . 44 4.5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . 45 vi Table of Contents 5 System Validation . . . . . . . . . . . . . . . . . . . . . . . . . 47 5.1 Phantom Validation . . . . . . . . . . . . . . . . . . . . . . . 47 5.1.1 Experimental Design . . . . . . . . . . . . . . . . . . 47 5.1.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.2 Human Subject Validation . . . . . . . . . . . . . . . . . . . 49 5.2.1 Clinical Procedure . . . . . . . . . . . . . . . . . . . . 50 5.2.2 Data Processing and Analysis . . . . . . . . . . . . . 50 5.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 53 5.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 62 6 Real-time Implementation and GUI Design . . . . . . . . . 65 6.1 Software Architecture, Packages and Libraries . . . . . . . . 65 6.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 66 6.2.1 Experiments on Spine Phantom . . . . . . . . . . . . 68 6.2.2 Graphical User Interface . . . . . . . . . . . . . . . . 69 6.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 7.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . 71 7.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 7.3 Other Potential Applications . . . . . . . . . . . . . . . . . . 73 7.4 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 vii List of Tables 4.1 Intrinsic parameters of the camera and their uncertainties. . . 29 4.2 Ultrasound image calibration results . . . . . . . . . . . . . . 41 4.3 Ultrasound image propagated position and orientation error, for worst case uncertainty in each intrinsic parameter. . . . . 43 4.4 Reprojection errors from the calibration and tracking algorithm. 43 4.5 Ultrasound position estimation error, using two sets of differ- ent patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.1 Overall system accuracy, compared to Optotrak measurements. 49 5.2 Percentage errors for LF depth measurements at each level. . 56 5.3 Overall accuracy of LF depth measurement from the panoramas. 56 5.4 Percentage errors for interspinous distance measurements at each level, as compared to sonographer measurements. . . . . 59 5.5 Overall accuracy of interspinous distance measurements. . . . 59 5.6 Result of registering the panoramas to the skin. . . . . . . . . 61 viii List of Figures 1.1 Lumbar vertebral anatomy, depicting the epidural space. . . . 3 1.2 Lumbar vertebral anatomy in parasagittal longitudinal ap- proach (left), and idealized US image (right). . . . . . . . . . 6 1.3 Overview of the system depicting the subject, the US acqui- sition, the camera system, and the GUI. . . . . . . . . . . . . 8 3.1 The ARTCAM miniature camera (left), mounted on the trans- ducer (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2 Stradwin GUI, depicting the 2D US image, the 3D volume and the reslice. . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3 Slicer GUI, depicting an US volume with a tracked US image (top), the 2D US image and different reslices (bottom). . . . 25 4.1 Flowchart showing the different steps of camera calibration, marker design, tracking and pose estimation, and camera to US image calibration. . . . . . . . . . . . . . . . . . . . . . . 27 4.2 The calibration grid, imaged from different positions. . . . . . 29 4.3 The crossratio of four collinear points is projective invariant. 30 4.4 Marker design with projective invariant pattern properties. . 31 4.5 Sample camera image, showing the detected features and collinear patterns. . . . . . . . . . . . . . . . . . . . . . . . . 34 4.6 Sample camera pose, showing the camera with respect to the marker. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.7 Setup and wall phantom for ultrasound image calibration. . . 39 4.8 US image in Stradwin, showing the detected line correspond- ing to the bottom of the wall phantom. . . . . . . . . . . . . 40 ix List of Figures 4.9 LED rigid body attached to the camera for tracking with the Optotrak. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1 Test phantom with four distances (top) and US image of beads (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5.2 Overview of setup for human subject validation. . . . . . . . 51 5.3 Marker strip attached to the skin (left), 3D volume in Strad- win (center), parasagittal reslice showing the lamina at six upper lumbar and lower thoracic levels (right). . . . . . . . . 53 5.4 Original 2D ultrasound image in the transverse plane identi- fying the ligamentum flavum. . . . . . . . . . . . . . . . . . . 54 5.5 Reslice panoramas in the parasagittal plane, identifying the epidural space from the ligamentum flavum. . . . . . . . . . . 55 5.6 Intervertebral spacings measured by the sonographer. . . . . 57 5.7 Sample reslice panoramas for identification of vertebrae and interspinous distance measurements. . . . . . . . . . . . . . . 58 5.8 Registering the interspinous levels obtained from the panora- mas to the skin. . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.9 Panorama vs. sonographer interspinous distance measurements. 64 6.1 Software structure, for sending ultrasound images and track- ing information to 3D Slicer. . . . . . . . . . . . . . . . . . . 67 6.2 Lumbar spine phantom with the marker strip, for demon- strating the real-time implementation. . . . . . . . . . . . . . 68 6.3 Graphical user interface in Slicer, depicting the position of the tracked B-mode ultrasound with respect to the volume as well as the marker. . . . . . . . . . . . . . . . . . . . . . . . . 69 x Glossary Biopsy: medical removal of sample tissues or cells from a living body for examination Cephalad: towards the head or anterior section Fluoroscopy: X-ray imaging for obtaining images of internal structures In vivo: in the living organism Ionizing radiation: radiation with enough energy for removing electrons from the atom Lateral: towards the side, away from the center of the body Midline: plane along the center of the spine, dividing the body into left and right sections Obstetrics: branch of medicine dealing with the care of women during pregnancy, childbirth and the postnatal period Panorama ultrasound: an extended field-of-view ultrasound image, obtained from combining 2D or 3D ultrasound images to show extended sections of the anatomy Parasagittal (paramedian): a plane parallel to the midline and slightly lateral to the spine Parturient: in labour Percutaneous: a medical procedure where access to internal organs is achieved through needle insertion Phantom: an object or setup used for visualization purposes in medical imaging, by simulating conditions encountered in the procedure Sonographer: a medical specialist in the use of ultrasound for producing diagnostic images Transverse: horizontal plane, dividing the body into superior and inferior parts xi Acknowledgements First and foremost, I would like to deeply thank my supervisors Dr. Robert Rohling and Dr. Purang Abolmaesumi, for their guidance, support, and encouragement throughout my master studies. Their dedication and thor- oughness has been inspiring, and I have truly benefited from their advice and expertise. I would like to thank Vickie Lessoway for her assistance in collecting data, for always making time, and for teaching me the clinical aspects of research. Special thanks to everyone who volunteered as subjects for this study: Raoul Kingma, Caitlin Schneider, Abtin Rasoulian, Michael Yip, Siavash Khallaghi, John Bartlett, Troy Adebar, Jeff Abeysekera, Alexis Cheng, Arthur Schiro, Julio Lobo, Jack Zheng, Guy Nir, Mehdi Ramezani, Keigo Karakama, Matthew Pan, Eric Pospisil, Tom Huryn, Sam Frew, Tom Curran, and Mohammad Najafi. To my colleagues and friends in the Robotics and Control lab, Abtin Rasoulian, Troy Adebar, Caitlin Schneider, Michael Yip, Jeff Abeysekera, John Bartlett, Raoul Kingma, Ali Baghani, Ramin Sahebjavaher, and all my other UBC friends, for all the fun and memories in the past two years. A sincere appreciation and thank you to my family, for being a constant source of inspiration and support throughout the years, and for always be- lieving in me. This research was funded by the Natural Sciences and Engineering Re- search Council of Canada, and the Canadian Institutes of Health Research. Another thank you to Vicky Earle for drawing some of the figures presented in this thesis. xii Dedication To my parents, whose unconditional love and support has guided me through every single step of my life, and without whose sacrifices none of this would be possible. xiii Chapter 1 Introduction 1.1 Motivation Epidural anesthesia, or “epidural”, is a common procedure in obstetrics and surgery. In obstetrics, lumbar epidural anesthesia is used to provide pain relief during labor and cesarean delivery, and it is an effective alternative to general anesthesia [53], especially in parturients. In surgery, thoracic epidu- ral anesthesia (TEA) has seen a steady increase as the standard of care in areas of abdominal, chest, vascular and cardiothoracic surgery. TEA may be used alone or in conjunction with a light general anesthetic for a variety of procedures including thoracotomy, mastectomy, chest wall mass excisions, intra-abdominal procedures, and nephrectomy [1]. When comparing TEA to general anesthesia, different studies have cited significant advantages such as reduced pulmonary complications and hospital stay, and improved pa- tient comfort and outcome [57, 92]. Other advantages include decreased morbidity and mortality in elderly patients, as well as better analgesia and cardiopulmonary function [81]. To illustrate the current practice of these procedures, the Canadian Institute for Health Information reports a 49.4% average rate of epidural anesthesia for labour delivery (in 2008-2009), while other sources in the USA, such as Listening to Mothers, report numbers as high as 76%. According to the American Cancer Society, approximately 13% of all cancer cases (1.4M/year) are lung and bronchus, 3% are kidney, and 2% are pancreas, adding up to 252,000 cases of cancer surgeries which are eligible to employ TEA. 1 1.1. Motivation 1.1.1 Difficulties with Epidural Anesthesia Epidural anesthesia involves inserting a needle and catheter into the epidural space between the ligamentum flavum and dura mater covering the spinal cord (see Figure 1.1). The standard midline needle trajectory for lumbar epidurals encounters skin, fat, supraspinous ligament, interspinous ligament, and the ligamentum flavum before entering this space. For TEA, a slightly paramedian needle trajectory is used which still terminates after passing the ligamentum flavum. This procedure has three main steps: 1) Identification of vertebrae; 2) Selection of desired puncture site and trajectory; 3) Needle insertion into the epidural space. The standard method for identifying the vertebrae and selecting the puncture site is through palpation of the spine, by feeling the spinous processes close to the skin surface and counting up or down from a known vertebra. The anatomic landmarks used for this purpose include Tuffier’s line, which joins the two iliac crests and aligns approximately with the L4 vertebrae, or the C7 vertebrae, or finding the vertebra that is attached to the twelfth rib [74]. Studies show that palpation identifies the interspaces correctly in only 29-30% of the cases for lumbar vertebrae [11, 25], mean- ing the actual level was often one or more levels above or below the level identified by the anesthesiologist. After the desired intervertebral space is identified, needle insertion is performed by attaching a syringe (filled with saline or air) to the needle, and feeling the position of the needle tip through tactile feedback from the syringe’s plunger by injecting saline or air into the tissue. As the needle tip passes the ligamentum flavum and enters the epidural space the fluid is easily injected and a feeling of loss-of-resistance is achieved. This technique of loss of resistance to saline or air is the cur- rent gold standard for finding the correct depth of needle insertion into the epdiural space [96]. These blind methods are technically challenging, especially for the obese patient, and can result in complications such as dural puncture and spinal nerve injury [17, 38]. The interfaces among ligamentum flavum, epidural space and dura mater are encountered several millimeters apart, so nee- 2 1.1. Motivation Figure 1.1: Lumbar vertebral anatomy, depicting the epidural space. 3 1.1. Motivation dle insertion must be controlled to a high level of accuracy. The success of epidurals varies with the institution, the patient population, and anes- thesiologist experience, with one study showing that a 90% succes rate is only achieved after 60 epidural attempts [45]. Some of these procedures have been performed under fluoroscopic guidance, using contrast material to evaluate the epidural spaces and enable accurate needle placement, at the cost of exposure to ionizing radiation [41]. This would not be possible for parturient patients, which make up a large population of lumbar epidural cases in obstetrics. Failure of epidurals can be divided into different categories, including failure of needle insertion, difficulties in delivering effective analgesia, and complications such as dural puncture and nerve injury. Needle injection can fail if the needle encounters bone before reaching the epidural space, requiring withdrawal of the needle and re-insertion at a different angle and position, thereby adding time and discomfort to the patient. Needle in- sertion can also fail through injuring the epidural vein, which may lead to extradural or epidural hematomas. The incidence of epidural nerve injury varies between 5-12%, depending on the size and type of the needle and catheter used [34]. Complications also arise due to anesthesia effects as a result of incorrect identification of the vertebrae. Incorrect catheter place- ment away from the operative site results in an increased spread of the local anesthetic to achieve appropriate analgesia. Complications may in- clude incomplete spread of the anesthetic, and cardiovascular complications resulting from blockage of sympathetic nerve transmission caused by the local anesthetic [94]. Epidurals can also fail through needle overshoot and accidental puncture of the dura mater. This overshoot may cause leakage of the cerebrospinal fluid (CSF) into the epidural space, which in turn re- sults in post dural puncture headaches that can last up to several days [61]. Studies in TEA report an overall complication rate of 3.1%, including dural perforation, unsuccessful catheter placement, and postoperative pain [27]. Other studies on lumbar epidural anesthesia show 2.5% rate of accidental dural puncture, as a result of inaccurate needle placement, leading to post dural puncture headache in 86% of cases with dural perforation [84]. Suc- 4 1.1. Motivation cess rates are even lower for university training programs, due to the steep learning curve associated with these procedures [20]. These complications may include temporary or permanent neurologic deficit. Such complications, although rare, inhibit the more widespread use of epidural anesthesia. More accurate needle and catheter insertion is needed to decrease the risk and increase the use of these procedures. Achieving this goal is significant because it means that the benefits of epidural anesthesia, such as faster recovery as well as acute and chronic pain management, can be realized for a larger number of people. Long-term benefits include reduction in the time needed to successfully perform the procedure, improved training for anesthesiology residents, and increased efficiency of the surgical unit. The ideal guidance system would provide identification of vertebrae and interspinous gaps, selection of puncture site for desired vertebral level, se- lection of needle trajectory to intersect the epidural space, and depth esti- mation along this trajectory from the skin to the epidural space. So far no guidance system has been developed that addresses all of these requirements. 1.1.2 Ultrasound Guidance As a safe, low-cost, real-time imaging method, ultrasound (US) has the po- tential to significantly improve epidural needle guidance by being able to depict the spinal anatomy and the epidural space [89]. As a result, there has been a growing interest in US imaging for epidural anesthesia. Re- searchers have looked into pre-puncture US for locating the puncture site at the desired intervertebral level, and measuring the depth from the skin to the epidural space [30]. More recently real-time US has been used for guidance of needle insertion in lumbar epidurals [32, 42, 88]. Furthermore, image processing techniques have been explored on US images of the lumbar spine for improving image quality [87] and automatic depth measurement of the epidural space [90]. US imaging has also been tested for pre-puncture detection and guidance of the thoracic epidural space. [31, 72]. The two common acoustic windows that are used for ultrasound imaging of the spine anatomy are the transverse midline approach and the parasagit- 5 1.1. Motivation tal longitudinal approach, with information from each complementing the other [14]. The key elements visualized in each approach include ligaments and bone structures. Figure 1.2 shows an idealized US image from a curvi- linear transducer, depicting echos obtained in the parasagittal plane. The bright reflections show the lamina and the epidural space at the interface between the ligamentum flavum and the dura mater. Figure 1.2: Lumbar vertebral anatomy in parasagittal longitudinal approach (left), and idealized US image (right). Studies show that US can improve the success rate during the learning phase for residents performing epidural anesthesia, by comparing the initial epidural success rate from conventional teaching techniques (60%) to the success rate achieved through US guidance (86%) [29]. US can therefore be used as a valuable teaching tool for regional anesthesia techniques. Other studies prove the effectiveness of US imaging in needle guidance and epidural depth estimation for obese patients, who face higher complication rates when traditional blind techniques are used [4]. Current evidence also suggests that US guided procedures lead to several patient-orientated benefits such as reduced procedure time, improved needle placement and fewer number of necessary punctures, proving the success of ultrasound usage in guidance of epidural anesthesia [52]. 6 1.2. Research Objectives 1.2 Research Objectives This research proposes a novel panorama US navigation and guidance sys- tem for epidural anesthesia with the following specifications: a) Tracking individual ultrasound images with six degree of freedom with respect to the patient; b) Creating 3D ultrasound volumes from the spine; c) Providing absolute, not relative, tracking with respect to the patient, thereby allowing the transducer to be removed and replaced. The volumes are created using a 2D transducer for its lower cost and simplicity. Panoramas are created by reslicing the 3D volume. Two such panoramas are desired for this application: a reslice in the parasagittal plane showing the target epidural spaces for trajectory selection and epidu- ral depth estimation, and a reslice in a more lateral sagittal plane, depicting the vertebral laminae with clear interspinous gaps, for identification of the vertebrae. The latter plane is more suitable for distinguishing the individ- ual vertebrae since the lamina show a large wave-like pattern with clear interspinous gaps, thereby allowing the anesthesiologist to select the desired level, puncture site and needle trajectory while reducing the risk of needle overshoot or bone contact. The two different reslice planes mentioned above are needed for these tasks, therefore a 3D panorama volume is required. A single US image can depict only one or two vertebrae because of the small field of view. To provide a single view of all vertebrae of interest in order to allow counting from a desired location on the spine (sacrum, C7, or T12 levels), an extended panorama is needed. The guidance system is designed in a way that it can be integrated with an existing clinically approved US machine, to be used in two stages for future practice: (1) pre-puncture scanning of the patient during surgery preparation to create the panoramas for identifying the levels and select- ing desired puncture site, trajectory and depth of insertion (2) real-time guidance of the needle insertion in the operating room. The system uses a single miniature camera attached directly on the US transducer and a specialized adhesive marker strip with unique high-contrast features attached on the skin, which allows absolute position estimation of 7 1.2. Research Objectives the camera and the transducer over the whole range of spine with respect to the patient. Absolute positioning is needed because real time guidance during needle insertion will be performed in the future by repositioning the transducer at the desired skin location, determined by analysis of the panorama images. An overview of the entire system is provided in Figure 1.3. Figure 1.3: Overview of the system depicting the subject, the US acquisition, the camera system, and the GUI. The overall hypothesis is that an optical US guidance system with a sin- gle inexpensive camera and a simple setup can be constructed and developed with sufficient accuracy for creating panoramas of the spine, identification of vertebrae and interspinous distances, and detection of epidural spaces, for selection of puncture site and trajectory with respect to the patient. To reach this goal the system should be able to: • depict the spinal anatomy for a wide range of patients • distinguish each vertebrae from its neighbors 8 1.3. Contributions and Applications • provide guidance of needle trajectory and depth of insertion to the epidural space • and register the ultrasound panoramas to the patient skin. Needle insertion will be performed using a combination of ultrasound guidance and traditional loss-of-resistance [88], as loss-of-resistance is still the standard of care for epidural anesthesia and provides a clear end-point for needle insertion into the epidural space. Therefore, the accuracy of the preceding panoramas must be sufficient to identify the vertebrae and select an approximate puncture site and trajectory to the epidural space. The accuracy is determined by a combination of factors, such as the measurement accuracy of the interspinous gap location and width, and the identification of the ligamentum flavum which is 5-6 mm thick [17]. The ligamentum flavum produces the echo defining the posterior aspect of the epidural space, and is the smallest anatomical structure that needs to be identified so that loss-of- resistance can be performed over the last centimeter of the needle insertion. 1.3 Contributions and Applications This research proposes a low-cost 3D US guidance system for epidural anes- thesia, using a miniature camera attached on the transducer and a special- ized marker strip attached on the skin for position tracking of US images with respect to the patient. The goal is to provide relatively simple ad- ditions to the steps of patient preparation and equipment on a standard ultrasound machine. The system can be easily integrated into clinical appli- cations for epidural anesthesia since equipments are simple and no external hardware or apparatus are required. Using a camera to relate US images to the patient can have a significant impact on the confidence of the anesthe- siologist performing the epidural, and increase the efficiency of the surgical unit. Furthermore, improved guidance of needle insertion will result in fewer incidences of dural puncture and nerve injury, therefore providing improved outcomes for epidural anesthesia and better patient care. 9 1.4. Thesis Outline Although the first intended application is the guidance of the commonly performed epidural needle insertions, other clinical applications are also fea- sible. A similar application is spinal taps where the needle is inserted along the same trajectory but at a deeper depth. Other possible applications in- clude spinal biopsies, nerve blocks, regional anesthesia techniques, spinal surgery and any applications where US to patient registration can be valu- able. Therefore, there are large numbers of patients that would benefit from development of safer and more efficient techniques for needle insertion and anesthesia, as a result of this research. 1.4 Thesis Outline This manuscript covers the background literature related to panorama gen- eration and position estimation of ultrasound images, the proposed system design, algorithm and graphical user interface, and the validation results on phantom as well as human subjects. The outline of the thesis is as follows: Chapter 2: Background Provides an overview of the previous research and relevant literature in ultrasound panorama generation, tracking of ul- trasound images, other applications of transducer mounted cameras, and use of markers in different areas of computer vision. Chapter 3: System Components Describes the overall system and apparatus, with specifications for ultrasound imaging, camera acquisition, and the different software that were used. Chapter 4: Calibration and Tracking Discusses the marker design, calibration, and tracking methods in detail and evaluates the sources of error in the algorithm. Chapter 5: System Validation Describes the overall system valida- tion, first through a phantom by comparing to the Optotrak tracking sys- tem, then through an extended study on human subject by comparing the 10 1.4. Thesis Outline panorama measurements against manual measurements by an experienced sonographer. Chapter 6: Real-time Implementation and GUI Design Discusses the software architecture and graphical user interface used for the real-time implementation of the algorithm, with experiments performed on a spine phantom. Chapter 7: Conclusion Summarizes the goals, results and contribu- tion of the research, discusses potential applications of the results, and also suggests possible future directions for improving the work presented in the thesis. 11 Chapter 2 Background 2.1 Ultrasound Panorama Generation In order to be able to obtain different reslices in different desired planes for depicting the epidural spaces or the laminae, a 3D panorama volume is desired for the proposed system. Different approaches exist in literature for generating panoramas from 2D or 3D ultrasound data. Compound 3D data sets have been obtained from 2D ultrasound trans- ducers, using position sensors and tracking systems to receive the exact spatial displacement of image data [75]. Position sensors have also been used with 3D ultrasound transducers for creating extended field-of-view ul- trasound panoramas from stitching the 3D volumes together [65], where the positions obtained from the sensors were combined with image-based reg- istration techniques to correct for misalignment errors in overlapping data regions. Panorama US imaging has also been explored, without the use of po- sition sensors, for generating compound 2D and 3D US panorama images from partially overlapping data sets [80]. The displacements between suc- cessive images were found through optimizing a similarity measure between frames, based on multiresolution approximation of the image intensities. Compound panoramas were then created by bilinear interpolation and aver- aging the displacement data. A similar approach has been used to create US panoramas of the lumbar spine, for automatic identification and counting of the vertebrae [43], where panoramas were created by direct stitching of ultrasound images obtained from a 2D transducer in the parasagittal plane. The transducer motion was measured from the movement of features within the images, by comparing pixel intensity values from pairs of consecutive 12 2.2. Position Estimation of Ultrasound Images images. Automatic identification was then achieved by applying different filters to highlight the edges of the bone. Using this method, the RMS er- ror for registering the US to the skin was rather high (11.8 mm). Greater accuracies are desired with the proposed research. Compared to these intensity-based algorithms, other image-matching ap- proaches use feature based methods to compute the displacement of charac- teristic landmarks for correlation of medical images [8]. In most cases, these landmarks have to be defined manually, making the approach time-intensive and inefficient. These methods are also unreliable in cases of complex US images where distinct features prove hard to find. Commercial systems have also developed panorama techniques using 2D transducers, by sweeping the transducer in the lateral direction over a re- gion of interest, detecting regions of overlap in sequential moving real-time images, and developing the panorama live while scanning [48]. Examples of these systems include SieScapeTM(Siemens Ultrasound, WA, USA) and LOGIQView (GE Medical Systems, Waukesha, USA). 2.2 Position Estimation of Ultrasound Images This section describes the relevant literature and common approaches used for tracking and position estimation of US images, including speckle track- ing, and explicit tracking using local and external position sensors. 2.2.1 Sensorless Approaches It is possible to track the position of US images for acquiring freehand 3D US without a positions sensor, solely by aligning the image features to detect the relative motion of two consecutive frames, and repeating this estimation over a series of successive images [26, 68]. In these approaches, the in-plane motion (translation in the axial and lateral directions, roll around the plane normal) and out-of-plane transformations between image frames (translation in the elevational direction, tilt and yaw) are found independently. In-plane motion between frames is found through standard 2D image registration 13 2.2. Position Estimation of Ultrasound Images techniques [40], while the out-of-plane motion is found through the decorre- lation of nearby frames as a result of backscattered US signals resulting in speckle, which contain position information due to their decorrelation over the distance between frames [91]. A major source of error in speckle tracking are drift errors caused by tissue inconsistencies and speckle patterns that are not fully developed [36]. Errors can also arise from probe rotations affecting the elevational correla- tion values [39]. As a result of these error, US position tracking using speckle information suffers from an overall lack of accuracy. To overcome these lim- itations, measurements from additional sensors have been fused with the data from image alignment techniques. In [49], electromagnetic sensor in- formation were combined with speckle tracking, using an Unscented Kalman filter framework, to reduce the drift errors and eliminate the high-frequency jitter noise from electromagnetic sensors. Overall, such “sensorless” approaches are not feasible for the spine, due to complex bone structures resulting in strong spatially-varying shadows and artifacts in the ultrasound image, and explicit tracking of the probe is preferred. 2.2.2 Explicit Tracking Techniques Position estimation and 3D volume generation from 2D US images is usually achieved by attaching position sensors to the transducer for receiving the exact position and orientation of each scan plane. Advantages of these free- hand tracked systems compared to 3D US transducers used with standard, commercial US machines include a wider range of motion, as well as the ability to acquire US images in a global coordinate system, to which other tracked surgical instruments or preoperative images (e.g. CT or MRI) can also be linked [60]. Conventional systems to explicitly track the ultrasound transducer posi- tion are based on mechanical, acoustic, electromagnetic, and optical tracking technologies [59]. Mechanical trackers that have been used for tracking of instruments during intra-operative image-guided procedures in the past usu- 14 2.2. Position Estimation of Ultrasound Images ally consisted of linkages or robotic-like articulated arms, whose tip positions were found through encoders placed in the joints [13, 76]. Patents also ex- ist, which implement US positioning on the patient back, through movable mounts attached to mechanical frames [83]. Acoustic systems are based on receivers that track the position of a desired instrument using air-based sonic wave emitters placed on the instrument, and have also been used for surgical procedures [73]. Electromagnetic tracking systems are based on placing a receiver on the transducer to measure induced currents when moved within a magnetic field generated by either an AC or DC transmitter. These systems are being used more widely for position estimation of US scans in image guided interventions and robotic surgery [7, 50, 62]. Tradeoffs exist among these technologies in terms of range, accuracy, external influences (e.g. line-of-sight and proximity of metal objects), ease of use, and cost. Mechanical trackers are accurate, but are bulky and can only track one object at a time. Sound-based systems are easily disturbed by ambient sound, and variations in temperature and humidity. Due to these limitations, mechanical and acoustic trackers have been replaced with optical or electromagnetic trackers in recent years. Electromagnetic systems have small sensors and do not require line of sight, which makes them attractive for embedding into medical instruments. Their main disadvantage is the distortion in measurement caused by the proximity of metal objects and magnetic fields (e.g hospital beds, power sources), which makes them non- ideal for operating room environments [28]. Optical Tracking of Ultrasound Optical tracking systems are generally extremely accurate (accuracies less than 1 mm), however the main disadvantage of optical tracking is the re- quirement for a direct line of sight from the camera to any marker that is used: if the camera system cannot “see” the marker, tracking is impossi- ble [28]. Furthermore, the trackers themselves tend to be relatively large. Common optical trackers use cameras to track the location of specialized markers. In active systems, these markers are IR-LEDs attached to the pa- 15 2.2. Position Estimation of Ultrasound Images tient or transducer. These require a power supply delivered by batteries or wires and make handling more cumbersome. In passive optical systems re- flective markers such as spheres or disks are used that reflect IR light emitted from the cameras. An alternative would be optical tracking of high contrast markers such as a checker board pattern using standard video lenses and cameras in ambient room light and performing image analysis to extract pose information. Popular commercial optical tracking systems include the Polaris (North- ern Digital Inc., ON, Canada) which uses both passive reflective markers or active infrared markers, Optotrak (Northern Digital Inc., ON, Canada) which uses active markers, and the MicronTracker (Claron Technologies, ON, Canada) which uses printed checker-board like patterns. External tracking systems are usually rather expensive, with costs ranging from ap- proximately $10,000 USD with the MicronTracker, to $30,000 and $80,000 for the Polaris and Optotrak. Optical tracking of the ultrasound transducer location using external cameras or trackers has been performed by many groups with different types of active or passive markers attached to the transducer and/or patient. IR- LEDs, rigidly attached to the transducer, are common for tracking the po- sition of US images with respect to the base of the tracker [3, 47, 64]. Some of these methods have been combined with fiducials, attached rigidly to the probe face, for direct positioning of 3D US volumes in space [66]. Passive markers, such as reflective spheres, have also been used by some groups [51]. In [24] the transducer location was measured with respect to the body by tracking both the probe and the patient using textured features and patterns attached on both the patient and the transducer. The use of checkerboard patterns attached on the US transducer, for tracking with regular digital cameras, has also been explored [2]. Different patents also employ cam- eras for tracking the position of the ultrasound transducer with respect to an external coordinate system and reconstructing 3D ultrasound volumes [78, 79]. Other approaches have used a combination of local sensors (e.g. optical mice, accelerometer) added to the transducer for trajectory reconstruction 16 2.2. Position Estimation of Ultrasound Images and position estimation, based on measuring relative translations and ab- solute orientations of the transducer with respect to an external frame of reference [85]. Ultrasound Image Calibration The tracking systems described above estimate the position of the sensors or markers attached to the transducer. In order to track the position of the ultrasound image itself, the mathematical transformation between the 3D coordinates in the frame of reference of the position sensor attached to the transducer and the 2D coordinates of pixels in the US image is needed. This is found through ultrasound calibration, which typically involves scanning an object with known geometrical properties (a phantom) [59]. Further details of this procedure and relevant background information are provided in Section 4.4 of this manuscript. Registration of US to Skin In order to measure the probe location with respect to the patient’s body, the common approach in literature is to track both the probe and the patient. This has been achieved by attaching active or passive markers [3] or magnetic sensors [18] to the desired body parts. Novel research in this area includes the use of image overlays on the skin, and has been explored for CT-guided needle insertions by registering the imaging modality to the patient through specialized skin fiducials [23]. Other approaches have used multiple external cameras for tracking of medical instruments and monitoring the patient movement, by using different markers or natural landmarks, to obtain the position of instruments with respect to the patient in clinical settings [19]. Movement of the patient is inevitable in epidural anesthesia applications, especially in the case of parturient patients where the epidural procedure can take up to five minutes to perform. Movement can be due to respiration or forces induced by the transducer, and includes external patient movement or internal organ movement. To compensate for external patient movement in most tracking systems, the position of the transducer, obtained with respect 17 2.3. Transducer-Mounted Cameras to the fixed external base of the tracker, is transformed into the coordinate system of the patient which is also tracked [24]. However, direct tracking of ultrasound to skin is preferred. Even though internal organ movement can still introduce errors, the proposed research introduces a more stable solution for direct tracking of ultrasound images with respect to the patient skin. 2.3 Transducer-Mounted Cameras Mounting miniature cameras on the transducer has been previously explored for needle guidance in ultrasound guided percutaneous needle insertions in- cluding drug deliveries and biopsies [15, 44]. The cameras are used for measuring the needle pose and estimating the trajectory, directly with re- spect to the US transducer, without the need for an external base coordinate system. A similar approach uses the camera on the transducer for obtaining video images of the patient skin at the intersection of the US imaging plane, together with overlaying graphics to indicate possible needle entry points [77]. No research has been performed so far on using cameras, mounted directly on the transducer, for absolute position tracking of US images with respect to the skin. By putting the camera directly on the transducer and eliminating the external base coordinate system, the tracker range can be greatly reduced, thereby improving the cost and accuracy trade off significantly. The re- duced range also improves errors caused by the lever-arm effect between the position sensors and the US image. 2.4 Marker Applications in Computer Vision The use of high-contrast markers is popular for optical tracking and pose es- timation in different areas of computer vision and augmented reality. Wang et al.[95] use a visual marker pattern with unique features attached to the operating table, together with a camera augmented C-arm X-ray device, for pose estimation and positioning of the X-ray source with respect to the pa- 18 2.5. Summary tient’s table. The marker pattern consists of high contrast black and white markers that are uniquely distinguished in the video images. Color-coded markers have also been used for pose estimation through video capture and image processing techniques [99]. Position tracking using square shaped markers is also very common in augmented reality applications [98]. Most of these markers contain checkerboard-like patterns, providing high contrast, easily identifiable feature points . One of the most popular commercial optical tracking systems that make use of high-contrast printed patterns is the MicronTracker (Claron Technolo- gies, Ontario, Canada). This system uses markers comprised of checkerboard targets with unique patterns and geometries. Target locations are computed at the intersection of four high-contrast regions (called Xpoints) by using the four black/white boundary lines independently to pinpoint the location of the target. Another real-time optical pose measurement system which also makes use of small x-shape points is the infiniTrack system (Atracsys, Switzerland). Checkerboard markers provide easily identifiable and clear edges and corners for reliable feature detection. Furthermore these markers can be easily applied to different uses, by being printed on reusable or ster- ilizable sheets, or attached directly on the tracked instrument or the patient skin. The low cost of the markers, and the wide variety of marker designs, are other advantages. 2.5 Summary This chapter explored the existing research and relevant literature for cre- ating US panoramas, and tracking the position of US scans. Sensorless speckle tracking approaches were reviewed and deemed unfeasible for the spine, where complex structures can result in strong artifacts in US images. Common explicit tracking techniques, including electromagnetic and optical sensors, were explored and compared against each other. While optical trac- ing systems offer great accuracies, they suffer from line-of-sight issues and relatively high cost of apparatus. Most tracking systems estimate the posi- tion of the transducer with respect to the external base of the tracker, and 19 2.5. Summary transform that information to the coordinate system of the patient which is also tracked. Patient movement is one of the complications of these ap- proaches, especially for parturient patients. Therefore, direct tracking of US to skin is preferred. Mounting miniature cameras directly on the US transducer has been tried before, for estimating needle pose and trajectory with respect to the patient in image-guided percutaneous needle insertions, but not for tracking of the transducer itself. The use of high contrast mark- ers and checkerboard patterns for pose estimation and tracking in different areas of computer vision and augmented reality, as well as existing com- mercial optical tracking systems that are based on printed markers was also explored. Based on existing literature, the proposed US panorama guidance system using a single low-cost camera mounted on the transducer for absolute posi- tion tracking of US images with respect to the skin is completely novel. By using a marker strip with unique high-contrast features attached directly to the patient skin the external base coordinate system is eliminated, thereby reducing the tracker range, reducing cost compared to external trackers, as well as increasing simplicity in terms of additional steps for patient prepa- ration and adding equipment to a standard ultrasound machine. Another major advantage of the proposed system is the elimination of any line-of- sight issues, by having the camera on the transducer directly looking at the marker from a close distance. This is a major problem with external tracking systems, especially in surgical environments which are often cluttered and therefore make it more challenging to maintain line of sight. Furthermore, by tracking the position of US images with respect to the skin, this system has the potential to accommodate patient motion during scanning. 20 Chapter 3 System Components This chapter provides details of the apparatus and software that were used. The implementation of the tracking system involves acquiring standard 2D ultrasound images, acquiring images of the marker strip attached to the skin using a camera mounted directly on the transducer, determining the pose of the transducer by processing the camera images, combining the ultra- sound images and the position data to generate 3D ultrasound panoramas and obtaining slices of the panorama using medical imaging visualization software. 3.1 Ultrasound Acquisition Ultrasound images were obtained from either the Sonix MDP or the Sonix RP500 ultrasound machines (Ultrasonix Medical Corporation, Richmond, BC) using a 5 MHz 2D curvilinear array transducer (model C5-1/60). This transducer had been used successfully for epidural guidance in the past [88]. Furthermore, using a low-frequency (2-5 MHz) curved probe is rec- ommended for most epidurals, as it provides deeper ultrasound penetration which makes it more suitable for these procedures [14]. The ultrasound machines are based on PC architecture and contain the SonixRP interface for streaming ultrasound data. Ultrasonix also provides research packages and SDKs, such as Ulterius, which allow full control of all the imaging pa- rameters and real-time access to the entire imaging pipeline over a network connection. Therefore, applications that connect to the Sonix using Ulterius can run directly on the ultrasound system, or run on a PC connected to the same network. All ultrasound images were obtained at a depth of 7 cm, to make certain that all desired anatomical structures would be captured in 21 3.2. Camera System the images. Other image settings, such as gain, focus, dynamic range, and frequency were adjusted depending on the application to ensure optimum quality for the images. 3.2 Camera System A single ARTCAM-130MI-IR-OP camera was used (ARTRAY Co. Tokyo Japan). This is a 1/2 CMOS near-infrared miniature camera with a maxi- mum resolution of 1280× 1024 pixels, a frame rate of 15 FPS, and a field of view close to 40 degrees. The camera’s dimensions are 43.5 (W )×43.4 (H)× 30.0 (D) mm and it weighs approximately 24 g. The camera has an embed- ded 8 mm lens and comes in a small board format. The camera was placed in an enclosure and mounted rigidly on the trans- ducer, by screwing it to an existing transducer clamp. The camera board and the setup used for mounting the camera on the transducer can be seen in Figure 3.1. The positioning of the camera on the probe was decided ex- perimentally based on the placement of the marker on the patient skin, the field of view of the camera, and the lens position (camera focusing limits) for obtaining high quality images at close distances to the skin. The ap- proximate distance from the camera lens to the skin in the current setup is 8 cm. Figure 3.1: The ARTCAM miniature camera (left), mounted on the trans- ducer (right). 22 3.3. Software and Graphical User Interface The camera has an associated SDK and sample C++ code provided by Artray for accessing the camera images. These were modified to provide continuous video capture, while saving the images and their time stamp information. The camera was connected to the Sonix machine via USB2.0 and controlled by the software directly implemented on the Sonix. Even though the current apparatus is too large and heavy for routine clinical use, it is sufficient for proof of concept and can be easily miniaturized for future use, together with a clear sterile drape for clinical applications. Another approach would be to implement the camera directly inside the transducer casing, as part of the transducer design. 3.2.1 Single Camera Versus Stereo Two cameras and stereo systems are more used for tracking and pose esti- mation in cases where no markers are used and no accurate knowledge of the model is available. When one camera is used for tracking, finding a unique solution for the pose of the camera with respect to the object requires three or more identifiable reference points on the object to be known, therefore markings or specialized markers are required [70]. Using an artificial marker strip based on high-contrast patterns with well defined features as reference points, and ensuring that enough features can be seen in every image, enables accurate six degree-of-freedom pose estimation for the single camera with respect to the marker. By establishing point correspondences between the 2D image points and the 3D feature points on the marker, the pose of the camera relative to the marker strip, and the skin, can be estimated while improving cost and simplicity compared to stereo camera solutions. 3.3 Software and Graphical User Interface Reconstruction of freehand panorama volumes, obtaining different reslices from the panoramas, and visualization of live US position tracking with respect to the skin was performed using two existing software packages for 23 3.3. Software and Graphical User Interface medical image visualization and analysis. These are described as follows: 3.3.1 Stradwin In order to reconstruct a 3D panorama volume from the obtained 2D ultra- sound images, Stradwin was used [67]. Stradwin is an experimental cross- platform tool for freehand 3D ultrasound calibration, acquisition, measure- ment, reconstruction and visualization. The software was installed on the ultrasound machine, and connected to Sonix using the Ulterius SDK for direct acquisition of ultrasound images. Stradwin’s different visualization and measurement capabilities, includ- ing landmark selection and distance measurement, reconstruction of 3D ul- trasound volumes, and obtaining reslices from different planes, were used for validating the overall system on phantom data as well as human sub- jects. This was done by combining the ultrasound images obtained directly through Stradwin with the desired position information obtained offline from the cameras, and is described in detail in Chapter 5. Furthermore, Strad- win’s built-in freehand 3D ultrasound calibration tool was used for calibrat- ing the camera frame of reference to the ultrasound frame of reference (See section 4.4). A sample screen shot of the Stradwin interface is shown in Fig- ure 3.2, depicting the original 2D US image, the 3D volume and the reslice. 3.3.2 3D Slicer 3D Slicer [63] was eventually used for implementing the tracking algorithm in real time. Slicer is an open-source software package that offers a modular framework for advanced visualization and medical image processing. It has become very popular in the field of image-guided therapy, for construction and visualization of pre- and intra operative image data to be used with different instrument tracking applications. Slicer provides an advanced, in- teractive graphical user interface that allows volume reslicing, as well as other common tasks including manual segmentation, image registration and creation of surface models. 24 3.3. Software and Graphical User Interface Figure 3.2: Stradwin GUI, depicting the 2D US image, the 3D volume and the reslice. Figure 3.3: Slicer GUI, depicting an US volume with a tracked US image (top), the 2D US image and different reslices (bottom). 25 3.4. Summary Using different computer vision libraries and open-source medical imag- ing software packages, live US images and their corresponding position infor- mation were sent to Slicer for visualization and further processing. Details of this process, the software architecture and the graphical user interface will be provided in Chapter 6. A screenshot of the graphical user interface, depicting the volume, the tracked US image and different reslices is shown in Figure 3.3. 3.4 Summary This chapter provided an overview of the different system components and details of the apparatus and software that are used for creating 3D US vol- umes and panoramas of the spine. These include the ultrasound acquisition system, the transducer-mounted camera system, and the software used for reconstruction of the volumes and visualization of the reslice panoramas. The next chapter will provide details of the tracking algorithm for process- ing the camera images and obtaining position information of the US images with respect to the skin. 26 Chapter 4 Calibration and Tracking The tracking algorithm is based on direct position estimation of the camera, and hence ultrasound images, with respect to the marker attached to the skin. The novel marker design uses unique patterns that enable absolute tracking through known 3D reference points for all patterns. The camera images were read into Matlab for offline image processing to estimate the pose of the camera attached to the transducer, and extracting the position of the US images. An overview of the required steps for system calibration and tracking is given in Figure 4.1. Figure 4.1: Flowchart showing the different steps of camera calibration, marker design, tracking and pose estimation, and camera to US image cali- bration. The required steps are camera calibration (to find the intrinsic param- eters of the camera), marker design, camera image processing (to detect 27 4.1. Camera Calibration marker features and estimate the camera pose), and camera to ultrasound calibration (to find the position of ultrasound images with respect to the skin). These are described in detail in this section, by providing details of the marker design, different steps of the tracking algorithm, the technique used for calibrating the camera to the transducer, and finally a discussion of the sources of error in the algorithm. 4.1 Camera Calibration Camera calibration was performed to determine the intrinsic parameters of the camera. These are the focal length of the camera (fc), the principle center (cc), and the distortion vector including radial and tangential distor- tion (Kc). Calibration was performed using the standard method of acquir- ing multiple images from different views of a checkerboard and performing least-squares fit of the camera parameters to match the known checkerboard features. This was done using an 8×8 checkerboard of 6.3 mm squares. The pattern was imaged from 30 different angles and positions, shown in Fig- ure 4.2. From each image the corners of the checkerboard were extracted and sent as input into a freely available camera calibration toolbox for Matlab [10]. Using the toolbox, calibration was performed in two steps: first a closed form solution of the camera parameters was computed excluding any lens distortion. Then non-linear optimization was performed through iterative gradient descent to minimize the reprojection error (in the least squares sense) over all calibration parameters. To further reduce this error, the corners were re-extracted using the results of the first calibration, and the optimization step was repeated. The standard deviation of the reprojection error was calculated to be [1.231 1.037] pixels in the x and y directions, respectively. The calibration results and their uncertainties are given in Table 4.1. 28 4.2. Marker Design Figure 4.2: The calibration grid, imaged from different positions. Intrinsic parameters in pixels Focal length [1747.62 1745.55 ] ± [5.22 5.18] Principal point [659.00 568.39 ] ± [10.19 9.53] Distortion [-0.46 0.29 -0.01 0.01 0] ± [0.02 0.14 0 0 0] Table 4.1: Intrinsic parameters of the camera and their uncertainties. 29 4.2. Marker Design 4.2 Marker Design The proposed marker strip is based on checkerboard-like patterns with unique projective invariant properties, which have been previously explored for pattern recognition and tracking of objects [86, 93]. The patterns are designed based on the invariant properties of the cross ratio. 4.2.1 Cross ratio Invariant The property of the cross ratio states that if four collinear points (A, B, C, D) are given, the cross ratio can be defined based on the distance between these points according to the following relationship [54]: Cross ratio (A,B,C,D) = |AC| / |BC| |AD| / |BD| . (4.1) The cross ratio is invariant under perspective projections, as shown in Figure 4.3, meaning that although the relative distances between the pro- jected points on the line change, the cross ratio remains constant [93]. This property can be used as a unique identifier in marker design, since a camera looking at the same pattern from different angles and scales can compute the cross ratio and identify that pattern using only the image coordinates of the four collinear points. Figure 4.3: The crossratio of four collinear points is projective invariant. 30 4.3. Tracking Procedure The marker design is based on high-contrast black and white checkerboard- like patterns with varying widths and orderings, with each pattern providing four collinear corners on each side, each with a unique cross-ratio that will be used as the identifier for that pattern. The width of the patterns is de- signed based on the camera’s field of view (1-2.5 cm), and the length of the whole strip covers the average length of the spine (approximately 45 cm). The height of the patterns and the spacing in between them was designed to ensure that at least 3-4 collinear patterns can be seen in each camera image. This marker meets the practical requirements for the intended clinical use, and can easily be attached to the skin with the goal of enabling accurate and robust tracking over the whole range of spine. The marker strip is shown in Figure 4.4. Figure 4.4: Marker design with projective invariant pattern properties. 4.3 Tracking Procedure This section explains in detail the approach that was taken for processing the camera images to obtain the position information of the camera with respect to the marker. The algorithm was implemented in Matlab using Matlab’s image processing toolbox. First an overview of different feature detection and pose estimation algorithms is provided, then each step of the tracking algorithm is explained. The algorithm’s purpose is to detect features in each image, group the features into collinear patterns (consisting of four feature points), calculate the cross ratio, use this identifier to detect the pattern and its 3D position in the marker frame, and finally extract the camera pose using the image coordinates of the pattern, their corresponding 3D coordinates and the intrinsic parameters of the camera. 31 4.3. Tracking Procedure 4.3.1 Background Finding desired features through image processing is one of the most com- monly performed tasks in computer vision, used in motion tracking, 3D pose estimation, and object and scene recognition. One of the popular algo- rithms for detecting and describing features in an image is the scale-invariant feature transform (SIFT) which is based on extracting highly distinctive fea- tures that are invariant to image scale and rotation [55], by generating large numbers of features that densely cover the image over the full range of scales and locations. SIFT is more suitable for object recognition and matching across images, in scenes that contain large number of objects and therefore suffer from clutter and occlusion. For corner identification in images such as checker-board patterns, ap- proaches that are more feasible include edge detectors such as Canny [12], or corner detectors such as the Harris operator [35], since these images con- tain easily identifiable, high-contrast, edges and corners. The Canny edge detector uses a multi-stage algorithm for detecting a wide range of edges, by finding the local maximum of the intensity gradients of the images. The Harris detector is a combined corner and edge detector, based on the local auto-correlation function for measuring the shift in image patches, and is a viable solution for finding corners in images of the marker strip. Pose estimation consists of determining the position and orientation of a calibrated camera with respect to an object, given a set of correspondences between the known 3D reference points on the object and their 2D image coordinates. The relationship between 2D image points and corresponding 3D object points is given here:  rc 1  =  fx 0 ccr0 fy ccc 0 0 1   1 0 0 00 1 0 0 0 0 1 0 [ R T 0 1 ] X Y Z 1  (4.2) Where r and c are the 2D image frame coordinates, X, Y , and Z are 32 4.3. Tracking Procedure the 3D object frame coordinates, and fx, fy, ccr and ccc are the intrinsic parameters of the camera (focal length and principal center). Pose estima- tion solves for the rotation, R, and translation, T , between the object frame and the camera frame of reference. When only one camera is used for track- ing, a unique solution for the pose of the camera can be achieved if at least three identifiable reference points on the object are known. This can also be achieved linearly using at least four coplanar points [70]. The classical approach for pose determination for larger number of reference points is to solve it by a nonlinear least-squares optimization, using an initial guess to converge to the correct solution. A common approach is based on Newton’s method of linearization and iteration to perform a least-squares minimiza- tion and then ensure convergence of the solution through the Levenberg- Marquardt method [56]. Another method uses an iterative algorithm, with four or more non-coplanar points, by starting from a scaled orthographic ap- proximation. Compared to the Newton’s method approach, no initial pose estimate is required [21]. 4.3.2 Feature Detection The desired features to be detected are the corners of the patterns on the marker. First, the distortion of the camera images were removed, using the intrinsic parameters obtained through camera calibration. For detecting the desired corners, a Harris corner detector was then implemented [46]. This de- tector contains parameters such as the standard deviation of the smoothing Gaussian, and the radius of the region considered in non-maximal suppres- sion, which were adjusted to achieve optimum results for feature detection. To improve the speed and robustness of corner detection, a curvature based corner detector was also implemented, which finds the edges using a Canny edge detector, and estimates the corners as local maxima of absolute cur- vature [37]. To use this detector the images were smoothed first using a Gaussian filter with a size of 17 and a standard deviation of 3. To avoid multiple-feature detection for the same corner, limits were added to elimi- nate features points that were too close to each other. The corners obtained 33 4.3. Tracking Procedure from these methods were then sent as initial guess into a corner-finder ob- tained from the calibration toolbox, which is based on the Harris detector and finds the actual corners within a desired window to a precision below 0.1 pixels. 4.3.3 Collinearity Test Once the corners in the image were detected, an algorithm was implemented to group together all the possible four point combinations of the detected features, and compute the collinearity to find the ones corresponding to the same pattern. The method for computing the collinearity is based on com- paring the slopes of the three lines between the first point and the other three points in each pattern, and setting a limit close to zero for accepting the four-point-pattern as collinear . Since the desired corners corresponding to the patterns are all in the horizontal direction, another limit was set on the slopes to eliminate collinear patterns found in the vertical direction. A sample image, showing the results of the feature detection and the collinear- ity test is shown in Figure 4.5. Figure 4.5: Sample camera image, showing the detected features and collinear patterns. 34 4.3. Tracking Procedure 4.3.4 Invariant Interval Test Once the four corners corresponding to the same pattern are identified, the cross ratio is computed to find the position of the pattern on the marker strip. First, each pattern was trained by calculating the cross-ratio from im- ages taken at different orientations and positions and generating the unique identifier for the pattern, defined by the range containing the minimum and maximum cross ratio values. The patterns were adjusted to ensure that the identifier is unique and cross ratio values for different patterns do not overlap. These were then stored in the pattern database together with the corresponding 3D coordinates of the corners in the marker strip frame of reference. Once all patterns were trained, the algorithm was used in track- ing mode by calculating the cross ratio of each pattern and comparing that to the database to obtain the 3D position of the pattern on the marker. Since the four corners on the top and bottom of each pattern result in the same cross-ratio values, a check was added to distinguish between these by selecting two windows above and below each pattern, and calculating the gradient by subtracting the sum of pixels from both windows. A positive gradient corresponds to the top row, and a negative gradient corresponds to the bottom row of corners. 4.3.5 Pose Estimation Using the 3D positions of the patterns obtained from the database, together with the corresponding 2D image coordinates and the intrinsic parameters of the camera, the pose of the camera can be estimated. If Xm is the 3D reference point coordinates in the marker frame and Xc is the same coordinates in the camera frame, pose estimation will solve for rotation (R) and translation (T ) in the following equation: Xc = RXm + T . The pose of the camera relative to the marker pattern was calculated using the algorithm employed by the calibration toolbox [10]. The method is based on calculating normalized point coordinates and then the 3D pose, using iterative optimization to converge to the accurate pose solution, and works for both planar and non-planar structures. 35 4.3. Tracking Procedure While pose estimation can be performed linearly using at least four copla- nar points [70], larger number of feature points were used to improve robust- ness to measurement error. For this research, eight feature points (corre- sponding to two collinear patterns) and their respective 3D coordinates are used in the pose estimation algorithm. Given that three to four collinear patterns can be seen in each camera image, when processing the data ob- tained from human subjects (Section 5.2) two poses were calculated from each image, one from each pair of patterns that provide eight feature points, and a weighted average of the two poses was used to estimate the actual camera pose for that image frame. This was done to provide an estimate of the curvature of the back, given that the marker is assumed to be placed on a flat surface, and thereby improve the quality of the results. An inverse distance weighted approach (Shepard’s method [82]) was used, based on the distance between the middle of each pair of patterns and the center of the image. An example result from the pose estimation algorithm is depicted in Figure 4.6, showing the position and orientation of the camera with respect to the eight reference points on the marker. Figure 4.6: Sample camera pose, showing the camera with respect to the marker. 36 4.4. Ultrasound Image Calibration 4.4 Ultrasound Image Calibration The developed tracking algorithm determines the position of the camera with respect to the fixed coordinate system on the marker. In order to obtain the position information of the US images and reconstruct 3D ultra- sound volumes, however, the mathematical transformations from the camera frame of reference to the coordinates of the ultrasound image needs to be determined. This is achieved through ultrasound probe calibration, as de- scribed in this section. First, a background of different calibration methods that are widely used in literature is provided, then the method used for this research and the obtained results are described. The calibration results were compared against calibrating the US image to the position sensor from an external optical tracking system. 4.4.1 Background The role of calibration is to find the transformation that converts the 2D ultrasound image coordinates to the 3D coordinates of the position sensor at- tached to the transducer. Ultrasound probe calibration is typically achieved by scanning an artificial object with known geometrical properties, known as a phantom, from different angles and positions and identifying its fea- tures from the ultrasound images. Calibration yields eight transformation parameters: three translation, three rotations and two image scaling factors [59]. The more widely used calibration phantoms include the following: • Single-point or cross-wire phantoms [22, 69]: The single point phantom is generally based on imaging a small spherical object such as a bead or a pin head, while the cross-wire phantom is made of two intersecting wires. The point target, or the intersection of the wires, are aligned in the ultrasound image and scanned from several viewing angles. • Multiple cross-wire phantoms [58]: These consist of three coplanar wires forming a triangle. The ultrasound images of these phantoms therefore are composed of points and/or lines. 37 4.4. Ultrasound Image Calibration • Three-wire phantoms [69] are made of three orthogonal wires, scanned sequentially along their length. The origin of the phantom coordinate system is placed at the intersection of the three wires. • N-wire or Z-fiducial phantoms [62, 97]: The wires of these phantoms form Z shapes, intersected by the ultrasound imaging plane. Up to 30 fiducials have been used for these phantoms, to ensure accuracy. • Wall phantoms [59, 69]: These methods produce a line in the US image, which is more attractive since the information is more redun- dant, compared to points (if a line is partially missing, it can still be identified). The phantoms in this category include the single-wall, membrane, and Cambridge phantoms, which are based on imaging a water tank, a thin membrane, and a thin brass bar respectively. Other phantoms have been used that are extensions of the described phantoms above, such as the IXI-wire phantom [64], and the Double-N phantom [16], which are analogous to the N-wire phantom but use addi- tional wires to aid registration. More recently, phantom-less calibration techniques have also been used, by using actual patient images instead of a calibration phantom [6]. 4.4.2 Single-wall Technique Probe calibration was performed using Stradwin’s single-wall ultrasound calibration tool. This is based on imaging the bottom of a water tank by holding the transducer at different angles and positions [59, 69]. Water is a good medium for obtaining clear ultrasound images. This method has some disadvantages when imaging at large angles from the normal, since most beams will be reflected away from the transducer because of specular reflection, resulting in less distinct image features. To compensate for the specular reflection problem, and to obtain higher quality calibration images, a metal block with a roughened surface was placed at the bottom of the water bath. 38 4.4. Ultrasound Image Calibration Figure 4.7: Setup and wall phantom for ultrasound image calibration. The tracking system was positioned so that the scanning tip of the ul- trasound probe was completely immersed in water and a clear straight line corresponding to the top of the metal block could be detected in the ultra- sound image. The marker was placed on a flat surface next to the water bath in such a way to be in clear view of the camera attached to the probe. The ultrasound depth setting was set to 7 cm. Approximately 100 ultra- sound images were obtained by moving the probe in various orientations in six degrees of freedom and ensuring that the metal surface would reflect as a clear straight line in each ultrasound image. The calibration setup is shown in Figure 4.7. Camera images were processed to obtain the position of the camera with respect to the frame of reference on the marker and were then transferred to Stradwin. Stradwin’s calibration tool automatically detects the straight line corresponding to the bottom of the wall phantom in every image, and using the position information calculates the calibration parameters using least squares minimization techniques. A sample ultrasound image in Stradwin, 39 4.4. Ultrasound Image Calibration showing the detected line, is given in Figure 4.8. Figure 4.8: US image in Stradwin, showing the detected line corresponding to the bottom of the wall phantom. The transformations are as follows: xm ym zm 1  = TMCTCU  sx.u sy.v 0 1  (4.3) In this equation, u and v are the image coordinates in the ultrasound image with the origin located at the top left corner of the image, TMC is the transformation between the camera frame (C) and the marker (world) frame of reference (M), obtained from processing the camera images of the marker, and xm, ym and zm are the coordinates in the marker frame of reference. sx and sy are the image scales. To provide an estimate of the ultrasound image scales for Stradwin, a small block with known geometry was scanned into Stradwin and distance measurements were obtained from the image. The calibration solves for the three translations and three rotations in TCU , the transformation between the ultrasound image and the camera 40 4.4. Ultrasound Image Calibration frame of reference. The results are shown in Table 4.2. The RMS resid- ual error of this calibration was calculated to be 1.8 mm. X (cm) Y (cm) Z (cm) α(deg) β(deg) γ(deg) -0.66 11.20 3.72 5.20 13.94 108.08 Table 4.2: Ultrasound image calibration results 4.4.3 Comparison against Optotrak Tracking System Calibration and tracking of the ultrasound image using the transducer- mounted camera was compared against an Optotrak 3020 (Northern Digital Inc., Waterloo, ON) as the gold standard. Four IR-LED’s were attached to the transducer for creating a rigid body to be tracked (Figure 4.9). Figure 4.9: LED rigid body attached to the camera for tracking with the Optotrak. The transformation from this Optotrak rigid body coordinate system to the ultrasound image was calibrated using the same single-wall technique 41 4.5. Error Analysis described above, and compared to ultrasound calibration with the camera, by acquiring simultaneous camera and Optotrak measurements and sending these together with the ultrasound images to Stradwin’s calibration system. The RMS residual error of this calibration was 0.6 mm. 4.5 Error Analysis This section describes the sources of error in the overall tracking algorithm. These include errors in the intrinsic calibration of the camera, errors in feature detection and pose estimation from the marker, and the error of calibrating the ultrasound image to the camera frame of reference. 4.5.1 Camera Calibration Accuracy One of the sources of error in the algorithm is the error in the intrinsic calibration of the camera. The effect of this error was estimated by calcu- lating how the uncertainty, or error, in each intrinsic parameter can affect the position of the center of the ultrasound image with respect to the marker (skin) coordinate system. These uncertainties were obtained through cam- era calibration, as explained in Section 4.1. By adding the uncertainties to each intrinsic parameter separately, the propagated position and orientation errors of the ultrasound image were obtained. The maximum propagated errors from a change in each intrinsic parameter are given in Table 4.3. When adding the uncertainties to all parameters at once, the maximum propagated position and orientation errors were 1.16 mm and 0.41 degrees, respectively. 4.5.2 Feature Detection and Pose Estimation Accuracy Another source of error comes from extraction of the corner features in the marker from the images and estimating the pose of the camera. These are evaluated by obtaining camera images from various parts on the marker strip (60 images in total), with each image containing at least three collinear 42 4.5. Error Analysis Intrinsic parameters Avg. value (pixel) Uncertainties (pixel) Propagated position error(mm) Propagated orientation error(deg) fc(x) 1747.62 5.22 0.172 0.303 fc(y) 1745.55 5.18 0.315 0.256 cc(x) 659.01 10.19 0.556 0.479 cc(y) 568.39 9.53 0.674 0.386 k1 -0.4588 0.0212 1.014 0.378 k2 0.2888 0.1389 1.585 0.536 k3 -0.0056 0.0011 0.149 0.145 k4 0.0051 0.0009 0.213 0.115 Table 4.3: Ultrasound image propagated position and orientation error, for worst case uncertainty in each intrinsic parameter. patterns. Two of these patterns, containing eight feature points, were used to estimate the pose of the camera. Then using this pose, the image plane projections for the third pattern were found and compared to the originally detected corners in the image. This was repeated for all 60 images. The results are given in Table 4.4. The maximum reprojection error is 0.46 mm with an RMS error of 0.15 mm. reprojection error(mm) n observations 60 max error 0.457 min error 0.016 mean error 0.123 SD 0.081 RMS error 0.146 Table 4.4: Reprojection errors from the calibration and tracking algorithm. To estimate the effect of these errors on the position of the ultrasound image, pose estimation of the center of the ultrasound image was performed 43 4.5. Error Analysis for each camera image using two marker patterns (eight feature points), and compared against the pose obtained from a different set of eight feature points from the same image. The differences between these two positions were calculated over all 60 images, and are reported as error values in Ta- ble 4.5. Position Orientation error(mm) error(deg) max error 3.87 2.76 mean error 1.01 0.47 SD 0.82 0.51 Table 4.5: Ultrasound position estimation error, using two sets of different patterns The maximum position and orientation errors are 3.87 mm and 2.76 degrees, respectively. 4.5.3 Ultrasound Calibration Accuracy Another error comes from the calibration between the camera and ultra- sound image. Comparing the ultrasound to camera calibration against the Optotrak rigid body, the RMS error of the camera calibration is 1.8 mm, compared to an RMS error of 0.6 mm for the Optotrak showing the greater accuracy of the Optotrak, as expected. Another source of error is the ul- trasound image scale (mm/pixel), which was obtained for ultrasound image calibration independently through scanning an object with known geometry (section 4.4.2). This can be different from the conversion factor based on a speed of sound of 1540 m/s in tissue, depending on the water tempera- ture and Stradwin’s estimation of speed of sound in the water bath used for calibration. 44 4.5. Error Analysis 4.5.4 Summary This section covered the major sources of error for tracking and position estimation of US images with respect to the marker. The errors come from different steps of the algorithm: • Error of intrinsic calibration: When calibrating the camera using the standard Matlab calibration toolbox, the standard deviation of the reprojection error was calculated to be [1.231 1.037] pixels, in the x and y directions, respectively. To analyze the effect of the error in the intrinsic parameters of the camera (including focal length, prin- cipal center, and distortion), the maximum propagated position and orientation errors in pose estimation were calculated by adding the uncertainties to each intrinsic parameter, and observing the change in position of the center of the ultrasound image with respect to marker. The effects of each change are given in Table 4.3. By adding all un- certainties, the maximum propagated position and orientation errors were 1.16 mm and 0.41 degrees respectively. • Error of pose estimation: Other errors arise from extraction of the corner features in the images and estimating the pose of the camera. These were evaluated through reprojecting the pose obtained from two patterns onto a third pattern, for multiple images. Results are given in Table 4.4, depicting a maximum reprojection error of 0.46 mm. To estimate the effect of these errors on the position of the ultra- sound images, pose estimation of the center of the ultrasound image was performed from different sets of feature points obtained from the same image, and compared. The maximum change in position and orientation for the ultrasound image was 3.87 mm and 2.76 degrees respectively. Results are shown in Table 4.5. • Error of ultrasound image calibration: The RMS error of the camera- to-ultrasound calibration was 1.8 mm compared to a 0.6 mm error for the same calibration achieved through the Optotrak tracking system. Given the much higher cost, and larger size of the Optotrak compared 45 4.5. Error Analysis to the camera, this error is reasonable. The effect of these accumulated errors will be explored in the next chap- ter, when the overall system is first validated on a phantom against the Optotrak tracking system, and then validated on human subjects. 46 Chapter 5 System Validation To evaluate the overall accuracy of ultrasound position estimation and panorama reconstruction, the system was validated first through distance measure- ments obtained from a phantom, then through in-vivo validation on human subjects. These are described in detail in this chapter. 5.1 Phantom Validation The accuracy of the camera-ultrasound system is evaluated by measuring inter-feature distances of a phantom of steel beads. The experimental design of the phantom, comparison against the Optotrak system, and the results of the distance measurements are presented in this section. 5.1.1 Experimental Design The phantom consists of 2 × 2 matrix of small steel beads embedded in a metal plate. Using similar phantoms of pins or beads for measuring distances of features and determining accuracy of ultrasound calibration is common in literature [5, 50, 62]. The beads have a diameter of 4.5 mm and are placed at row separations of 20 mm and column separations of 40 mm. The phantom with the four inter-bead distances is shown in Figure 5.1. The phantom was placed in water heated to approximately 50 ◦C to match the speed of sound in tissue. The marker was placed on a flat surface next to the water bath, in clear view of the camera. Each bead was imaged eight times at different ultrasound probe orientations and depths. The transducer was adjusted until the clearest possible surface reflection was obtained in each image, in order to ensure that the bead is centered 47 5.1. Phantom Validation Figure 5.1: Test phantom with four distances (top) and US image of beads (bottom). in the elevation direction of the ultrasound image. For every ultrasound image, position measurements were obtained from both the camera and the Optotrak 3020 (using the Optotrak rigid body constructed on the camera, described in section 4.4.3). The distances between the beads were measured using the Stradwin software by manually selecting the top of the bead in each image, based on the assumption that the sphere surface detected in the ultrasound image corresponds to the top surface of the bead, as demon- strated in [33]. 5.1.2 Results Using all combinations of the eight observations of each pin gives 64 possible combinations for each pair for each of the four distances, therefore resulting 48 5.2. Human Subject Validation in a total of 256 measurements. The measurements based on the camera positions were compared to measurements obtained from the Optotrak as the gold standard, and the difference between these values is used as an estimate of the overall accuracy of the system for ultrasound position estimation. The mean, standard deviation, and maximum values are used to characterize these differences, as shown in Table 5.1. Accuracy of camera compared to Optotrak (mm) n observations 256 max error 3.51 mean error 0.64 SD 0.89 RMS error 0.90 Table 5.1: Overall system accuracy, compared to Optotrak measurements. The results show a mean error of 0.64 mm between between camera and Optotrak measurements, with a standard deviation of 0.89 mm. This error includes a combination of errors from different steps of the calibration and tracking procedure, that were explained in Section 4.5. 5.2 Human Subject Validation To test the ability of the system in creating panoramas of the spine for iden- tification of vertebrae and selection of puncture site, trajectory, and depth of insertion, the algorithm was tested on volunteer human subjects. The results were validated against measurements obtained from an experienced sonographer (Victoria A. Lessoway, BC Women’s Hospital and Health Cen- tre). This section describes the experiment setup and procedure for obtain- ing the data, the processing and analysis steps performed in Stradwin, and different measurements that were obtained from the panoramas including the depth measurements to the ligamentum flavum, interspinous distance measurements, and registering the panoramas to the skin. The results are 49 5.2. Human Subject Validation presented and a discussion of the errors is provided. 5.2.1 Clinical Procedure To depict the spinal anatomy and identify the vertebrae in vivo, the algo- rithm was implemented on 20 human subjects (ethical approval number: H07-01691) by moving the transversely-oriented transducer, with the cam- era attached, in the parasagittal plane. The transducer was moved cephalad on the right side of the spine by starting from the sacrum and going up as far as the mid-thoracic levels. The subject was seated in an upright posi- tion, similar to clinical practice, with a straight back. The marker strip was attached lateral to the spine and was always in clear view of the camera. An overview of the system, depicting the ultrasound machine, the patient with the attached marker, and the transducer with the mounted camera can be seen in Figure 5.2 5.2.2 Data Processing and Analysis The camera was connected to the Sonix machine via USB2.0 and controlled by the software directly implemented on the Sonix, for saving a continuous stream of time-stamped images. Ultrasound images were acquired simul- taneously through Stradwin, which was also installed on the ultrasound machine. Camera images were processed afterwards to obtain the position information for the ultrasound images, using the tracking algorithm and cal- ibration information described in Chapter 4. These positions were paired to the corresponding ultrasound images using CPU time stamps of the ultra- sound scans, which are saved by Stradwin. The 6-DOF position information obtained from the camera images were smoothed using a moving average filter on a unit quaternion representation, with a span of 11 for each subset of data. Since the ultrasound acquisi- tion rate was faster than the frame rate of the camera, the positions from the cameras were interpolated to match the ultrasound images. The three translations were interpolated using cubic spline interpolation, while the 50 5.2. Human Subject Validation Figure 5.2: Overview of setup for human subject validation. 51 5.2. Human Subject Validation three rotations were interpolated through a linear interpolation on their quaternions (smoothing and interpolation of position information were not performed for the previous phantom tests in Section 5.1, where the marker was placed on a flat surface and position measurements were obtained for each ultrasound image one-by-one). By adding the positions for each ultra- sound frame to the Stradwin data file, together with the calibration matrix between the ultrasound and camera frame of reference, 3D volumes were re- constructed in Stradwin, and panorama reslices were visualized. These were obtained through Stradwin’s Reslice tool, which uses linear interpolation for creating reslices at desired angles and position. Stradwin also allows placing landmarks on the image, for selecting desired feature points and obtaining distance measurements between them. To confirm the ability of the system for guidance of needle trajectory and depth of insertion, depth measurements from the skin to the ligamentum flavum were obtained from the panoramas, ranging from the L5-Sacrum interface to the mid-thoracic levels (T7-8 interface). These are compared to depths obtained from the original 2D ultrasound image at the corresponding levels. The ability to distinguish each vertebra from its neighbors is validated by measuring intervertebral spacings in the panoramas and comparing these to measurements made manually by an experienced sonographer. Sonographer measurements were obtained using a transducer in the parasagittal plane, depicting the lamina of 2-3 consecutive levels in a single ultrasound frame. Distances were measured between the superior borders of consecutive lam- ina, and measurements were obtained for both lumbar and thoracic (up to the T8 level). Panoramas are registered to the skin by relating the interspinous gaps detected in the panorama to their relative positions on the marker strip. These were compared to marks placed by the sonographer directly on the marker at each gap, from sacrum to the T7-8 interface. Identification of the interspinous levels with respect to patient skin is defined as successful if the difference between the marker location obtained from the panorama and the location obtained by the sonographer is less than half the intervertebral 52 5.2. Human Subject Validation spacings. 5.2.3 Results For the human data, panorama images of the spine are created by gener- ating reslices in the parasagittal plane through the created 3D ultrasound volumes in Stradwin. A sample image of the tests in vivo is shown in Fig- ure 5.3, showing the marker position on the body, the created 3D volume in Stradwin, and the reslice depicting the lamina for six consecutive vertebrae as clear, distinguishable curves. Although spine images are obtained from the sacrum up to the mid thoracic levels, the Stradwin software can only depict panoramas up to 1024 pixels at a time, therefore only seven vertebrae are shown in the figure. Figure 5.3: Marker strip attached to the skin (left), 3D volume in Stradwin (center), parasagittal reslice showing the lamina at six upper lumbar and lower thoracic levels (right). 53 5.2. Human Subject Validation Ligamentum Flavum Depth Measurements Reslices can be created in the parasagittal plane for depicting the ligamen- tum flavum (LF), which is the last interface encountered by the needle be- fore entering the epidural space. Figure 5.4 shows the original 2D ultrasound image, depicting how the ligamentum flavum and the epidural space is iden- tified from an ultrasound image in the transverse plane. Figure 5.4: Original 2D ultrasound image in the transverse plane identifying the ligamentum flavum. Figure 5.5 shows some sample reslice panoramas depicting the ligamen- tum flavum at consecutive levels for different lumbar and thoracic vertebrae, confirming the ability of the system for guidance of needle trajectory and depth of insertion. Depth measurements of the ligamentum flavum were obtained from the panoramas at different levels, ranging from sacrum to mid-thoracic, resulting 54 5.2. Human Subject Validation Figure 5.5: Reslice panoramas in the parasagittal plane, identifying the epidural space from the ligamentum flavum. 55 5.2. Human Subject Validation in a total of 177 measurements for all 20 patients. These were compared against the ligamentum flavum depth in the original 2D ultrasound image. The average percentage errors at each level are given in Table 5.2. Average percentage error T7 - 8 1.29 T8 - 9 1.68 T9 - 10 3.61 T10 - 11 2.96 T11 - 12 4.63 T12 - L1 4.92 L1 - 2 4.66 L2 - 3 5.62 L3 - 4 3.68 L4 - 5 3.85 L5 - S 4.85 Table 5.2: Percentage errors for LF depth measurements at each level. The mean, standard deviation, and RMS values of the error over all the measurements are shown in Table 5.3. The mean error is 1.69 mm, corresponding to 4.23%, and the standard deviation is 1.45 mm, showing promise for confirming the ability of the panoramas in estimating the depth of the epidural space. Accuracy of panorama LF depth measurement compared to 2D US image (mm) n observations 177 mean error 1.69 SD 1.45 RMS error 2.23 Table 5.3: Overall accuracy of LF depth measurement from the panoramas. 56 5.2. Human Subject Validation Intervertebral Spacing Measurements Reslices were also obtained from a more lateral sagittal plane to depict the laminae for identification of vertebrae. Intervertebral spacings obtained from the panoramas were compared to measurements made manually by the sonographer. Figure 5.6 shows some sample images of the detected levels and measurements obtained by the sonographer in the parasagittal plane. Figure 5.6: Intervertebral spacings measured by the sonographer. 57 5.2. Human Subject Validation Figure 5.7 shows some sample reslice panoramas identifying different lumbar and thoracic vertebrae. (The top reslice depicts the intervertebral spacings for the five lumbar vertebra). Figure 5.7: Sample reslice panoramas for identification of vertebrae and interspinous distance measurements. Interspinous distance measurements were obtained from the panoramas in Stradwin (ranging from sacrum to mid-thoracic levels) resulting in a total 58 5.2. Human Subject Validation of 156 measurements over all 20 patients, and compared against the mea- surements obtained by the sonographer. The percentage difference between these measurements at each level are given in Table 5.4. Average percentage error T8 - T9 16.45 T9 - T10 11.76 T10 - T11 15.01 T11 - T12 11.20 T12 - L1 16.87 L1 - L2 15.27 L2 - L3 15.14 L3 - L4 14.13 L4 - L5 17.37 L5 - S 16.99 Table 5.4: Percentage errors for interspinous distance measurements at each level, as compared to sonographer measurements. The mean, standard deviation, and RMS values of the error between the manual and panorama measurements, over all the levels, are shown in Table 5.5. The mean error is 4.44 mm, corresponding to 15.2% of the actual interspinous distances, suggesting that each vertebra can be distinguished from its neighbour. This is apparent in the panorama images as well, since the lamina of the vertebrae in a parasagittal reslice appear as clearly distin- guishable curves. Interspinous distance measurements compared to sonographer measurement (mm) n observations 156 mean error 4.44 SD 2.81 RMS error 5.26 Table 5.5: Overall accuracy of interspinous distance measurements. 59 5.2. Human Subject Validation Sonographer repeatability In order to get an estimate of the variance in the sonographer’s measurements, interspinous distance measurements at a certain level were obtained repeatedly from the same subject (L3 - L4, n=10). The standard deviation of this measurement was calculated to be 0.83 mm (2.5 %), proving the high precision in sonographer measurements, as a viable gold standard for our validation. Given this measure of the sono- grapher’s repeatability, the differences between the panorama measurements and the sonographer’s measurements are interpreted to include errors from both the panorama and sonographer’s techniques. Registration to the Subject Skin The panoramas were registered to the skin, by comparing the position of the interspinous spaces with respect to the marker strip, obtained from the panoramas, to marks placed by the sonographer directly on the marker strip at each interspinous level, as shown in Figure 5.8. The marks ranged from the sacrum to the mid-thoracic levels, for a total of 166 measurements over all patients. The results are given in Table 5.6. The mean error from all the measurements was 6.65 mm which corresponds to 18.5% of the interspinous gap, proving that the algorithm can identify the vertebral levels with respect to the skin. Successful registration to the skin was also analyzed by assigning a suc- cess ID, where the identification of the level with respect to the patient skin is defined as successful if the difference between the skin (marker) location obtained from the panorama and the location obtained by the sonographer is less than half the intervertebral spacing (measured from sonographer mark- ings placed on the marker strip). Based on this definition, from a total of 166 measurements obtained from the panoramas, 160 of the vertebrae were identified correctly, resulting in a success ID of 160/166. 60 5.2. Human Subject Validation Figure 5.8: Registering the interspinous levels obtained from the panoramas to the skin. Registering panorama to marker on skin compared to sonographer marks (mm) n observations 166 mean error 6.65 SD 4.30 RMS error 7.86 Table 5.6: Result of registering the panoramas to the skin. 61 5.2. Human Subject Validation 5.2.4 Discussion Validation of the overall guidance system was achieved by creating panorama images from human subjects (n = 20), for depicting the spinal anatomy and identifying the vertebrae. The results show that a miniature optical guidance system with a single inexpensive camera and a simple setup can be constructed and calibrated with sufficient accuracy for creating panoramas of the spine, detection of the epidural spaces, and for identification of the vertebrae and the interspinous distances, to enable guidance of puncture site, trajectory, and depth of insertion with respect to the patient skin. The required accuracy for the epidural anesthesia application is divided into several components: identification of the vertebrae, selection of punc- ture site, and estimation of the depth of needle insertion. Based on the results obtained in this section, the proposed guidance system satisfies these accuracies: • identification of the vertebrae with respect to the skin, with a mean error of 18.5% over the interspinous distances • identification of intervertebral spaces, with a mean error of 4.44 mm • depth estimation for needle insertion, with a mean error of 1.69 mm (4.23%) The 6.65 mm mean error for relating the desired target space to the skin means that this technique provides only an approximate puncture site, and that real-time ultrasound guidance should be used for selecting the final puncture site. A change in the puncture site location (e.g. 6.6 mm) has a small effect on the depth of the needle insertion to the target (less than 1 mm based on trigonometry with an average depth of 43.8 mm [90]), but the change in trajectory may produce bone contact. With real-time ultrasound guidance during needle insertion, the bone contact problem is avoided, while retaining the high accuracy of the depth estimation which is the most critical aspect of this procedure to avoid overshoot and nerve damage. Sources of error include the errors in camera calibration, tracking algo- rithm and ultrasound image calibration, which were explained in detail in 62 5.2. Human Subject Validation Section 4.5. The cross ratio property used in designing the marker strip can be sensitive to noise, including errors in the location of the marker feature points, caused by lower image quality and varying lighting conditions, and errors in feature detection. These can cause a change in the value of the invariant, therefore the identifier is defined by a range containing a mini- mum and maximum cross ratio value for each pattern. Through high-quality printing of the marker strips in order to ensure high contrast features and reasonable camera image quality, reliable and accurate pattern recognition can be achieved. Given the fact that the marker is assumed to be on a flat surface, errors can also arise from skin deformations as well as the curvature of the back, causing distortions in the marker. Even though weighted average estima- tion of two poses obtained from the same image was used to improve these effects, further analysis should be performed in the future to compensate for curvature and marker deformations. Some error results from interpolat- ing the positions obtained from the camera over all the ultrasound frames. However, given the fast acquisitions rate, and the smooth movement of the probe, the change in position and orientation from one frame to the next is very small so the error due to interpolation is insignificant in comparison to other errors. For the intervertebral spacings, the mean of the sonographer measure- ment over all distances for all the subjects was 2.98 cm, compared to a mean of 3.33 cm for the panorama measurements. Figure 5.9 compares the two measurements against each other over all the distances, showing that the panorama measurements tend to be larger than the sonographer measure- ments for most distances. Given that the mean sonographer measurement at each individual level was also smaller than the mean measurement obtained from the panoramas at that level, there is a high possibility of a system- atic error in the measurements obtained from the panoramas. This error can be caused by the blurring effect of ultrasound images in the elevational direction of the panoramas, due to the ultrasound beam thickness, which can cause landmarks or anatomical structures to appear further apart than they actually are. Also the assumption in the pose estimation algorithm 63 5.2. Human Subject Validation that the marker strip is placed on a flat surface, and not taking into account the curvature of the back, can also contribute to this error by causing the lamina and interspinous gaps to appear more stretched and extended in the panoramas. Figure 5.9: Panorama vs. sonographer interspinous distance measurements. 64 Chapter 6 Real-time Implementation and GUI Design The tracking algorithm was implemented in C++ for live position estima- tion of ultrasound images with respect to the marker frame of reference, using open-source computer vision libraries and medical imaging software packages. Visualization and further processing of the tracked ultrasound images was achieved in 3D Slicer. Slicer supports an IGT protocol, which allows receiving of US images and tracking information from a network con- nection. This chapter provides and overview of the software architecture and details of the implementation and the graphical user interface. The sys- tem was tested by depicting volumes, reslices, and live tracked ultrasound images obtained from a spine phantom. 6.1 Software Architecture, Packages and Libraries Camera images were obtained through continuous video capture using the camera’s SDK (image size 1280× 1024, frame rate 15 FPS), and processed using OpenCV, a commonly used library of programming functions for real- time computer vision. The implemented algorithm is similar to the steps described in section 4.3. Camera images are undistorted and smoothed with a Gaussian filter before corner detection is performed using a feature de- tector based on the Harris operator. The detected corners are refined to sub-pixel accuracies through an iterative OpenCV function, are grouped in four, and the collinearity test is performed to detect the ones corresponding 65 6.2. Implementation to the same pattern. The cross-ratio is then computed to find the position of the pattern on the marker strip by comparing the value to a database containing the minimum and maximum cross ratio values for each pattern, as well as the corresponding 3D coordinates of the corners. Using this in- formation, together with the 2D image coordinates of the corners and the intrinsic parameters of the camera, the pose of the camera is estimated. This was done using eight feature points and their respective 3D coordinates by minimizing the reprojection error in the least square sense. Ultrasound image acquisition was consolidated into the same thread as the camera tracking, to synchronize the two data streams. While grab- bing position information from the cameras, ultrasound images (image size 640×480) were obtained from the ultrasound machine through a VTK-based C++ class obtained from the SynchroGrab software [9], which uses the Ul- terius SDK to connect to the SonixRP interface over a network connection in real-time, for direct streaming of ultrasound images. SynchroGrab is a software package which allows collection of interventional ultrasound images from an open-interface ultrasound system such as the SonixRP, and their synchronization with a stream of pose measurements. The ultrasound images, and their corresponding position information ob- tained from the camera images, were sent to the 3D Slicer software through OpenIGTLink, which allows the exchange of images and tracking informa- tion using a standardized network protocol. The OpenIGTLink protocol is a simple but extensible data interface for connecting software and hard- ware and transferring various types of data used in image-guided therapy applications, and is directly implemented in 3D Slicer as one of the modules. The overall software design is shown in Figure 6.1. 6.2 Implementation To test the implementation of the system, tracked ultrasound images were obtained from a lumbar spine phantom, and sent to 3D Slicer for visual- ization, and further processing. The software was implemented on a PC 66 6.2. Implementation Figure 6.1: Software structure, for sending ultrasound images and tracking information to 3D Slicer. 67 6.2. Implementation (Windows XP, Intel Core2 2.93 GHz, 3.24 Gb of RAM) running a copy of Slicer 3.6. The ultrasound images and their position information were trans- ferred to a TCP/IP socket, at a rate of 5-10 FPS, which the OpenIGTLink module of Slicer3 then reads and makes available within Slicer. 6.2.1 Experiments on Spine Phantom The phantom consists of an artificial lumbar spine model embedded in a PVC substance, as shown in Figure 6.2. The marker strip is attached lateral to the spine in clear view of the camera. Using pre-recorded 2D ultrasound images and their position information, an ultrasound volume of the phantom was constructed through a nearest neighbor interpolation approach (volume size = 35 MB), and was rendered in Slicer for visualization purposes. Live track- ing was then implemented by moving the transducer along the phantom, as well as removing and replacing the transducer at different desired locations, with Slicer depicting the ultrasound image at each updated location with respect to the volume. Figure 6.2: Lumbar spine phantom with the marker strip, for demonstrating the real-time implementation. 68 6.2. Implementation 6.2.2 Graphical User Interface The ultrasound volume obtained from the phantom was rendered in Slicer through VTK OpenGL 3D texture mapping. Figure 6.3 shows the graphical user interface in Slicer, depicting the volume as well as the live tracked 2D ultrasound image with respect to the volume. A model of the marker strip is also implemented that shows the absolute position of the ultrasound image with respect to the marker as a red line that updates as the transducer moves. The other windows in the GUI are used for depicting the live 2D ultrasound image (top right), as well as different reslices obtained from the volume at angles selected by the user, depicting the lamina and other desired target regions (center and bottom right). Figure 6.3: Graphical user interface in Slicer, depicting the position of the tracked B-mode ultrasound with respect to the volume as well as the marker. 69 6.3. Discussion 6.3 Discussion 3D Slicer has been used for implementing the tracking system live, by us- ing the OpenCV library for obtaining pose information from the camera images, using Ulterius and Synchrograb software packages for receiving ul- trasound images from the SonixRP, matching ultrasound images with the corresponding position information and sending these to 3D Slicer using an OpenIGTLink protocol. These software packages and the implementation can also be installed on the ultrasound machine itself, with the camera con- nected to the machine via USB2.0, without the need for a second computer. In the current implementation, the volumes were generated offline and imported into Slicer for visualization and reslicing. This can be improved in the future by implementing the volume re-construction and panorama generation in real-time, as the ultrasound images and their position infor- mation are sent into Slicer. Furthermore, for this application synchroniza- tion is achieved by incorporating the camera tracking and the ultrasound acquisition into the same thread. Given the faster frame rate of ultrasound acquisition, this solution is not optimal and should be improved by obtaining ultrasound images at the original frame rate, and interpolating the tracking data to find the position of the probe at the time the ultrasound image was acquired. 70 Chapter 7 Conclusion 7.1 Summary of Contributions A low-cost 3D ultrasound panorama guidance system has been proposed, using a single miniature camera attached directly on the transducer, and a specialized adhesive marker strip attached on the skin, which allows 6- DOF absolute position estimation of the camera and the transducer over the whole range of the spine with respect to the patient. The goal is to depict the spinal anatomy and identify the vertebrae for a wide range of patients, and provide guidance of puncture site, trajectory, and depth of insertion for epidural anesthesia applications. Optical tracking of the ultrasound transducer, using external tracking systems where the position of the transducer is calculated with respect to the fixed external base of the tracker, has been performed in the past. Mounting miniature cameras on the transducer has also been explored for estimating needle pose and trajectory with respect to the probe for percutaneous needle insertions and biopsies. However, position tracking of ultrasound images with respect to the skin using a single camera mounted on the transducer has never been tried before. The present device eliminates the external base coordinate system, thereby reducing the tracker range, reducing cost compared to external trackers, as well as increasing simplicity in terms of additional steps for patient preparation and adding equipment to a standard ultrasound machine. Other significant advantages include reduction of line- of-sight problems, and accommodation of body motion during scanning. Furthermore, the ability of the system in providing absolute, not relative, tracking with respect to the patient allows for the transducer to be removed and replaced during scanning. This allows the anesthesiologist to replace the 71 7.2. Limitations transducer at the desired skin location, selected from the panorama images, for real-time guidance during needle insertion. The method was validated on a phantom against the Optotrak tracking system, as well as on 20 human subjects against measurements obtained by an experienced sonographer. The results showed that a miniature optical guidance system with a single low-cost camera can be designed and cali- brated for creating panoramas of the spine, detection of the epidural spaces, and identification of vertebrae and interspinous gaps. The result of this research can have a significant impact on the speed and confidence of the operator performing epidural anesthesia, by linking pre-puncture scanning in the preparation room, where more time is available, to the scanning in the operation room. Improved guidance of epidural anesthesia will impact patient care through the reduction in incidence of dura puncture and nerve injury, and increased efficiency of the surgical unit. The system can also be used as a valuable training tool for residents and starting anesthesiologist, to improve the steep learning curve associated with these procedures. 7.2 Limitations Sources of error in the algorithm include errors in the intrinsic calibration of the camera, errors in feature detection and pose estimation, and errors of ultrasound image calibration (described in detail in Section 4.5). The system was first validated on a phantom and compared against the Optotrak tracking system, resulting in a 0.9 mm RMS error and a maximum error of 3.5 mm between camera and Optotrak. For the human data, the proposed guidance system satisfies the accuracy for identification of the vertebrae with respect to the skin (average error is 18.5% of the interspinous distances), identification of intervertebral spaces (average error of 4.44 mm), and depth of needle insertion (average error of 1.69 mm). Given that the average error for relating the interspinous gaps (target spaces) to the skin was 6.65 mm, this system can provide an approximate puncture site and should be used with real-time ultrasound for selecting the final puncture site. This error has a very small effect on the depth of needle insertion to the target, but may 72 7.3. Other Potential Applications produce bone contact which can be compensated with real-time ultrasound guidance during needle insertion. Given that the marker strip allows 6-DOF position estimation with re- spect to the skin, the system is expected to accommodate patient motion during scanning, however this was not explicitly tested. Human data for system validation was obtained from the subjects in a comfortable upright seated position, without imposing any constraints. However, rapid patient movements might cause errors in camera pose estimation depending on cam- era frame rate and lags, and increase the errors caused by smoothing and interpolating the positions obtained from the camera over all the ultrasound frames. Compression from the ultrasound transducer on the skin can also cause a small difference in the measurement of the ligamentum flavum depth. It has been shown that transducer pressure can change the depth of the epidural space by 6.4% (2.8 mm with an average depth of 43.8 mm) [90]. Given the high accuracy in the system’s depth estimation, and the thickness of the ligamentum flavum itself (5-6 mm [17]), this effect can be compensated to a degree. Furthermore, since this guidance system will be combined with traditional loss-of-resistance during needle insertion, the accuracies are promising for selecting an approximate puncture site and trajectory to the epidural space. 7.3 Other Potential Applications Even though the primary intended application for this system is navigation and guidance of the commonly performed epidural needle insertions, other clinical applications can also benefit from the techniques developed in this research. These include spinal taps where the needle is inserted along the same trajectory but at a deeper depth, spinal biopsies, nerve blocks or re- gional anesthesia techniques including intercostal and paravertebral nerve blocks, and spinal surgery. The ability to relate US images to the skin sur- face could also have a significant impact on other image guided procedures, such as port planning for laparoscopy. The next expected clinical applica- 73 7.4. Future Work tion to be investigated with this guidance system is regional anesthesia using needle insertions into the facet joints of the vertebrae. 7.4 Future Work Future research directions will involve improving the graphical user interface to create 3D ultrasound volumes and matched panoramas of both the spine and the skin surface (with the marker) in real time, for registering the ultra- sound panoramas directly to the skin. Another feature that can be added to this graphical user interface is automated ultrasound image analysis to show target, trajectory, and depth of needle insertion, in an effort to increase the ease of use in clinical applications. Further a priori information about the desired target regions can be added to the system by incorporation existing X-ray, CT, MRI, or anatomical atlas images, identifying the desired target region from these images with respect to the marker coordinate system, and thereby aligning subsequent ultrasound images with the target. Further- more, using the position information of the ultrasound images with respect to the skin, an augmented reality system can be designed for overlaying the ultrasound images on the skin surface to further improve guidance of needle insertion. The marker design can be improved to ensure accuracy over larger lengths of the marker strip, and to guarantee the uniqueness of the patterns, by im- plementing patterns with six collinear features instead of four, leading to three cross ratio values (instead of one) that can be checked for higher re- liability. Other improvement can be made to the tracking algorithm by incorporating models to estimate skin deformations and the curvature of the back, and compensate for these in the pose estimation algorithm. Given that the CMOS sensor of the camera is sensitive to near infrared light, it would be feasible to add a low-cost infrared diode as a constant light source for reliable skin imaging under variable lighting conditions. The camera-probe combination can be miniaturized and made sterile for clinical application, or the camera can be incorporated directly within the ultra- sound transducer casing for future use. The marker can also be printed on 74 7.4. Future Work sterile sheets for use in clinical settings, or printed on the patient body as a tattoo, to be used for both pre-puncture scanning and inside the operating room. 75 Bibliography [1] A. Afzal, N. Haider, and R.W. Rosenquist. Thoracic epidural anesthesia in the perioperative period. Techniques in Regional Anesthesia and Pain Management, 6(2):50–55, 2002. [2] A. Ali and R. Logeswaran. A visual probe localization and calibration system for cost-effective computer-aided 3D ultrasound. Computers in Biology and Medicine, 37(8):1141–1147, 2007. [3] D. Atkinson, M. Burcher, J. Declerck, and J.A. Noble. Respiratory motion compensation for 3-D freehand echocardiography. Ultrasound in Medicine & Biology, 27(12):1615–1620, 2001. [4] M. Balki, Y. Lee, S. Halpern, and J.C.A. Carvalho. Ultrasound imaging of the lumbar spine in the transverse plane: the correlation between estimated and actual depth to the epidural space in obese parturients. International Anesthesia Research Society, 108(6):1876–1881, 2009. [5] D.C. Barratt, A.H. Davies, A.D. Hughes, S.A. Thom, and K.N. Humphries. Accuracy of an electromagnetic three-dimensional ultra- sound system for carotid artery imaging. Ultrasound in medicine & biology, 27(10):1421–1425, 2001. [6] D.C. Barratt, G.P. Penney, C.S.K. Chan, M. Slomczykowski, T.J. Carter, P.J. Edwards, and D.J. Hawkes. Self-calibrating 3D-ultrasound- based bone registration for minimally invasive orthopedic surgery. IEEE Transactions on Medical Imaging, 25(3):312–323, 2006. [7] S. Berg, H. Torp, D. Martens, E. Steen, S. Samstad, I. H∅ivik, and B.Olstad. Dynamic three-dimensional freehand echocardiography us- 76 Bibliography ing raw digital ultrasound data. Ultrasound in Medicine & Biology, 25(5):745–753, 1999. [8] R. Boesecke, T. Bruckner, and G. Ende. Landmark based correlation of medical images. Physics in Medicine and Biology, 35(1):121–129, 1990. [9] J. Boisvert, D. Gobbi, S. Vikal, R. Rohling, G. Fichtinger, and P. Abol- maesumi. An open-source solution for interactive acquisition, process- ing and transfer of interventional ultrasound images. MIDAS Journal - Systems and Architectures for Computer Assisted Interventions (MIC- CAI Workshop), 2008. [10] J.Y. Bouguet. Visual methods for three-dimensional modeling. PhD thesis, California Institute of Technology, Pasadena, CA, 1999. [11] C. R. Broadbent, W. B. Maxwell, R. Ferrie, D. J. Wilson, M. Gawne- Cain, and R. Russell. Ability of anaesthetists to identify a marked lumbar interspace. Anaesthesia, 55(11):1122–1126, 2000. [12] J. Canny. A computational approach to edge detection. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, PAMI-8, 1986. [13] A.S. Carney, N. Patel, D.L. Baldwin, H.B. Coakham, and D.R. Sande- man. Intra-operative image guidance in otolaryngology - the use of the ISG viewing wand. The Journal of Laryngology and Otology, 110:322– 327, 1996. [14] J.C.A. Carvalho. Ultrasound-facilitated epidurals and spinals in obstet- rics. Anesthesiology Clinics, 26(1):145–158, 3 2008. [15] C. Chan, F. Lam, and R. Rohling. A needle tracking device for ul- trasound guided percutaneous procedures. Ultrasound in Medicine & Biology, 31(11):1469–1483, 2005. [16] T.K. Chen, A.D. Thurston, R.E. Ellis, and P. Abolmaesumi. A real- time freehand ultrasound calibration system with automatic accuracy feedback and control. Ultrasound in Medicine & Biology, 35(1):79–93, 2009. 77 Bibliography [17] D.H. Chestnut. Obstetric Anesthesia: Principles and Practice. St. Louis, Mosby Inc., 3rd edition, 2004. [18] M.L. Chuang, M.G. Hibberd, R.A. Beaudin, M.G. Mooney, M.F. Riley, J.T. Fearnside, and P.S. Douglas. Patient motion compensation during transthoracic 3-D echocardiography. Ultrasound in Medicine & Biology, 27(2):203–209, 2001. [19] E.R. Cosman. Apparatus and method for determining the location of an anatomical target with reference to a medical apparatus. U.S. Patent 6,405,072, 2002. [20] G.R. de Oliveira Filho. The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method. Anesthesia and Analgesia, 95(2):411–416, 2002. [21] D.F. Dementhon and L.S. Davis. Model-based object pose in 25 lines of code. International Journal of Computer Vision, 15:123–141, 1995. [22] P.R. Detmer, G. Bashein, T. Hodges, K.W. Beach, E.P. Filer, D.H. Burns, and D.E. Strandness Jr. 3D ultrasonic image feature localiza- tion based on magnetic scanhead tracking: In vitro calibration and validation. Ultrasound in Medicine & Biology, 20(9):923–936, 1994. [23] G. Fichtinger, A. Deguet, K. Masamune, E. Balogh, G.S. Fischer, H. Mathieu, R.H. Taylor, S.J. Zinreich, and L.M. Fayad. Image over- lay guidance for needle insertion in ct scanner. IEEE Transactions on Biomedical Engineering, 52(8):1415–1424, 2005. [24] G. Flaccavento, P. Lawrence, and R. Rohling. Patient and probe track- ing during freehand ultrasound. In Medical Image Computing and Computer-Assisted Intervention, volume 3217, pages 585–593. 2004. [25] G. Furness, M.P. Reilly, and S. Kuchi. An evaluation of ultrasound imaging for identification of lumbar intervertebral level. Anaesthesia, 57(3):277–280, 2002. 78 Bibliography [26] A.H. Gee, R.J. Housden, P. Hassenpflug, G.M. Treece, and R.W. Prager. Sensorless freehand 3D ultrasound in real tissue: Speckle decorrelation without fully developed speckle. Medical Image Analy- sis, 10(2):137–149, 2006. [27] R.M. Giebler, R.U. Scherer, and J. Peters. Incidence of neurologic com- plications related to thoracic epidural catheterization. Anesthesiology, 86(1):55–63, 1997. [28] N.D. Glossop. Advantages of optical compared with electromagnetic tracking. The Journal of Bone and Joint Surgery, 91:23–28, 2009. [29] T. Grau, E. Bartusseck, R. Conradi, E. Martin, and J. Motsch. Ultra- sound imaging improves learning curves in obstetric epidural anesthesia: a preliminary study. Canadian Journal of Anesthesia, 50:1047–1050, 2003. [30] T. Grau, R.W. Leipold, R. Conradi, E. Martin, and J. Motsch. Effi- cacy of ultrasound imaging in obstetric epidural anesthesia. Journal of Clinical Anesthesia, 14(3):169–175, 2002. [31] T. Grau, R.W. Leipold, S. Delorme, E. Martin, and J. Motsch. Ultra- sound imaging of the thoracic epidural space. Regional Anesthesia and Pain Medicine, 27(2):200–206, 2002. [32] T. Grau, R.W. Leipold, S. Fatehi, E. Martin, and J. Motsch. Real-time ultrasonic observation of combined spinalepidural anaesthesia. Euro- pean Journal of Anaesthesiology, 21(01):25, 2004. [33] I. Hacihaliloglu, R. Abugharbieh, A.J. Hodgson, and R.N. Rohling. Bone surface localization in ultrasound using image phase-based fea- tures. Ultrasound in Medicine & Biology, 35(9):1475–1487, 2009. [34] D. Harney, C.A. Moran, R. Whitty, S. Harte, M. Geary, and J. Gar- diner. Influence of posture on the incidence of vein cannulation dur- ing epidural catheter placement. European Journal of Anaesthesiology, 22(2):103–106, 2005. 79 Bibliography [35] C. Harris and M.J. Stephens. A combined corner and edge detector. In Fourth Alvey Vision Conference, pages 147–152, 1988. [36] P. Hassenpflug, R.W. Prager, G.M. Treece, and A.H. Gee. Speckle classification for sensorless freehand 3-D ultrasound. Ultrasound in Medicine & Biology, 31(11):1499–1508, 2005. [37] X.C. He and N.H.C. Yung. Corner detector based on global and local curvature properties. Optical Engineering, 47(5):057008, 2008. [38] T. T. Horlocker. Complications of spinal and epidural anesthesia. Anes- thesiology Clinics of North America, 18(2):461–485, 2000. [39] R.J. Housden, A.H. Gee, R.W. Prager, and G.M. Treece. Rotational motion in sensorless freehand three-dimensional ultrasound. Ultrason- ics, 48(5):412–422, 2008. [40] R.J. Housden, A.H. Gee, G.M. Treece, and R.W. Prager. Subsample in- terpolation strategies for sensorless freehand 3D ultrasound. Ultrasound in Medicine & Biology, 32(12):1897–1904, 2006. [41] B.A. Johnson, K.P. Schellhas, and S.R. Pollei. Epidurography and therapeutic epidural injections: Technical considerations and experi- ence with 5334 cases. American Journal of Neuroradiology, 20:697–705, 1999. [42] M.K. Karmakar, X. Li, A.M.-H. Ho, W.H. Kwok, and P.T. Chui. Real- time ultrasound-guided paramedian epidural access: evaluation of a novel in-plane technique. British Journal of Anaesthesia, 102(6):845– 854, 2009. [43] B. Kerby, R. Rohling, V. Nair, and P. Abolmaesumi. Automatic iden- tification of lumbar level with ultrasound. In IEEE Engineering in Medicine and Biology Society, pages 2980–2983, 2008. [44] S. Khosravi, R. Rohling, and P. Lawrence. Image guidance using camera and ultrasound images. In IEEE Ultrasonics Symposium, pages 2251– 2254, 2007. 80 Bibliography [45] D.J. Kopacz, J.M. Neal, and J.E. Pollock. The regional anesthesia “learning curve”. What is the minimum number of epidural and spinal blocks to reach consistency? Regional Anesthesia, 21(3):182–190, 1996. [46] P.D. Kovesi. Matlab and octave functions for computer vision and image processing. School of Computer Science & Software Engineering, The University of Western Australia. [47] J. Kowal, C.A. Amstutz, M. Caversaccio, and L.P. Nolte. On the de- velopment and comparative evaluation of an ultrasound B-mode probe calibration method. Computer Aided Surgery, 8(3):107–119, 2003. [48] K. Kröger, K. Massalha, G. Dobonici, and G. Rudofsky. SiescapeTM: a new sonographic dimension with fictive images. Ultrasound in Medicine & Biology, 24(8):1125 – 1129, 1998. [49] A. Lang, P. Mousavi, G. Fichtinger, and P. Abolmaesumi. Fusion of electromagnetic tracking with speckle-tracked 3D freehand ultrasound using an unscented kalman filter. In Proceedings of SPIE, volume 7265, page 72651A, 2009. [50] D.F. Leotta. An efficient calibration method for freehand 3D ultrasound imaging systems. Ultrasound in Medicine & Biology, 30(7):999–1008, 2004. [51] F. Lindseth, G.A. Tangen, T. Lango, and J. Bang. Probe calibra- tion for freehand 3-D ultrasound. Ultrasound in Medicine & Biology, 29(11):1607–1623, 2003. [52] S.S. Liu, J.E. Ngeow, and J.T. YaDeau. Ultrasound-guided regional anesthesia and analgesia: a qualitative systematic review. Regional Anesthesia & Pain Medicine, 34(1):47–59, 2009. [53] S.S. Liu, W.M. Strodtbeck, J.M. Richman, and C.L. Wu. A compari- son of regional versus general anesthesia for ambulatory anesthesia: A meta-analysis of randomized controlled trials. Anesthesia and Analge- sia, 101(6):1634–1642, 2005. 81 Bibliography [54] M. Loaiza, A. Raposo, and M. Gattass. A novel optical tracking algo- rithm for point-based projective invariant marker patterns. In Advances in Visual Computing, volume 4841, pages 160–169. 2007. [55] D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91–110, 2004. [56] D.G. Lowe. Fitting parameterized three-dimensional models to im- ages. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(5):441–450, 1991. ID: 1. [57] E.P. Lynch, K.J. Welch, and J.M. Carabuena. Thoracic epidural anesthesia improves outcome after breast surgery. Annals of Surgery, 222(5):663–669, 1995. [58] S. Meairs, J. Beyer, and M. Hennerici. Reconstruction and visualiza- tion of irregularly sampled three- and four-dimensional ultrasound data for cerebrovascular applications. Ultrasound in Medicine & Biology, 26(2):263–272, 2000. [59] L. Mercier, T. Lang∅, F. Lindseth, and D.L. Collins. A review of calibration techniques for freehand 3-D ultrasound systems. Ultrasound in Medicine & Biology, 31(4):449–471, 2005. [60] D.M. Muratore and R.L. Galloway. Beam calibration without a phan- tom for creating a 3-D freehand ultrasound system. Ultrasound in Medicine & Biology, 27(11):1557–1566, 2001. [61] M.C. Norris, B.L. Leighton, and C.A. DeSimone. Needle bevel direc- tion and headache after inadvertent dural puncture. Anesthesiology, 70(5):729 – 731, 1989. [62] N. Pagoulatos, D.R. Haynor, and Y. Kim. A fast calibration method for 3-D tracking of ultrasound images using a spatial localizer. Ultrasound in Medicine & Biology, 27(9):1219–1229, 9 2001. 82 Bibliography [63] S. Pieper, M. Halle, and R. Kikinis. 3D slicer. In IEEE International Symposium on Biomedical Imaging: Nano to Macro, pages 632–635, 2004. [64] T.C. Poon and R.N. Rohling. Comparison of calibration methods for spatial tracking of a 3-D ultrasound probe. Ultrasound in Medicine & Biology, 31(8):1095–1108, 2005. [65] T.C. Poon and R.N. Rohling. Three-dimensional extended field-of-view ultrasound. Ultrasound in Medicine & Biology, 32(3):357 – 369, 2006. [66] T.C. Poon and R.N. Rohling. Tracking a 3-D ultrasound probe with constantly visible fiducials. Ultrasound in Medicine & Biology, 33(1):152–157, 2007. [67] R.W. Prager, A. Gee, and L. Berman. Stradx: real-time acquisition and visualization of freehand three-dimensional ultrasound. Medical Image Analysis, 3(2):129–140, 1999. [68] R.W. Prager, A.H. Gee, G.M. Treece, C.J.C. Cash, and L.H. Berman. Sensorless freehand 3-D ultrasound using regression of the echo inten- sity. Ultrasound in Medicine & Biology, 29(3):437–446, 2003. [69] R.W. Prager, R.N. Rohling, A.H. Gee, and L. Berman. Rapid calibra- tion for 3-D freehand ultrasound. Ultrasound in Medicine & Biology, 24(6):855–869, 1998. [70] L. Quan and Z Lan. Linear n-point camera pose determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8):774– 780, 1999. [71] H. Rafii-Tari, P. Abolmaesumi, and R. Rohling. Panorama ultrasound for guiding epidural anesthesia: A feasibility study. In Information Pro- cessing in Computer-Assisted Interventions, volume 6689, pages 179– 189. 2011. 83 Bibliography [72] A. Rasoulian, J. Lohser, M. Najafi, H. Rafii-Tari, D. Tran, A. Kamani, V. Lessoway, P. Abolmaesumi, and R. Rohling. Utility of prepuncture ultrasound for localization of the thoracic epidural space. Canadian Journal of Anesthesia, 58(9):815–823, 2011. [73] H.F. Reinhardt, G.A. Horstmann, and O. Gratzl. Sonic stereometry in microsurgical procedures for deep-seated brain tumors and vascular malformations. Neurosurgery, 32(1):51–57, 1993. [74] F. Reynolds. Logic in the safe practice of spinal anaesthesia. Anaes- thesia, 55(11):1045–1046, 2000. [75] R. Rohling, A. Gee, and L. Berman. Three-dimensional spatial com- pounding of ultrasound images. Medical Image Analysis, 1(3):177 – 193, 1997. [76] R. Rohling, P. Munger, J.M. Hollerbach, and T. Peters. Compari- son of relative accuracy between a mechanical and an optical posi- tion tracker for image-guided neurosurgery. Journal of Image Guided Surgery, 1(1):30–34, 1995. [77] F. Sauer and A. Khamene. Video-assistance for ultrasound guided biopsy. U.S. Patent 6,612,991, 2003. [78] F. Sauer, A. Khamene, and B. Bascle. System and method for three- dimensional (3D) reconstruction from ultrasound images. U.S. Patent 7,079,132, 2006. [79] F. Sauer, A. Khamene, and B. Bascle. Marking 3D locations from ultrasound images. U.S. Patent 7,251,352, 2007. [80] O. Scherzer and A. Schoisswohl. A fast and robust algorithm for 2D/3D panorama ultrasound data. Real-Time Imaging, 8(1):53–60, 2002. [81] B. Schramm and J. Kay. Thoracic epidural for thoracic trauma. Tech- niques in Regional Anesthesia and Pain Management, 6(1):19–26, 2002. 84 Bibliography [82] D. Shepard. A two-dimensional interpolation function for irregularly- spaced data. In Proceedings of the 1968 ACM National Conference, pages 517–524, 1968. [83] P.D. Sorenson and D.A. Dickson. Ultrasound imaging system for scan- ning the human back. U.S. Patent 4,457,311, 1984. [84] J. S. Sprigge and S. J. Harper. Accidental dural puncture and post dural puncture headache in obstetric anaesthesia: presentation and manage- ment: A 23-year survey in a district general hospital. Anaesthesia, 63(1):36–43, 2008. [85] P.J. Stolka, H.J. Kang, M. Choti, and E.M. Boctor. Multi-DoF probe trajectory reconstruction with local sensors for 2D-to-3D ultrasound. In International Symposium on Biomedical Imaging: From Nano to Macro, pages 316–319, 2010. [86] T. Suk and J. Flusser. Point-based projective invariants. Pattern Recog- nition, 33(2):251–261, 2000. [87] D. Tran, K.W. Hor, V.A. Lessoway, A.A. Kamani, and R.N. Rohling. Adaptive ultrasound imaging of the lumbar spine for guidance of epidu- ral anesthesia. Computerized Medical Imaging and Graphics, 33(8):593– 601, 12 2009. [88] D. Tran, A. Kamani, E. Al-Attas, V. Lessoway, S. Massey, and R. Rohling. Single-operator real-time ultrasound-guidance to aim and insert a lumbar epidural needle. Canadian Journal of Anesthesia, 57:313–321, 2010. [89] D. Tran, A.A. Kamani, V.A. Lessoway, C. Peterson, K.W. Hor, and R.N. Rohling. Preinsertion paramedian ultrasound guidance for epidu- ral anesthesia. Anesthesia & Analgesia, 109(2):661–667, 2009. [90] D. Tran and R.N. Rohling. Automatic detection of lumbar anatomy in ultrasound images of human subjects. IEEE Transactions on Biomed- ical Engineering, 57:2248 –2256, 2010. 85 Bibliography [91] T.A. Tuthill, J.F. Krcker, J.B. Fowlkes, and P.L. Carson. Auto- mated three-dimensional us frame positioning computed from eleva- tional speckle decorrelation. Radiology, 209(2):575–582, 1998. [92] D.A. Ullman, R.E. Rollin, J.B. Fortune, T.M. Kennedy, and B.B. Greenhouse. The treatment of patients with multiple rib fractures us- ing continuous thoracic epidural narcotic infusion. Regional Anesthesia and Pain Medicine, 14(1):43–47, 1989. [93] R. van Liere and J.D. Mulder. Optical tracking using projective invari- ant marker pattern properties. In IEEE Virtual Reality, pages 191–198, 2003. [94] B.T. Veering. Cardiovascular and pulmonary effects of epidural anaes- thesia. Minerva Anestesiologica, 69(5):433–437, 2003. [95] L. Wang, R. Zou, S. Weidert, J. Landes, E. Euler, D. Burschka, and N. Navab. Closed-form inverse kinematics for intra-operative mobile C-arm positioning with six degrees of freedom. SPIE, 7964(1):79641A, 2011. [96] M.J.A. Wilson. Epidural endeavour and the pressure principle. Anaes- thesia, 62(4):319–322, 2007. [97] W.Y. Zhang, R.N. Rohling, and D.K. Pai. Surface extraction with a three-dimensional freehand ultrasound system. Ultrasound in Medicine & Biology, 30(11):1461–1473, 2004. [98] X. Zhang, S. Fronz, and N. Navab. Visual marker detection and decod- ing in AR systems: A comparative study. In 1st International Sympo- sium on Mixed and Augmented Reality, pages 97–106, 2002. [99] X. Zhang and N. Navab. Coded visual markers for tracking and camera calibration in mobile computing systems. U.S. Patent 7,809,194, 2010. 86