Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Touchless gesture recognition system for imaging controls in sterile environment Hsieh, Derick 2014

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2014_september_hsieh_derick.pdf [ 2.08MB ]
Metadata
JSON: 24-1.0167492.json
JSON-LD: 24-1.0167492-ld.json
RDF/XML (Pretty): 24-1.0167492-rdf.xml
RDF/JSON: 24-1.0167492-rdf.json
Turtle: 24-1.0167492-turtle.txt
N-Triples: 24-1.0167492-rdf-ntriples.txt
Original Record: 24-1.0167492-source.json
Full Text
24-1.0167492-fulltext.txt
Citation
24-1.0167492.ris

Full Text

   TOUCHLESS GESTURE RECOGNITION SYSTEM FOR IMAGING CONTROLS IN STERILE ENVIRONMENT  BY  DERICK HSIEH B.A.Sc., University of British Columbia, 2011   A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  MASTER OF APPLIED SCIENCE  in  The Faculty of Graduate and Postdoctoral Studies   (Electrical and Computer Engineering) The University of British Columbia (Vancouver) June 2014   ©  Derick Hsieh, 2014 ii  ABSTRACT Physicians often rely on a patient’s imaging to accurately complete a surgical procedure. To be able to navigate and manipulate these imaging files, physicians resort to using the traditional keyboard and mouse. However, keyboards and computer mice are common mediums for bacterial transfer. As a result, physicians need to re-perform the time-consuming scrubbing techniques after interaction with these devices.  With gesture-based control systems becoming an alternative interface over traditional mice and keyboard input systems, new interactive methods can be implemented in a medical environment. Gesture-based interactions allow touchless control of a system that removes the need of the re-sterilization process. This may reduce procedure time and allow the physician to focus on the primary task at hand.  We propose a simple method where the primary user can perform most common interactions such as scroll, zoom, pan and window width/level adjustments with just one hand using a Leap Motion®  sensor and an open source DICOM viewer, Weasis. The tool, developed as an open-source plugin for the Weasis PACS system, gives the user the ability to use one hand to efficiently manipulate medical imaging data. Our tool can be easily integrated into existing systems, requires no calibration prior to each usage, and is very low cost. An experiment was conducted at a local hospital, with 9 radiologists, 3 surgeons, 3 operating room support staff and 1 engineer to validate the adoptability and usability of our plugin tool. From the results, we can conclude that the participants are receptive to our hand-gesture recognition system as an alternative to using the traditional mouse and keyboard when viewing the imaging or to asking an assistant outside of the sterile field to operate the computer. iii  PREFACE The prototyping of force sensitive resistor (FSR) and clear FSR fabrications reported in Section 4.1 were assisted by Sensitronics and UBC AMPEL Nanofabrication Facility (ANF) support engineer. However, the firmware and the gesture recognition library development were collaborated with Louise Oram, a master student from the Department of Computer science, and Evelyn Tsai, a master student from the Department of Computer Engineering. I was the primary investigator and developer for the software design described in Section 4.2 and Chapter 5. The work presented in this thesis is under review as a conference submission. The co-authors include Dr. Darra Murphy from St. Paul’s Hospital (SPH) and my supervisor Dr. Philippe Kruchten. The user study described in Chapter 6 was conducted with the help of Dr. Darra Murphy (SPH) under the approval of the University of British Columbia (UBC) Behavioural Research Ethics Board (BREB) with the certificate number H13-01972. The analysis of the data were tabulated and totalled by Excel for interpretation.  iv  TABLE OF CONTENTS ABSTRACT ........................................................................................................................ ii PREFACE .......................................................................................................................... iii TABLE OF CONTENTS ................................................................................................. iiiv LIST OF TABLES ............................................................................................................ vii LIST OF FIGURES ......................................................................................................... viii LIST OF ABBREVIATIONS ............................................................................................ ix ACKNOWLEDGMENTS .................................................................................................. x Chapter 1 INTRODUCTION ........................................................................................ 1 1.1. Contribution ......................................................................................................... 2 1.2. Thesis Overview ................................................................................................... 3 Chapter 2 BACKGROUND .......................................................................................... 4 2.1. Picture Archiving and Communication System (PACS) ..................................... 4 2.2. Sterile Environment Constraints and Problems ................................................... 5 Chapter 3 RELATED WORK ....................................................................................... 8 3.1. Vision-Based Gesture Systems ............................................................................ 8 3.2. Speech-Based Systems ......................................................................................... 9 3.3. Kinect-Based Gesture Systems ............................................................................ 9 3.4. Leap-Based Gesture Systems ............................................................................. 11 3.5. Other Gesture Systems ....................................................................................... 12 Chapter 4 RESEARCH ROADMAP ........................................................................... 14 4.1. Force Sensitive Resistors ................................................................................... 14 4.1.1. Creating a Firmware for FSR Matrix .......................................................... 15 4.1.2. Manufacturing the Clear FSR ..................................................................... 16 v  4.1.3. Alternative Solution .................................................................................... 17 4.2. Microsoft Xbox Kinect....................................................................................... 17 4.2.1. Gesture and Speech Recognition System ................................................... 17 Chapter 5 DESIGN AND IMPLEMENTATIONS ..................................................... 20 5.1. Leap Motion ....................................................................................................... 20 5.1.1. The Physical Hardware Device ................................................................... 20 5.1.2. The Software ............................................................................................... 21 5.2. Leap WEASIS-Plugin (LeWP) .......................................................................... 21 5.2.1. System Architecture .................................................................................... 21 5.2.2. Gesture Recognition Functions and its Algorithms .................................... 23 5.2.2.2. Cursor Navigation ................................................................................... 24 5.2.2.3. Two-Finger Image Scrolling ................................................................... 25 5.2.2.4. Two-Finger Image Zoom ........................................................................ 26 5.2.2.5. Three-Finger Window Width (WW) and Window Level (WL) ............. 27 5.2.2.6. Four-Finger Image Pan............................................................................ 28 5.2.2.7. Window Width and Window Level Presets ............................................ 28 5.2.2.8. Next and Previous Image Scroll .............................................................. 29 5.2.3. User Interface .............................................................................................. 29 Chapter 6 VALIDATION AND RESULTS ................................................................ 32 6.1. Experiment and Validation Method ................................................................... 32 6.2. Validation Results .............................................................................................. 33 6.3. Difficulties .......................................................................................................... 37 6.4. Discussion .......................................................................................................... 37 Chapter 7 FUTURE WORK AND CONCLUSIONS ................................................. 40 7.1. Future Work ....................................................................................................... 41 vi  BIBLIOGRAPHY ............................................................................................................. 42 APPENDICES .................................................................................................................. 47 A.1 Consent Form ......................................................................................................... 47 A.2 Questionnaire Form................................................................................................ 48 A.3 Complete Questionnaire results ............................................................................. 52 A.4 LeWP Software ...................................................................................................... 54    vii  LIST OF TABLES TABLE 1          SHOWS THE PROBLEMS AND REASONING FOR THE DIFFERENT METHODS OF FSR FABRICATION. .......................................................................................................................... 16 TABLE 2          A COMPARISON CHART OF DEVICES EXPLORED IN THIS RESEARCH. ...................................... 38   viii  LIST OF FIGURES FIGURE 1       WEASIS DICOM VIEWER [32]...................................................................................................... 4 FIGURE 2       OUR FSR DEVICE WITH ARDUINO® CIRCUIT TO READ RAW VALUES. ...................................... 15 FIGURE 3       OUR KINECT GESTURE AND SPEECH RECOGNITION SYSTEM SETUP. ....................................... 19 FIGURE 4       LEAP MOTION® DEVICE. ........................................................................................................... 20 FIGURE 5       FLOWCHART DIAGRAM OF LEWP GESTURE RECOGNITION SYSTEM. ...................................... 22 FIGURE 6       A TOP-DOWN VIEW OF THE LEWP ZONES FOR GESTURE DETECTION. .................................... 23 FIGURE 7       MOUSE CURSOR NAVIGATION GESTURE WITH 1-FINGER HAND POSTURE. ............................ 25 FIGURE 8       ZOOM FUNCTION BY USING FAST FINGER PINCHING AND EXPANDING MOVEMENTS........... 26 FIGURE 9       SCROLL GESTURE BY MOVING 2-FINGER HAND POSTURE UP AND DOWN. ............................ 26 FIGURE 10     WINDOW WIDTH (WW) AND WINDOW LEVEL (WL) ADJUSTMENTS WITH 3-FINGER HAND POSTURE. THE WW ADJUSTMENTS ARE INTO THE PAGE (HAND PALM VERTICALLY DOWNWARD) AND OUT OF THE PAGE (HAND PALM VERTICALLY UPWARD). ........................ 27 FIGURE 11     PAN GESTURE BY USING 4-FINGER HAND POSTURE WITH MOVEMENT TO THE LEFT, RIGHT, PALM OUT OF PAGE AND PALM INTO THE PAGE. ................................................................... 27 FIGURE 12     SWIPE GESTURE: SWIPE WHOLE PALM LEFT OR RIGHT IN THE SWIPE ZONE. ......................... 28 FIGURE 13     SHOWS THE BEFORE (LEFT) AND AFTER (RIGHT) OF GESTURE DETECTION IN VARIOUS ZONES. ................................................................................................................................................. 30 FIGURE 14     WEASIS DICOM VIEWER WITH LEWP PLUGIN ENABLED ON THE RIGHT AND SAMPLE PATIENT IMAGING LOADED. ................................................................................................................... 30 FIGURE 15     LEWP PLUGIN DOCKABLE TOOLBAR. ....................................................................................... 31 FIGURE 16     EXAMPLE LEWP SETUP FOR OUR PARTICIPANT USER STUDY. ................................................. 32 FIGURE 17     SHOWS HOW IMPORTANT PATIENT'S IMAGINGS ARE DURING A PROCEDURE. ..................... 34 FIGURE 18     SHOWS HOW FREQUENT THEY USE IMAGINGS DURING A PROCEDURE................................. 34 FIGURE 19     SHOWS IF PARTICIPANTS COULD MANIPULATE IMAGINGS THROUGH A STERILE FASHION WOULD THEY BE MORE INCLINED TO VIEW THE IMAGINGS. .................................................. 34 FIGURE 20     VARIOUS BAR CHARTS THAT SHOWS THE USABILITY OF EACH GESTURE FUNCTION: (A) OVERALL SATISFACTION OF USING THE LEWP SYSTEM, (B) USABILITY OF THE ZOOM FUNCTION, (C) USABILITY OF THE SCROLL FUNCTION, (D) USABILITY OF THE WW/WL FUNCTION, (E) USABILITY OF THE PRESET SWIPE GESTURE, AND (F) USABILITY OF THE PAN FUNCTION. ............................................................................................................................... 35 FIGURE 21     A BOXPLOT SHOWING THE SPREAD OF USER EXPERIENCE ON EACH OF THE MAIN GESTURES. ................................................................................................................................................. 36 FIGURE 22     SHOWS THE RESULT FOR THE QUESTION THAT ASKS IF THE SYSTEM HAS ALL THE IMAGE MANIPULATION TOOLS NEEDED DURING A PROCEDURE. ...................................................... 36 FIGURE 23     SHOWS THE RESULT OF USABILITY AND LEARNABILITY OF THE SYSTEM. ............................... 36  ix  LIST OF ABBREVIATIONS UBC  University of British Columbia PACS  Picture Archiving and Communication System DICOM          Digital Imaging and Communications in Medicine ROI Region of Interest SSI  Surgical Site Infection LeWP  Leap Weasis Plugin CT  Computed Tomography US  Ultrasound MR  Magnetic Resonance PET  Positron Emission Tomography FSR  Force Sensitive Resistor ANF  AMPEL Nanofabrication Facility ITO  Indium Tin Oxide HRS  Horizon Radiology Station LED  Light Emitting Diodes IR  Infrared NUI  Natural User Interface   x  ACKNOWLEDGMENTS I would like to acknowledge some very important individuals that I could not have completed my Masters without. Firstly, I would like to acknowledge my supervisor, Dr. Philippe Kruchten, for his continuous support and guidance throughout my research. Working and learning from him has been an immense pleasure.  Secondly, I would like to thank Cliff Edwards, my supervisor at Mckesson Corp. for providing me with ideas and programming tips in addition to his support throughout my research and second reading of my thesis.  Thirdly, I would like to thank Dr. Darra Murphy, MD and the doctors who partook in my user studies for your help in both testing out my project and offering your feedback.  In addition, I am grateful to SurfNet (NSERC) for funding this research throughout the program.  Last but not least, I would like to thank my family, girlfriend, and friends. The support you guys have provided me was invaluable.    1  CHAPTER 1 INTRODUCTION Medical treatments have advanced rapidly throughout the history of mankind. Every year new treatment methods are developed to replace old inefficient procedures. One such set of techniques long under development is called surgery. Surgeries involve treatments of many illnesses including cancer, physical bone fractures, and many more. However, to complete a surgical operation, the treatment involves breaking the human being’s natural infection barrier, the skin. Therefore, any surgery could potentially lead to postoperative infection. Many scientist and physicians [1] [2] [3] have conducted research on how surgical site infection (SSIs) or post-surgery infections can affect the healing of a wound or even the condition of the patient’s health.  To prevent these infections, modern procedures include conducting surgeries in sterile surgical sites and following aseptic procedures such as hand scrub techniques [4] (techniques to properly sterilize one’s hands) and wearing scrub outfits. However, despite the existence of these new prevention methods, there is still a 1 to 10 percent chance [3] that infection may occur even when the majority of the equipment in the operating room is properly sterilized. This is because when physicians need to navigate patient imagings, the physical contact with the mouse or keyboard can increase chances of bacterial transfer.  There are a large number of patient imaging systems that exist today to assist physicians in their work. Patient’s radio-imaging data including MRI, CT, X-ray, and etc. are often used as maps and guidelines for doctors to perform surgeries and other advanced procedures on a patient’s body. These imagings are often presented on a Picture Archiving and Communication System (PACS) workstation. In the operating room, PACS is either controlled by an assistant or the primary doctor. Several problems arise from these situations that bring us back to the infection risks aforementioned: 2   If controlled by an assistant, in most cases, it is hard for the primary reader of the information to obtain the precise location of the object of interest in a quick and effective manner. The physician would often have to spend more time directing the assistant rather than focusing entirely on the patient.   If controlled by the primary user themselves, they are required to repeat the process of re-sterilizing before continuing the procedure.  Both of these problems lead to excessive overhead in time used [5]. As touchless interaction in sterile environment have become an active field of research, we as researchers also need to look at various aspects such as learnability, usability, and portability of such systems to get these technologies to integrate seamlessly into the operating room [6]. 1.1. Contribution In this research, we propose a hand gesture recognition system called Leap Weasis-Plugin (LeWP). These gestures will be captured by using Leap Motion®  sensor and interpreted to perform some common image interpretation commands such as zoom, pan, and scroll.  The result of this research will remove the problem of having to re-sterilize since it enables the surgeon to perform gestures in the air without contacting any surface. It is also apparent that this is an active research area and there are multiple projects around the world trying to answer the similar question. However, two commonalities we have observed from these proposed methods of using two-handed gesture systems are that the systems lack fluidity and a natural interaction speed. Gestures need to be performed slowly in order for the system software to recognize the corresponding gestures and map effectively to the specific actions. Our goal is to assist physicians in making procedures flow more efficiently by answering the following research questions:  Is there a way to manipulate images without having to rescrub during a procedure? 3   Is it possible to define a method that completes all tasks required to manipulate the imagings with a low number of additional tools?  Is there a gesture recognition system that will be intuitive and easy to use? 1.2. Thesis Overview This thesis is organized in 7 chapters. Chapter 2 gives further background into the problems identified in chapter 1 and chapter 3 describes the related work performed in other research labs. Chapter 4 outlines the work that was done prior to the final prototype including the different systems and prototypes that have been attempted. Chapter 5 explains the final prototype that was produced and how each component functioned. Chapter 6 presents experiments conducted and includes all the data and analysis of the system. Chapter 7 offers suggestions for future work and a conclusion.   4  CHAPTER 2  BACKGROUND 2.1. Picture Archiving and Communication System (PACS) A picture archiving and communication system (PACS) is a comprehensive medical imaging software system that allows physicians to store, diagnose, and annotate information onto electronic images such as computed tomography (CT), ultrasound (US), magnetic resonance (MR), positron emission tomography (PET) images and many more. From here on, we refer to imaging as sets of patients’ US, MR, CT, etc. data. Most PACS have standardized features much like any photo gallery viewer presented on a layman user’s personal computers. Physicians can use any PACS workstation to access the patient’s data file anywhere from the hospital’s network. They can perform features such as picture magnification, window width (contrast), window level (brightness), and point-to-point measurements (annotations) on the images.  PACS workstations can be found in physicians’ offices or dedicated radiology reading rooms for them to complete their diagnosis, or they can be found on carts or mounted on the walls inside the operating room (OR).  Figure 1 WEASIS DICOM Viewer [32]. 5  In this research, two different systems have been used. The first was McKesson Corporation’s proprietary Horizon Rad Station (HRS) and the second was an open-source viewer called WEASIS developed by Dcm4che. For prototype purposes, WEASIS was chosen for our integration of the tool because of its open-source and simplicity as shown in Figure 1 on the previous page.  2.2. Sterile Environment Constraints and Problems As medical procedures improve each year, the processes associated with each individual procedure have also evolved.  It was not until the mid-19th century that surgeons began to create sterile, aseptic environments in operating rooms under the influence of a British surgeon named Joseph Lister [7]. Research by physicians [8] over the years has shown that post-operative infection may cause various complications such as slower healing, further antibiotic treatments and pain. Over the past few decades, standard procedure and regulations have made it mandatory for primary personnel in the operating room to be scrubbed and be wearing aseptic clothing. Furthermore, hospitals have created sterile zones to ensure the whole operating procedure is aseptic. This raises many challenges regarding the efficiency and usability of equipment around the physicians during a procedure.  One such equipment is the PACS workstation.  Throughout the procedure, physicians often require review of patient images several times to accurately make decisions in the process of treating the patient. These imagings provide them a map to obstructed areas and provide guidelines and measurements such as needle length required for the specific operation. The current option to view these imagings from the PACS workstations in the sterile environment still requires physicians to use the traditional mouse and keyboard setup and re-scrub when they are finished or they must dictate instructions to the lab assistant or an unscrubbed nurse. These two methods raise many problems that can hinder the speed and efficiency of the procedure and may lead to a slower recovery for the patient if the procedure has been extended for a long period of time.  6  In the first scenario where the physicians want to retain control of the PACS workstation, they would be required to take their gloves off and use the mouse and the keyboard to adjust the imagings. Once they complete the tasks, they are required to re-scrub before they can enter the sterile zone in the operating room. The time it takes for scrubbing typically stretches from two to five minutes depending on specific policies by the hospital. However, if the physicians require seeing these imagings a few times throughout the procedure, then significant downtime would be incurred. To bypass the rescrubbing procedure, physicians have tried many shortcuts; for example, O’Hara et al. [9] reported that the doctor uses the non-sterile side of their gown to cover the mouse to study and manipulate the imagings. This allows them to quickly see the imagings and get back to the procedure. However, there are still many concerns that this method may cause bacterial transfers.  Relying on an unscrubbed nurse or assistants to help with manipulating the imagings may also have its own problems. It is often very difficult to communicate effectively to one another for instructions. Graetzel et al. [5] has shown in one study that it took 7 minutes for a nurse to complete the tasks with physician’s verbal instruction. The difficulty with completing these tasks efficiently comes from the difference in the technical knowledge on the subject between the physician and the support staff. This presents the problem where the support staff unintentionally navigate to the wrong region of interest (ROI) specified by the physician due to unfamiliarity. In addition, each person perceives and interprets information differently. For example, a given instruction to pan the image left shows a lot of ambiguity as to how far left the user will have to pan the image. Although this may seem like a minor problem, with accumulated repetitive errors during a single procedure this may incur more time than needed to finish the procedure. Also, after many incorrect manipulations the primary physician may decide to just unscrub and control the PACS system themselves. As we can see, giving instructions and verifying these instructions can produce a high rate of errors. Each year, new technologies are developed to help surgical operating rooms to function more efficiently. However, when new technologies and new devices are created, it means more 7  space in the operating room is required for these devices. With such a cluttered space, the proximity to the required tool is another problem to overcome. Greenberg et al [10] illustrated the behaviour of how people and devices interact. Also, Mentis et al [11] studied how the PACS station is located on the wall and when the physicians view the imagings their back will be facing the patient. The location of the PACS station causes inefficient communication back and forth between the surgeon at the PACS station and the surgeons who are around the patient. In addition, new hand-gesture based research require clear line of sight to a sensor such as a camera. This is not always available or possible when physicians and nurses surround the patient. In one scenario [12], we can see that when a doctor on the right needs to see the images, the doctor on the left side of the patient needs to move away so the sensor can detect the user’s gesture.   8  CHAPTER 3 RELATED WORK The area of touchless interaction has been increasing popular in the last four years after the release of Microsoft’s Xbox Kinect sensor and followed by the Leap Motion® . In this section, we will discuss the relevant work and research that have been done in this field. 3.1. Vision-Based Gesture Systems Prior to the release of Wii and Kinect, there have been decades of research in the field of computer vision. Computer vision encompasses algorithms for processing and analyzing images or videos from the analog to digital data. It is intended to duplicate the human vision ability to perceive and understand images by computers. Computer vision techniques were primarily implemented with color RGB cameras as the input source. To detect hands or finger, systems usually rely on heavy pre-processing of the captured image frame. This pre-processing stage usually involves a combination of segmentation, background subtraction, and filtering techniques. The next step is to perform object detection through blob detections and contouring algorithms to find the hand outline, for example. When the outline has been found, special algorithms can find the number of fingers raised. For example, Graetzel et al [5] and Wachs et al [13][14] developed systems that use a color stereo camera to perform image pre-processing, hand detection, hand tracking and gesture classification.  However, even as algorithms improve, vision-based techniques still suffer from lighting issues which leads to the lengthy process of re-calibration at every system start up and change in environmental setting. Under different lighting, a colour stereo camera cannot distinguish a particular area accurately when the RGB hue and saturation is different. In addition, without the help of any other sensors, segmentation of an object from two-dimensional images that captures a three-dimensional space is complicated and error-prone. 9  A group of researchers at University of Luebeck, developed the WagO [15] system that uses two stereo cameras to triangulate hand positions in the 3-dimensional space. Although this system removes some lighting constraints, the system takes up a lot of room and supports too few gestures. 3.2. Speech-Based Systems Another popular choice for touchless interaction is through speech recognition or in combination with a vision-based system. By speaking directions into a headpiece, one can also manipulate images in increments. For example, each zoom command will magnify with a preset zoom scale. Speech recognition is a very simple concept; however, implementations still present many problems before it can be efficiently used for PACS navigation. Machine learning has been applied to improve speech recognition, but due to different accents, tone, speed and other factors, complete accurate recognition is still impossible.  Furthermore, operating rooms are filled with unwanted noise such as discussions between members of the surgical team, and sounds of fans and pumps from medical equipment.  3.3. Kinect-Based Gesture Systems In 2010, Microsoft launched the controller-free Xbox Kinect to the public. What seemed like a gaming peripheral quickly found its way into a big research community in the field of natural user interface (NUI). The ability of a sensor to process and track the body of the players in games have allowed many researchers to re-engineer this $149 device and apply this new technology into other products and services to make daily work flow better. A touchless interaction system will especially benefit environments where sterility is an important constraint. As with any other camera-based object detection system, external light source and depth perception have always been the source of problems for accurate object detections. To solve 10  these problems, Microsoft’s Kinect sensor incorporated a few additional hardware and software technologies. The sensor couples infrared light emitters and infrared camera sensor with a RGB camera. This allows the sensor to create a 3D depth map of the surrounding area for further image processing regardless of lighting conditions. Along with the depth sensors, Kinect includes a microphone array to isolate and pinpoint the location of the voices of the players. The Kinect needs a capable software implementation to make great use of the data it collects. The sensor has been trained many times using machine-learning algorithms [16] to detect user movements and user’s body parts. After the processing, the software can locate 48 skeletal points of the user’s body including the hand and the foot. This allows the Kinect to predict and detect user’s gestures and their partial body even if they are obstructed by furniture. Once the locations of the hands are detected, the system can now use specialized algorithms to track and classify simple hand gestures and map gestures to commands. Another way to make use of the Kinect’s skeletal tracking was to create invisible interaction boxes around the user. Reality Controls Inc. [17] developed a mapping system of interaction boxes to trigger keyboard and mouse simulators. By moving hands into the zones, different preset commands can be activated.  A wide variety of Kinect-based projects have been developed in parallel during our development. The first of such project’s published was developed by a group of surgeons and engineers at Sunnybrook Hospital, Toronto [18]. There are also various improvements to and new gesture recognition systems such as [19][20][21][22][23][24][25] that use the Kinect. As these systems have shown, Kinect has allowed great improvement for hand gesture recognition and usability for PACS navigation in sterile environments. These systems have a combination of one-handed or two-handed gestures, and speech recognition. We see that to perform all the necessary gestures, two hands will be required.  11  With new technologies being developed, new limitations are also created. Sometimes these limitations discourage the physicians from adopting the new system. For example, from the systems aforementioned, when both of the physician’s hands are busy during a procedure, he or she cannot take control of the gesture recognition system that requires two-handed gestures. In this case, the problem of having the assistant to manipulate the imagings returns. We also identified another problem during a video demo of [12]. We see that another doctor needs to dodge away so that the Kinect sensor can be used by another doctor standing across the surgical table. As mentioned in the last section, proxemics has become really important on how natural user interface should be built, placed, and used. 3.4. Leap-Based Gesture Systems Approximately two years after the release of Kinect, a new device, called Leap Motion® , has been introduced and released to the public. It is a small $79 USB peripheral device that outperforms Microsoft’s Kinect sensor for hand and finger tracking. As opposed to the Kinect’s whole-body tracking, the Leap differentiates itself by tracking only the user’s hands and fingers. This difference in tracking allows finer granularity in the resulting control than the Kinect and enables the device to “see” each individual finger. In addition, this technology requires less surface area of the user’s body parts than the Kinect. This allows the device to be placed in closer proximity to the doctor and reduces the requirement of line-of-sight and chance of being blocked. The authors of [26] have also developed a Leap Motion®  plugin as the basis of their ClearCanvas PACS gesture recognition system. To activate each command, the user will have to perform a swipe gesture to select the correct command from a menu. After that, the user will move a single finger horizontally and vertically to adjust the image. Although this is a very simple system, the ability to have only one gesture activated at one particular time may be frustrating to power users who would like to quickly perform multiple repetitive adjustments (eg. zoom, scroll, zoom, pan).  12  3.5. Other Gesture Systems It is very interesting to note that there are also various non-traditional gesture recognition systems. Tani et al [27] created a wearable gesture interface glove to control the PACS system. With sensors attached to the glove, gestures can be detected in real-time and mapped to different commands. However, this design cannot be used in sterile settings as the glove cannot be easily sterilized. Another interesting technology is called Omni Touch. [29]. They combined a Primesense sensor (similar to a Kinect) and a picoprojector strapped onto the shoulder of the user. The projector projects interaction zones onto any surface. Then, the user, with the help of detection from the Primesense sensor, can interact freely with that surface. The system transforms any surface into touch surfaces. But for this reason, the physical interaction does not benefit the surgeons who need to remain sterile.  On a very different approach, the creator of WiSee [28] uses sensors to detect the change of WiFi signals generated from cellphones, laptops, and routers in a room. The recorded change can then be mapped to an existing database of signal changes to translate into user commands. This may be an excellent idea where no additional device will hinder the doctor; however, in an operating room, many of the equipment may affect wireless signals in real-time causing the need to frequently calibrate the room. Similar to the WiSee, another group of researchers from University of Washington [30] built a cellphone attachment that uses existing surrounding wireless signals to recognize gestures. They captured signals such as TV transmissions and recognize the changes in signal amplitudes to distinguish the different gestures performed near the device. Again, like the situation with WiSee, radio signals can be affected by the surrounding medical equipment.  13  Similar to the Leap Motion® , a device called the CamBoard pico S [31] can recognize gestures based on time-of-flight depth sensing algorithms. Unlike the Leap Motion®  limited to level palm detections, the Pico camera uses a CMOS sensor to map the surrounding area to create a 3-dimensional map of the whole hand regardless of orientation. Although this device is slightly more expensive than the Leap Motion® , it can be a viable alternative to support more complicated 3D manipulations. This device can be considered for future development.  14  CHAPTER 4 RESEARCH ROADMAP In this section, the process of research and experimentation into different devices and methodologies eventually set aside are documented and explained. The two main devices we experimented with were the Force Sensitive Resistors and Microsoft Kinect. 4.1. Force Sensitive Resistors In early stages of our preliminary research, we have looked into many interactive devices and different ways of interfacing with McKesson’s Horizontal Rad Station (HRS) system. One such device was the force sensitive resistor (FSR). The device was introduced as an alternative to a capacitive touch screen with the additional feature of pressure sensitivity.  With the new dimension of pressure sensing, new methods of gesture inputs can be added to allow a three-dimensional interacting space rather than the traditional two-dimensional plane on a touch screen or touch pad. The key problem to answer was to find a way to add depth manipulation with the help of FSRs to current digital display screens. Our initial approach to this device was to offer an alternative device that can be easily made sterile. Since the surface of the FSR matrix sensor is plastic, it can be effectively cleaned in comparison to the standard mouse or keyboard. Hence, we decided to conduct further research into this device.  The FSRs that we were introduced to were developed by an US company based in Bellingham, Washington called Sensitronics. These traditional FSRs are printed with layers of patented resistive polymer and combined with electrical circuitry to form the FSR device. Essentially the FSR device acts like a physically-flat variable resistor. When the user applies pressure to the device the resistance between the electrodes decreases. The resistance is effectively infinite when no pressure is applied. During our collaboration period, a new type of FSR was being developed by Sensitronics. This new type of FSR is translucent as opposed to the traditional non-transparent FSRs. Due to the work in progress by the company we were unable to attain a working prototype. Thus, this resulted in manufacturing our own version of clear FSR with the help from the company. 15  4.1.1. Creating a Firmware for FSR Matrix Before working on the clear FSR, we purchased a ready-made normal FSR matrix from Sensitronics for our experimental procedure. This matrix has a resolution of 10 by 16 pixels. To drive the FSR circuit, we couple the circuit with an Arduino Uno board and multiplexing circuitry to select our read inputs as seen in Figure 2 below. We created a simple algorithm of reading voltage data from each pin, with values ranging from 0 to 1024 representing each line of intersection in the matrix and serially sent the data to C++ analysis software. The C++ software parsed and processed the data and computed touch points. Once the touch points were identified, the C++ software also served as a bridge to connect the $1 Gesture Recognition library [32]. The first version of the un-optimized firmware can recognize simple gestures with slow touch motion. There was no particular reason to choose this gesture recognition library – we simply wanted to demonstrate the feasibility of the device to recognize touch-sensing gestures. Figure 2 Our FSR device with Arduino®  circuit to read raw values. 16  4.1.2. Manufacturing the Clear FSR The clear FSR involves a very different technique of manufacturing than the traditional printed FSRs. Although the device still requires printing of the clear resistive polymer, the other layers of the electrical circuitry involve a costly fabrication of nano-width electrodes. To allow the transparent ability, both the electrodes and the resistive polymer must be as transparent as possible. As stated by our contacts at Sensitronics, indium tin oxide (ITO) is required to ensure conductivity and transparency of the FSR device. However, due to the size and material constraints of these electrodes, special nanofabrication methods are needed. The physical state of ITO and the size of the electrodes prohibit the inkjet printing method of these conductors. Alternatively, two methods were investigated: sputter deposition and electronic-beam (E-beam) disposition. Both fabrication methods are available at UBC’s AMPEL Nanofabrication Facility (ANF) and the problems and reasoning can be seen on Table 1 below. Sputter Deposition Problem Reason 1. High temperature  2. Small target size   3. Very long manufacturing process 1. Very few substrates can withstand the heat, alternatives costly 2. Yield less than 2 inch by 2 inch samples; too small for a pressure sensing touchpad 3. Since we have not receive fabrication trainings, getting the ANF engineers to manufacture the device is very costly E-beam Deposition Problem Reason 1. High temperature to guarantee a better result 2. The distance for the electron beam to go from target to source is large 1. Very few substrates can withstand the heat, alternative costly 2. Cannot yield good quality conducting electrodes Inkjet Printing Problem Reason 1. ITO is in solid state at room temp. 2. Requires calibration based on the materials printing 1. Requires a liquid solution 2. Takes greater than 6 months to get printer setup and produce a result that will work Table 1 shows the problems and reasoning for the different methods of FSR fabrication. 17  The sputter deposition method uses a physical vapour deposition cured to a temperature greater than 400 Centigrade to deposit target material onto the substrate. The E-beam deposition method is also a form of physical vapour deposition which under high vacuum, target material is vaporized with an electron beam and precipitated onto the substrate. Both of these methods require masking and etching of the pattern required. Comparably, the sputter deposition method will yield a better result due to its high temperature process. However, after much research and collaborating with ANF engineers, it was realized that both methods induce problems that lead to looking for other solutions.  4.1.3. Alternative Solution Due to the problems summarized in Table 1, manufacturing the clear FSR prototype poses great challenges for completion during the Master’s program timeline and within the budget allotted. As a result, the continuation research in this path was abandoned in search of a better alternative interfacing with McKesson’s HRS portal. 4.2. Microsoft Xbox Kinect As introduced in section 3, the release of Microsoft Kinect sensor gave us the idea of building a touchless interaction technology.  4.2.1. Gesture and Speech Recognition System With the ability to detect users, we decided to approach our investigation further by building a prototype of a gesture and speech recognition system for the operating room. In this section, we detail our implementation of the two versions of the gesture recognition systems using Microsoft’s Kinect sensor. 18  4.2.1.1. Version 1 – Two-handed Gesture Prototype In version 1.0, we built a prototype that allows normal hand gestures. Much like a capacitive smartphone, you can perform two-hand pinch and zoom, one hand cursor movement, and two hand image scroll.  To activate the gestures, there is an invisible bounding three-dimensional interaction box “floating” around the centre of the user. To zoom, the user places both hands in the center of the interaction space and moves the hands in opposite directions. To scroll, the user raises one hand and the other can move into or away from the screen to scroll through image slices. If only one hand is detected in the box, the mouse gesture will be activated. When the hands are within this box, the gestures that the user performs are detected and translated into different commands. This version of the prototype was very unreliable and required a lot of calibration to allow the software to recognize a gesture command. In addition, the number of gestures was limited to three. At this point in time, we were using the McKesson Horizon Radiology PACS workstation as our image viewer. However, we were constrained to perform image data manipulations through simulated keyboard and mouse events. The simulated events caused many reliability issues that could not be improved with software tweaks. 4.2.1.2. Version 2 – Single-handed Gesture Prototype In version 2.0, as the Kinect SDK was updated, we added the capability of doing single-hand gestures illustrated in Figure 3 on the next page. The move to single-hand gestures was built in mind to allow doctors a greater flexibility in the operating room. With one hand freed, the doctors could now perform more tasks. The single-hand gestures involved using the new palm open and closed feature from the Kinect SDK update. With this feature, our software can now distinguish the activation and end of a gesture from anywhere. When the user closes his or her fist, the gesture is activated and deactivated if the palm is open. To help with the single-hand gestures, we also built a voice-activated pop-up menu to allow the user to switch between the actions: mouse cursor, zoom, scroll, brightness, contrast, and pan. This version of the software incrementally improved the reliability of the system. In addition, the software required pre-learning of the system since all actions are mapped to the same gesture. 19   4.2.2. Radiologist Interview and Change in Sensor Furthermore, after interviewing and performing a trial-run with a radiologist from St. Paul’s Hospital, we realized that the requirement of switching between the gesture functions through a menu drastically lowered the convenience and efficiency of the interaction. If gestures need to be switched back and forth for repetitive quick observations, it may become more time-consuming and decrease the interest of using such system. The problem was to find a way for the gestures to be done with a single hand and without a menu.    Figure 3 Our Kinect Gesture and Speech Recognition system setup. 20  CHAPTER 5 DESIGN AND IMPLEMENTATIONS To solve the problems we described at the end of Chapter 4, we needed a new way to support single hand gesture recognition without relying on a menu system that slows the user down. In this section, the final version of our gesture recognition system was developed with the help of the Leap Motion®  device and the inner implementations will be described in detail. 5.1. Leap Motion In this section, the inner workings of the Leap Motion®  device will be explained. 5.1.1.The Physical Hardware Device The Leap device itself embeds three infrared light emitting diodes (LED) and two monochromatic infrared (IR) cameras as seen in the Figure 4 below. By outputting infrared light upward in a hemispherical shape, the device can use the two IR cameras to capture the reflected infrared lights. The device connects and sends data to a host workstation through a USB cable for processing. The Leap controller firmware uses proprietary algorithms to process the three dimensional raw data and create a capture image of the air space above the device. Figure 4 Leap Motion®  device. 21  5.1.2. The Software  The Leap Motion®  also comes with software support for software developers. Leap is available on all three major operating system platforms including Windows, Macintosh, and Linux. Along with the primary controller software, Leap has included additional software development kits (SDKs) to allow software developers to access various processed data and integrate special gesture and motion recognitions into their own applications.  5.2. Leap WEASIS-Plugin (LeWP) In this section, the implementation detail of our gesture recognition system will be discussed in detail. Additional installation details can be found in Appendix A.4. 5.2.1. System Architecture In Figure 5, the flowchart illustrates a cycle of the software implementation. When the program is started, the user interface and default threshold values are programmatically loaded. The program then initializes and connects to the Leap. Once everything has been initialized, Leap will start obtaining frames captured by the device at approximately 20 to 40 frames per second depending on system load. In each frame, the program will check if any hands and fingers are available, if not the main loop will end and next frame will be checked. If the hand is available however, the system will continue to check how many fingers the user has put up and depending on the number of fingers, the gestures will be activated. The gestures and its detection algorithms will be explained in detail in the next section.  22    Figure 5 Flowchart diagram of LeWP Gesture Recognition System. 23  5.2.2. Gesture Recognition Functions and its Algorithms In this section, each individual gesture function and its implementation details will be discussed. 5.2.2.1. Activation Zone To interact with the Leap and allow the software to know that a gesture is activated, activation zones are created. The interaction space of LeWP is divided into three zones: gesture activation zone, hover zone, and swipe zone. These zones are indicated in the Figure 6. Figure 6 A top-down view of the LeWP zones for gesture detection. 24  The gesture activation zone is located just one centimeter behind the center of device. When the user’s hand enters this zone, the program will signal each individual gesture to activate their functionality. One can imagine that this green zone is similar to a laptop’s trackpad but oriented vertically in front of the user.  When the user enters this zone, a green banner will be shown to represent gesture activation. The hover zone is located in between the activation and the swipe zone indicated by the red area in the previous figure. In this zone, no gestures except for the cursor will be recognized. This is to allow the user to move the cursor without accidently triggering other gestures. The swipe zone as indicated above is the area where the default swipe gesture included in the Leap SDK will be detected. 5.2.2.2. Cursor Navigation To use the cursor function, the user needs to raise one finger and point it towards the screen as seen in Figure 7. In the hover zone, one finger will allow the user to navigate and move the cursor around the screen. The coordinates of cursor movements have smoothing functions applied and have been normalized by the Leap SDK. Once the normalized coordinates have been obtained, a scaling will be applied to the coordinates to match the primary screen width and height. To activate the cursor click action, the user needs to move the one finger directly into the green activation. To unclick, the user will just remove the finger from the activation zone. 25   5.2.2.3. Two-Finger Image Scrolling Imaging scrolling is one of the most important tools required by doctors. To activate the gesture, the user needs to raise two fingers up and enter into the green activation zone demonstrated in Figure 9. Once in the activation zone, the user can move the hand vertically up and down to scroll through the image. The LeWP calculates the average tip positions of the two fingers and determine if that average has moved vertically up and down in the physical space and then sends the scrolling command to the WEASIS viewer. The scrolling of the images is mapped using one centimeter of movement to one image slice change. Because of the fine movements, the user will most likely run out of space to move vertically. For example if the user would like to scroll down and the hand has reached relatively close to the surface, the user needs to repetitively move the fingers out of the activation zone, and enter the activation zone at the top to continue scrolling down. It is very similar to the trackpad where the user runs out of scrolling space and needs to lift the fingers up and start scrolling from the top. Default scroll increments can be programmatically set for user preferences.  Figure 7 Mouse cursor navigation gesture with 1-finger hand posture. 26  5.2.2.4. Two-Finger Image Zoom In LeWP, image magnification is also mapped to the two-finger orientation. To activate the zoom function, the two fingers must move in a quick pinching movement seen in Figure 8. Each quick pinch zoom magnifies the imaging by a preset increment. When the two fingers have entered the activation zone, the system will calculate the Euclidean distance of the two finger tips for previous and current frame. If there is a difference between the two different frames, the zoom function will be activated; otherwise, the scroll function will be checked instead. To zoom in, the distance of the two finger tips must be increasing, and to zoom out, the distance between the two points must be decreasing. To adjust for the need of continuously moving in and out of the activation zone to zoom, the user can stay in the activation zone and magnify the image in one direction. To achieve this, the user needs to perform either a quick pinch or quick finger expand. This way, only the fast two-finger movement will be recognized as zoom functions.  Figure 9 Scroll gesture by moving 2-finger hand posture up and down. Figure 8 Zoom function by using fast finger pinching and expanding movements. 27  5.2.2.5. Three-Finger Window Width (WW) and Window Level (WL) To change the window width (brightness) and window level (contrast), the user needs to use three fingers. In the activation zone, moving the three fingers horizontally will change the window width and moving vertically will change the window level (refer Figure 11). Figure 11 Window width (WW) and window level (WL) adjustments with 3-finger hand posture. The WW adjustments are into the page (hand palm vertically downward) and out of the page (hand palm vertically upward). Figure 10 Pan gesture by using 4-finger hand posture with movement to the left, right, palm out of page and palm into the page. (Out of the page) (Into the page) (Out of the page) (Into the page) 28  5.2.2.6. Four-Finger Image Pan Sometimes after zooming, the user will need to move the images around to find a region of interest. To do so, the user will require four fingers in the activation zone. When the fingers move horizontally, the images will pan left or right and moving them vertically, the image will pan up and down. The hand gesture is illustrated in Figure 10 on the previous page.   5.2.2.7. Window Width and Window Level Presets In many situations, based on the imaging (CT, MRI, X-Ray), the doctor needs different window width and window level to view the proper region of interest (ROI) such as the bone or abdomen. The settings of this ROI usually differ by magnitude of a few hundreds so that three-finger WW and WL change will not be efficient. This is why generic presets of WW and WL values have been mapped to the default Leap Swipe Gesture (refer Figure 12). The presets include the WW and WL for bone, brain, abdomen, and lung. When the swipe is detected in the swipe zone, it would set the imaging to the next or the previous preset Figure 12 Swipe gesture: swipe whole palm left or right in the swipe zone. 29  depending on the direction of the swipe. After the presets have been applied, the doctors can continue to use the three fingers to fine tune WW and WL of the image. 5.2.2.8. Next and Previous Image Scroll The two-finger scrolling offers a faster power scroll so that the user can quickly scroll through the image data set. However, sometimes, the user needs to examine only a few slices of images in a large stack and two-finger scrolling will not allow the fine scrolling the user needs. Therefore, a toggle button is built so that the user can toggle between using the swipe gesture for WW and WL preset or next and previous image. If the toggle for the next and previous image is on, the user can swipe to next image slice or previous image slice for a fine control. 5.2.3. User Interface The user of the LeWP is presented on a dockable sidebar into the Weasis graphical user interface (GUI). The dockable sidebar includes three large buttons, a preset combo box, and a visualizer as shown in Figure 14 below. Figure 15 gives the detailed breakdown of the dockable sidebar. The three large buttons include the WW and WL Preset, Next / Prev Image, and Reset. The first two toggles enable the swipe gestures as indicated previously. Only one of the two can be activated at any time. The reset button is also included in case the user feels the need to reset the image back to default values after complicated changes. The preset combo box is activated only when the WW and WL Preset is toggled. The combo box shows the current preset value and gives the user a visual cue. 30  The visualizer at the bottom is divided into two parts. The first part of the panel offers the top-down, birds-eye view of the interaction space. When the LeWP is activated, the panel will show where the finger-tip points are located, as well as indicating each of the zones by colour coded lines. The black line gives the user an idea of where the center is. The second part of the visualizer is the small banner. By default, the banner will be red indicating the hands are either non-detected, in swipe zone or in hover zone. When any finger crosses the activation zone, the banner will turn green indicating the activation of the gestures. Furthermore, the banner will display text to the user on what gestures are currently activated. Figure 13 shows the before and after of activating a gesture.  Figure 14 Weasis DICOM viewer with LeWP Plugin enabled on the right and sample patient imaging loaded. Figure 13 shows the before (left) and after (right) of gesture detection in various zones.  31    Swipe Gesture Activation Buttons Figure 15 LeWP Plugin dockable toolbar. WW/WL Preset Leap Zones & Hand Viewer Activation Indicator 32  CHAPTER 6 VALIDATION AND RESULTS In this section, the result of this research, the objectives achieved, the difficulties and problems faced will be described in detail. 6.1. Experiment and Validation Method To validate our Leap Weasis-Plugin prototype, we conducted user study sessions on the LeWP system. The system is intended to help doctors and operating room technicians manipulate imagings of the patient in the sterile environment. Therefore, we require participants of our study to have backgrounds and knowledge in the medical field. Our primary participant target group consisted of radiologists, surgeons, and other technicians and assistants that support the doctors inside the sterile environment. Furthermore, our participants also include research engineers from McKesson Corporation who specialize in human computer interaction and provide research inputs in updating McKesson’s radiology PACS system. Figure 16 Example LeWP setup for our participant user study. 33  The experiment began by debriefing the participants on the purpose of this research and giving them a short demonstration of how to use the LeWP gesture recognition system with the setup as shown in Figure 16. Before the participants continue to experiment with the system, a few questions were asked to classify their area of expertise and for demographic collection. The participants then proceeded with the instructions on looking for a specific region of interest in anonymized sample imagings including MRI, CT, X-Ray, and 3D scans. For example, the participant may have been asked to find a specific bone area in a CT scan. After that, the participants began to simulate their experiment with the system. When the participants finished interacting with the system, they were given a questionnaire form that includes 5-point Likert Scale questions and open-ended discussions. These questions are used to find out the weaknesses and strengths of the LeWP gesture system compared to the traditional mouse and keyboard setup. The results will be discussed in greater detail in the next sub-section. The full consent form and questionnaire can be found in the Appendix A.1 and A.2. 6.2. Validation Results The user study ran for three consecutive days at St. Paul’s Hospital in downtown Vancouver. Each participant was given a questionnaire to fill out before and after the experimentation. Each session lasted from 15 to 45 minutes in length and a total of 16 participants experimented with the LeWP system. Our participants included 9 radiologists, 3 surgeons, and 4 participants that consist of surgical room assistants and a PACS research engineer. Of the 16 participants, 11 people have been in their field of expertise for at least 6 years but more than half of them did not have experience with any NUI sensors such as Kinect, Wii, or Leap Motion® .  34  From Figure 18 and Figure 17 above, we can see that 11 of the participants use patient’s prior imaging during procedures and that these participants think it is important that they are able to view and manipulate them. Furthermore, all of the participants would be more inclined to use the imagings during a procedure if they are allowed to do so in a sterile fashion as seen from Figure 19.  After a short demonstration and user experimentation, the Likert Scale questions were asked to evaluate several aspects of the system and the experience using the system. The overall satisfactions of the system as well as each of the gesture functions including zoom, scroll, window width and window level, preset swipe, and pan functions were rated by each participant and the results are shown in Figure 20 and Figure 23 on the next page. From the 0 0 0 4 12 StronglyDisagreeDisagree Neutral Agree StronglyAgreeNo. of People 2 1 2 0 11 Never Sometimes AlwaysNo. of people Usage Frequency Figure 17 shows how important patient's imagings are during a procedure. 0 1 1 5 9 Not Important Neutral Very ImportantNo. of People Importance Figure 18 shows how frequent they use imagings during a procedure. Figure 19 shows if participants could manipulate imagings through a sterile fashion would they be more inclined to view the imagings. 35  bar charts and the boxplots, we can see that the majority of the participants (spread) are satisfied with the system and find all the gestures are at least understandable and easy to use. The participants have been asked to rate their experience against the traditional mice and keyboard setup with the PACS system.  Figure 20 Various bar charts that shows the usability of each gesture function: (a) overall satisfaction of using the LeWP system, (b) usability of the zoom function, (c) usability of the scroll function, (d) usability of the WW/WL function, (e) usability of the preset swipe gesture, and (f) usability of the pan function. 36  Out of the 16 participants, 13 people agree that this system will reduce the time needed to perform a procedure by removing the rescrubbing steps and that the system has all the manipulation tools needed during a procedure as seen in Figure 21. In addition, Figure 22 illustrates that they all agree that the one-handed gestures are easy to understand and navigate. The full questionnaire results can be found in the appendix. 0 0 3 7 6 StronglyDisagreeDisagree Neutral Agree StronglyAgreeNo. of People Manipulation Tools 0 0 0 9 7 No. of People One-handed Gesture Navigation 1 Hard, counter intuitive, didn’t get it 2 Somewhat difficult 3 Neutral 4 Understandable, relatively intuitive 5 Easy, very intuitive Figure 23 A boxplot showing the spread of user experience on each of the main gestures. Figure 22 shows the result of usability and learnability of the system. Figure 21 shows the result for the question that asks if the system has all the image manipulation tools needed during a procedure. 37  6.3. Difficulties One difficulty found during the experimental section was the swipe gesture; participants found it hard to get a consistent WW/WL preset change. It was interesting to note that each participant’s definition of a swipe gesture differs from each other. While most participants were able to get the presets to change easily, there are a few participants that had a different interpretation to what swipe length and swipe speed would be. This misinterpretation of the gesture, despite the given demo, left some participants frustrated on this specific gesture. Another minor problem faced was the zoom function. It takes most of the participants a few trials with the quick pinch movement to complete the zoom function. As many users are used to slow fine pinch-and-zoom movements on their laptop trackpad or personal tablets, they could not get the zoom on their first try. 6.4. Discussion Most participants noted that they see this system reducing the time required during a procedure because they are able to focus on the images at hand and would not need to rescrub every time they manipulate the imagings. They preferred the use of only one-handed gestures so that in complicated procedures, they will still have the additional hand for various tasks. In our observations during the experimental sessions, many surgeons and radiologists explained that our system takes out the frustration of having to unscrub or asking someone to manipulate the images and that they were happy to see this finally implemented efficiently. Error! Reference source not found. shows the comparison of each interaction device that as been investigated during this research.   38   Many participants noted that since it was their first time using the gesture system, it would take a few more trials to completely master the program but regardless it was an easy-to-use and easy-to-understand system. The reason why they may need more practice of these hand gestures is because many of us are accustomed to our tablets and the sensitivity that you can control by physical contact. However, it is this precise reason that the participants find that the gestures are easy to remember since they are very similar to what they would use on a laptop trackpad.  In Section 1.1, we introduced our goals to answer the following research questions: R1: Is there a way to manipulate images without having to rescrub during a procedure? R2: Is it possible to define a method that completes all tasks required to manipulate the imagings with a low number of tools? R3: Is there a gesture recognition system that will be intuitive and easy to use? System FSR Microsoft Kinect Leap Motion Cost High, > $1000 Low, $149 Low, $79 Interaction Requirements Finger gestures with direct surface contact Hand gestures, direct line of sight of user body with camera Finger gestures, direct line of sight of fingers with sensor Sterility Plastic surface can be easily cleaned No surface, gestures performed in air No surface, gestures performed in air Usability Perform gestures like using laptop trackpad Repetitive gesture with menu activation system  Perform gestures like using laptop trackpad but in air Level of difficulty for user Easy Medium Easy Table 2 A comparison chart of devices explored in this research. 39  From the results in section 6.2, we illustrated that we have answered all three of our research questions. Physicians are able to manipulate efficiently in air to avoid contamination and the need to rescrub by coupling a Leap Motion®  sensor and an open-source PACS system. R1 is accepted by measuring the ease-of-use of each gesture compared to the traditional mouse and keyboard setup. We also used one-handed gestures that duplicate current finger gestures physicians would use on their laptop to provide seamless interaction and low learning-curve. From this, we also validated R2 and R3 of our research questions. In addition, during the period of our research, many similar systems were developed by various research groups which show that this is an important problem that needs to be solved in the medical field.  By making systems that may reduce procedure time while remaining sterile, physicians are able to decrease the risk of post-operative infections and allow patients to have quicker recovery.    40  CHAPTER 7 FUTURE WORK AND CONCLUSIONS The goal of the research is to develop a new interactive method to allow surgeons and radiologists to navigate and manipulate imagings on PACS workstations in the sterile environment. Not only does this method have to be more efficient in terms of getting work done, but it must also help the physicians reduce time needed in the operating room. The key problem to solve is the sterilized environment. Once we have determined that sterilization is the main problem, it became easier in identifying an approach to solve the remaining issues.  During this research, many natural user interface computing technologies were investigated to see if they can solve the problem. Firmware drivers and gesture recognition software were written for the FSR to create a desktop prototype. Microsoft Kinect was used in many ways to provide a better gesture recognition algorithm. Two versions of the Kinect Gesture Recognition System were built. However in the end, the prototype that solved all three of the goals was the plugin tool that uses the Leap Motion® . The result of this research provides the surgeons with alternative methods to interact with the PACS workstation. Not only does this decrease the time they need in a surgery, it also increases the efficiency for the primary user of this technology. Upon starting the software, the primary surgeon can easily take control of the system and obtain all the information they require. While in control, the user only needs to use one hand versus the traditional two-handed gestures. This allows the other hand to be used for other tasks. In addition, with the use of Leap Motion® , the hands do not need to be re-sterilized thus allowing the surgery to be done more efficiently. Another main contribution of this thesis is the lack of calibration needed. This means that anyone who wishes to take control of the software can easily do so without further time wasted on calibrating the system for the specific user.  We conducted several user study sessions to experiment with the LeWP system with professional users. From our gathered results, we can see that more than half of the users 41  agree that this system will benefit them positively and that they can focus more on the procedure at hand than wasting time on tedious tasks. All feedbacks were very positive and reinforced that further work can be done to improve the current system to provide physicians tools that they actually need.  7.1. Future Work Although we have evaluated our system with mock settings, reliability and usability can be further verified if clinical studies and procedures are done. As imaging tools become more advanced, we also need to factor in 3-dimensional image manipulations that may be needed in the near future. Further research can be done to see if it is possible to provide 6 degrees of freedom to gesture recognition systems. In addition, we can expand the functionality of the LeWP system by providing one-handed gesture measuring tools to give the physicians additional functionalities that may support the procedure. As for the hardware device, we hope to make data transmission wireless so that the device can be efficiently placed as preferred by the physicians.   42  BIBLIOGRAPHY [1] Torpy JM, Burke AE, Glass RM. Postoperative Infections. JAMA.2010; 303(24):2544. doi:10.1001/jama.303.24.2544. [2] The Johns Hopkins Univeristy. Surgical Site Infections. Hopkinsmedicine.org. Retrieved July 3, 2013, from http://www.hopkinsmedicine.org/healthlibrary/conditions/surgical_care/surgical_site_infections_134,144/ [3] Philip J. B. Davis, MD; Donald Spady, MD, MSc; Chris de Gara, MB, MS; Sara E. D. Forgie, MD. Practices and Attitudes of Surgeons Toward the Prevention of Surgical Site Infections: A provincial survey in Albert, Canada. Infection Control and Hospital Epidemiology, Vol. 29, No. 12 (December 2008), 1164-1166.  [4] Gardner, D., Anderson-Manz, E. (May 1, 2001). How to Perform Surgical Hand Scrubs. ICT: Infection Control Today. Retrieved November 14, 2013, from http://www.infectioncontroltoday.com/articles/2001/05/how-to-perform-surgical-hand-scrubs.aspx [5] Graetzel, C., Fong, T., Grange, S., & Baur, C. (2004). A non-contact mouse for surgeon-computer interaction. Technology and Health Care, 12(3), 245-257. [6] O'Hara, K., Gonzalez, G., Sellen, A., Penney, G., Varnavas, A., Mentis, H., & Carrell, T. (2014). Touchless interaction in surgery. Communications of the ACM, 57(1), 70-77. 43  [7] Markel, H. (July 26, 2013) How “Going Under the Knife” Became Much Less Deadly. PBS Newshour. Retrieved August 4, 2013, from http://www.pbs.org/newshour/rundown/how-going-under-the-knife-became-much-less-deadly/ [8] Gaston, R. G., & Kuremsky, M. A. (2010). Postoperative infections: prevention and management. Hand clinics, 26(2), 265-280. [9] Johnson, R., O'Hara, K., Sellen, A., Cousins, C., & Criminisi, A. (2011, May). Exploring the potential for touchless interaction in image-guided interventional radiology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3323-3332. [10] Greenberg, S., Marquardt, N., Ballendat, T., Diaz-Marino, R., & Wang, M. (2011). Proxemic interactions: the new ubicomp? Interactions, 18(1), 42-50. [11] Mentis, H. M., O'Hara, K., Sellen, A., & Trivedi, R. (2012, May). Interaction proxemics and image use in neurosurgery. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems. 927-936 [12] Surgeons: Operate with Kinect. GestSure.com. Retrieved February 1, 2014 from https://www.youtube.com/watch?v=_KcZwP8L4z0  [13]  Wachs, J., Stern, H., Edan, Y., Gillam, M., Feied, C., Smithd, M., & Handler, J. (2007). Real-time hand gesture interface for browsing medical images. International Journal of Intelligent Computing in Medical Sciences & Image Processing, 3(1), 175-185. 44  [14]  Wachs, J. P., Stern, H. I., Edan, Y., Gillam, M., Handler, J., Feied, C., & Smith, M. (2008). A gesture-based tool for sterile browsing of radiology images. Journal of the American Medical Informatics Association, 15(3), 321-323. [15]  Kipshagen, T., Graw, M., Tronnier, V., Bonsanto, M., & Hofmann, U. G. (2009, January). Touch-and marker-free interaction with medical software. In World Congress on Medical Physics and Biomedical Engineering, September 7-12, 2009, Munich, Germany, 75-78. [16] Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., and Moore, R. (2013). Real-time human pose recognition in parts from single depth images. Communications of the ACM, 56(1), 116-124. [17] Reality Controls Inc. Control:mapper. Retrieved April 24, 2013 from http://www.controlmapper.com/ [18] Sunnybrook Hospital, Toronto, Canada. Xbox Kinect in the hospital operating room. Sunnybrook.ca. Retrieved January 5, 2012 from http://sunnybrook.ca/media/item.asp?i=616  [19]  Strickland, M., Tremaine, J., Brigley, G., & Law, C. (2013). Using a depth-sensing infrared camera system to access and manipulate medical imaging from within the sterile operating field. Canadian Journal of Surgery, 56(3), E1-E6.  [20] Ebert, L. C., Hatch, G., Ampanozi, G., Thali, M. J., & Ross, S. (2012). You Can’t Touch This Touch-free Navigation Through Radiological Images. Surgical innovation, 19(3), 301-307. 45  [21] Gallo, L., Placitelli, A. P., & Ciampi, M. (2011, June). Controller-free exploration of medical image data: Experiencing the Kinect. In Computer-Based Medical Systems (CBMS), 2011 24th International Symposium. 1-6. [22] Jacob, M. G., Wachs, J. P., & Packer, R. A. (2012). Hand-gesture-based sterile interface for the operating room using contextual cues for the navigation of radiological images. Journal of the American Medical Informatics Association, 20(e1), e183-e186. [23] Suelze, B., Agten, R., Bertrand, P. B., Vandenryt, T., Thoelen, R., Vandervoort, P., & Grieten, L. (2013, September). Waving at the Heart: Implementation of a Kinect-based real-time interactive control system for viewing cineangiogram loops during cardiac catheterization procedures. In Computing in Cardiology Conference (CinC), 2013. 229-232. [24] Tan, J. H., Chao, C., Zawaideh, M., Roberts, A. C., & Kinney, T. B. (2013). Informatics in radiology: Developing a touchless user interface for intraoperative image control during interventional radiology procedures. Radiographics, 33(2), E61-E70. [25] Tedcas & Clear Canvas: Touch free DICOM viewer based on Kinect. Tedcas.com. Retrieved from July 18, 2013 from https://www.youtube.com/watch?v=PxMr6GKBi4U  [26] Tedcas & Leap Motion. Tedcas.com. Retrieved November 30, 2013 from https://www.youtube.com/watch?v=6d_Kvl79v6E  46  [27]  Tani, B. S., Maia, R. S., & von Wangenheim, A. (2007, June). A gesture interface for radiological workstations. In Computer-Based Medical Systems, 2007. CBMS'07. Twentieth IEEE International Symposium. 27-32. [28] Pu, Q., Gupta, S., Gollakota, S., & Patel, S. (2013, September). Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. 27-38. [29] Harrison, C., Benko, H., and Wilson, A. D. 2011. OmniTouch: Wearable Multitouch Interaction Everywhere. In Proceedings of the 24th Annual ACM Symposium on User interface Software and Technology. 441-450. [30]  Kellogg, B., Talla, V., & Gollakota, S. Bringing Gesture Recognition To All Devices. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation. 303-316. [31] CamBoard pico S – 3D reference camera with high frame rate. PMDTEC.com. Retrieved August 23, 2013 from http://www.pmdtec.com/news_media/video/camboard_pico_s_high_framerate.php  [32] Wobbrock,  J. O., Wilson, A. D., and Li, Y. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 159–168.  [33] WEASIS Dicom Viewer. Dcm4che.org. Retrieved December 20, 2013 from http://www.dcm4che.org/confluence/display/WEA/Home    47  APPENDICES A.1 Consent Form    48  A.2 Questionnaire Form   49    50    51             52  A.3 Complete Questionnaire results   Radiologist Orthopedic Surgeon Other     Specialty 9 3 4    56% 19% 25%      0 to 5 6 to 10 11 to 15 16 to 20 21 to 25 Years of experience (since beginning of subspecialty training): 5 5 4 2 0 31% 31% 25% 13% 0%   Never Almost Never Sometimes Almost Always Always Do you review patient's prior imaging (ie X-Rays / MRI / CT etc) while performing a procedure? 2 1 2 0 11 13% 7% 13% 0% 69%   Not Important   Neutral   Very Important Is being able to view and manipulate imaging during a procedure important to you? 0 1 1 5 9 0% 6% 6% 31% 56%   Strongly Disagree Disagree Neutral Agree Strongly Agree Would you be likely to view a patient's imaging during a procedure more frequently if you could do so in a sterile fashion? 0 0 0 4 12 0% 0% 0% 25% 75%   No experience Kinect Wii Leap Motion   Do you have any prior experience with interactive gesture sensors? 11 3 4 2     Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied How was the overall experience with the interactive component of the experiment? 0 0 0 10 6 0% 0% 0% 63% 38%   Hard Somewhat Difficult Neutral Understandable Easy Were the instructions easy to follow? 0 0 0 10 6 0% 0% 0% 63% 38%   Hard Somewhat Difficult Neutral Understandable Easy How was your user experience with the zoom function? 0 0 2 9 5 0% 0% 13% 56% 31%   Yes No Other     Were you able to obtain the 13 0 3    53  specified zoom size you desired? 81% 0% 19%      Hard Somewhat Difficult Neutral Understandable Easy How was your user experience with the scroll function? 0 0 3 8 5 0% 0% 19% 50% 31%   Yes No Other     Were you able to scroll to the exact frame of image you desired? 13 1 2    81% 6% 13%      Hard Somewhat Difficult Neutral Understandable Easy How was your user experience with fine tuning the window width & level? 0 1 2 11 2 0% 6% 13% 69% 13%   Yes No Other     Were you able to get the specific window for the image you desired? 16 0 0    100% 0% 0%      Hard Somewhat Difficult Neutral Understandable Easy How was your user experience with the window level presets (swipe)? 0 0 2 8 6 0% 0% 13% 50% 38%   Hard Somewhat Difficult Neutral Understandable Easy How was your user experience with the pan function? 0 0 0 7 9 0% 0% 0% 44% 56%   Yes No Other     Were you able to pan the image to the exact location you wanted? 16 0 0    100% 0% 0%      Hard Somewhat Difficult Neutral Understandable Easy How easy was it to change between swipe functions (ie from ww/wl presets to next/prev image)? 0 0 3 10 3 0% 0% 19% 63% 19%   Strongly Disagree Disagree Neutral Agree Strongly Agree Do you think this prototype could reduce the time needed to perform a procedure? 0 0 4 7 5 0% 0% 25% 44% 31%   Strongly Disagree Disagree Neutral Agree Strongly Agree Do you think this prototype has all the manipulation tools needed during a procedure? 0 0 3 7 6 0% 0% 19% 44% 38%   Hard Somewhat Difficult Neutral Understandable Easy Were the one-handed gestures easy to understand and navigate? 0 0 0 9 7 0% 0% 0% 56% 44% 54  A.4 LeWP Software The Weasis DICOM Viewer source code that I used for the LeWP System can be found at http://www.dcm4che.org/confluence/display/WEA/Home. Apache Maven is also required in order to develop the plugin for Weasis. Apache Maven can be downloaded from http://maven.apache.org/.  To read the sensor inputs from the Leap Motion®  and use its tracking algorithms, the program will rely on drivers and Leap Motion’s Software Development Kit (found at https://developer.leapmotion.com/).   

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.24.1-0167492/manifest

Comment

Related Items