Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Image and haptic guidance for robot-assisted laparoscopic surgery Mohareri, Omid 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2015_november_mohareri_omid.pdf [ 28.45MB ]
JSON: 24-1.0165820.json
JSON-LD: 24-1.0165820-ld.json
RDF/XML (Pretty): 24-1.0165820-rdf.xml
RDF/JSON: 24-1.0165820-rdf.json
Turtle: 24-1.0165820-turtle.txt
N-Triples: 24-1.0165820-rdf-ntriples.txt
Original Record: 24-1.0165820-source.json
Full Text

Full Text

Image and Haptic Guidance forRobot-Assisted Laparoscopic SurgerybyOmid MohareriM.Sc., American University of Sharjah, 2009B.Sc., Shiraz University, 2007A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Electrical & Computer Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)October 2015© Omid Mohareri 2015AbstractSurgical removal of the prostate gland using the da Vinci surgical robot isthe state of the art treatment option for organ confined prostate cancer. Theda Vinci system provides excellent 3D visualization of the surgical site andimproved dexterity, but it lacks haptic force feedback and subsurface tissuevisualization.The overall objective of the work done in this thesis is to augment theexisting visualization tools of the da Vinci with ones that can identify theprostate boundary, critical structures, and cancerous tissue so that prostateresection can be carried out with minimal damage to the adjacent criticalstructures, and therefore, with minimal complications.Towards this objective we designed and implemented a real-time imageguidance system based on a robotic transrectal ultrasound (R-TRUS) plat-form that works in tandem with the da Vinci surgical system and tracks itssurgical instruments.In addition to ultrasound as an intrinsic imaging modality, the systemwas first used to bring pre-operative magnetic resonance imaging (MRI) tothe operating room by registering the pre-operative MRI to the intraop-erative ultrasound and displaying the MRI image at the correct physicallocation based on the real-time ultrasound image. Second, a method of us-ing the R-TRUS system for tissue palpation is proposed by expanding it tobe used in conjunction with a real-time strain imaging technique. Third,another system based on the R-TRUS is described for detecting dominantprostate tumors, based on a combination of features extracted from a novelmulti-parametric quantitative ultrasound elastography technique.We tested our systems in an animal study followed by human patientstudies involving n = 49 patients undergoing da Vinci prostatectomy. TheiiAbstractclinical studies were conducted to evaluate the feasibility of using these sys-tems in real human procedures, and also to improve and optimize our imag-ing systems using patient data.Finally, a novel force feedback control framework is presented as a so-lution to the lack of haptic feedback in the current clinically used surgicalrobots. The framework has been implemented on the da Vinci surgical sys-tem using the da Vinci Research Kit controllers and its performance hasbeen evaluated by conducting user studies.iiiPrefaceThis thesis is written based on several published manuscripts resulting fromthe work done by the author and in collaboration with multiple researchers.Clinical research ethics approvals for the clinical studies conducted for thisthesis were obtained from the UBC Clinical Research Ethics Board (CREB)(application numbers: H11-02267 and H08-02696).A modified version of Chapter 2 has been published in the followingpublications:ˆ O. Mohareri, J. Ischia, P. Black, C. Schneider, J. Lobo, S. L. Golden-berg and S. E. Salcudean, “Intraoperative Registered Transrectal Ul-trasound Guidance for Robot-Assisted Laparoscopic Radical Prosta-tectomy,” The Journal of Urology, vol. 193, Issue 1, pp: 302–312,2015.ˆ O. Mohareri, J. Ischia, P. Black, S. L. Goldenberg, J. Lobo, C.Schneider and S. E. Salcudean, “Intraoperative Registered TransrectalUltrasound Guidance for Robot-Assisted Laparoscopic Radical Prosta-tectomy: A 20 patient study,” The Journal of Urology (AUA AnnualMeeting), vol. 191, Issue 4, pp: 398–399, 2014.ˆ O. Mohareri, J. Ischia, P. Black, S. L. Goldenberg, J. Lobo, C.Schneider and S. E. Salcudean, “Real-time robotic transrectal ultra-sound guidance for robot-assisted laparoscopic radical prostatectomy:a 20 patient clinical study,” British Journal of Urology (BJU) Inter-national, vol. 113, pp: 84–85, 2014.ˆ O. Mohareri, C. Schneider, T. K. Adebar, M. Yip, P. Black, C.Nguan, D. Bergman, J. Sorger, S. DiMaio and S. E. Salcudean, “Ultra-ivPrefacesound Based Image Guidance for Robot-Assisted Laparoscopic Radi-cal Prostatectomy: Initial in-vivo results,” Information Processing inComputer Assisted Interventions (IPCAI), LNCS vol. 7915, pp: 40–50, June 2013.ˆ O. Mohareri, J. Ischia, C. Schneider, P. Black and S. E. Salcud-ean, “Robotic, Registered Transrectal Ultrasound Guidance during daVinci Radical Prostatectomy: Initial Clinical Experience,” in Proc.The Hamlyn Symposium on Medical Robotics, pp: 11–12, ImperialCollege London, June 2013.A modified version of Chapter 3 has been published as the followingpublications:ˆ O. Mohareri, G. Nir, J. Lobo, R. Savdie, P. Black and S. E. Sal-cudean, “A System for MR-Guidance during Robot-Assisted Laparo-scopic Radical Prostatectomy,” in Proc. Medical Image Computingand Computer Assisted Interventions (MICCAI), 2015.ˆ O. Mohareri, G. Nir, J. Lobo, R. Savdie, P. Black and S. E. Sal-cudean, “Fused MRI-Ultrasound Guidance for Robot-Assisted Laparo-scopic Prostatectomy: System Architecture and First Clinical Use,” inProc. The Hamlyn Symposium on Medical Robotics, Imperial CollegeLondon, June 2015.A modified version of Chapter 4 has been published as the followingpublications:ˆ O. Mohareri, Mahdi Ramezani, T. K. Adebar, P. Abolmaesumi andS. E. Salcudean, “Automatic Localization of the da Vinci SurgicalInstrument Tips in 3-D Transrectal Ultrasound,” IEEE Transactionson Biomedical Engineering, vol. 60, No. 9, pp: 2663–2672, 2013.ˆ O. Mohareri, M. Ramezani, T. K. Adebar, P. Abolmaesumi and S.E. Salcudean, “Automatic Detection and Localization of da Vinci tooltips in 3D ultrasound,” Information Processing in Computer AssistedInterventions (IPCAI), LNCS vol. 7330, pp: 22–32, 2012.vPrefaceˆ T. K. Adebar, O. Mohareri and S. E. Salcudean, “Instrument-BasedCalibration and Control of Intraoperative Ultrasound for Robot-AssistedSurgery,” in Proc. IEEE Int. Conf. on Biomedical Robotics andBiomechatronics (BioRob), pp: 38–43, 2012.The author’s contribution in the work resulting into the above articleswas: integrating the system in terms of both software and hardware com-ponents; building and conducting lab experiments; formulating and imple-menting the automatic registration algorithm; coordinating and conduct-ing one animal study at Intuitive Surgical Inc. research and developmentlabs in Sunnyvale, California; coordinating with Vancouver General Hospi-tal (VGH) and UBC department of Urological Sciences staff to conduct theclinical studies; help writing the study clinical research ethics application;conducting the clinical studies in the robotic operating room (OR) of VGH(n = 24 patients); system sterilization for operating room re-use; OR datacollection software writing; processing the acquired data and evaluating theresults; and writing the manuscripts.A modified version of Chapter 4 has been published as the followingpublications:ˆ O. Mohareri, C. Schneider and S. E. Salcudean, “Instrument-basedregistered strain imaging for remote palpation in robot-assisted laparo-scopic surgery,” in Proc. The Hamlyn Symposium on Medical Robotics,Imperial College London, pp: 57, July 2014.We are in the process of acquiring more patient data to complete the studypresented in the above paper and submit the journal version of this work.The author’s contribution in the above articles was: integrating the systemin terms of both software and hardware components; coordinating with Van-couver General Hospital (VGH) and UBC department of Urological Sciencesstaff to conduct the clinical studies; help writing the study clinical researchethics application; conducting the clinical studies in the robotic operatingroom (OR) of VGH (n = 5 patients); system sterilization for operatingroom re-use; OR data collection software writing; processing the acquireddata and evaluating the results; and writing the manuscript.viPrefaceA modified version of Chapter 5 has been published as the followingpublications:ˆ O. Mohareri, A. Ruszkowski, J. Lobo, A. Baghani, J. Ischia, E.Jones, L. Fazli, L. Goldenberg, M. Moradi and S. E. Salcudean, “Multi-parametric 3D Quantitative Ultrasound Elastography Imaging for De-tecting Palpable Prostate Tumors,” Medical Image Computing andComputer Assisted Interventions (MICCAI), Part 1, LNCS vol. 8673,pp: 561–568, 2014.ˆ M. Moradi, S. S. Mahdavi, G. Nir, O. Mohareri, A. Koupparis, L.O. Gognan, R. G. Casey, E. C. Jones, S. L. Goldenberg and S. E.Salcudean, “Multiparametric 3D Ultrasound vibroelastography andradiofrequency imaging of prostate cancer,” Medical Physics, vol. 41,Issue 7, 073505 (2014).We are in the process of acquiring more patient data to complete thestudy presented in the MICCAI 2014 paper and submit the journal versionof this work.The author’s contribution in the above articles was: conducting the clin-ical studies in the robotic operating room (OR) of VGH (n = 20 patients);data processing using machine learning algorithms; coordinating the re-search with Vancouver General Hospital (VGH) and UBC department ofUrological Sciences staff to conduct the clinical studies; system sterilizationfor operating room re-use; evaluating the results; desing and building of theinternal shaker device; and writing the manuscript.A modified version of Chapter 6 has been published as the followingpublications:ˆ O. Mohareri and S. E. Salcudean, “A Novel Force Feedback ControlStructure for Bimanual Telerobotic Surgery,” IEEE Transactions onRobotics, in press (T-RO 13-0441).ˆ O. Mohareri, C. Schneider and S. E. Salcudean, “Bimanual Robot-Assisted Surgery with Asymmetric Haptic Force Feedback: A da VinciviiPrefacesurgical system implementation,” in Proc. IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS), pp: 4272–4277,2014.ˆ O. Mohareri, S. E. Salcudean and C. Nguan, “Asymmetric ForceFeedback Control Framework for Robot-Assisted Surgery,” in Proc.International Conference on Robotics and Automation (ICRA), pp:5800–5806, May 2013.The author’s contribution in the above articles was: formulating andimplementing the algorithm first in Matlab simulation; implementing theidea on four haptic devices in the lab and conducting experiments withthem; establishing the da Vinci research kit (dVRK) system in the lab byintegrating all hardware and software components to the da Vinci Standardsystem; implementing the control method on the da Vinci system; softwareand hardware debugging and maintenance for the dVRK system; conductinguser studies with the system (n = 9 users); data processing and evaluatingthe results; and writing the manuscript. The control algorithm and in-frastructure developed by the author for this work was also used for otherprojects in the lab which resulted into the following publications:ˆ A. Ruszkowski, O. Mohareri, S. Lichtenstein, R. Cook and S. E.Salcudean, “On the Feasibility of Heart Motion Compensation on theda Vinci Surgical Robot for Coronary Artery Bypass Surgery: Imple-mentation and User Studies,” in Proc. International Conference onRobotics and Automation (ICRA), May 2015.ˆ I. Tong, O. Mohareri, S. Tatasurya and S. E. Salcudean, “A RetrofitEye Gaze tracker for the da Vinci and its Integration in Task Ex-ecution using the da Vinci Research Kit,” submitted to IEEE/RSJInternational Conference on Intelligent Robots and Systems (IROS),2015.A modified version of Appendix B has been published as the followingpublications:viiiPrefaceˆ O. Mohareri and S. E. Salcudean, “da Vinci® auxiliary arm as arobotic surgical assistant for semi-autonomous ultrasound guidanceduring robot-assisted laparoscopic surgery,” in Proc. The HamlynSymposium on Medical Robotics, Imperial College London, pp: 45,July 2014.ixTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Clinical Motivation . . . . . . . . . . . . . . . . . . . . . . . 11.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Thesis Outline and Contributions . . . . . . . . . . . . . . . 42 Intraoperative Transrectal Ultrasound Guidance for Robot-Assisted Laparoscopic Prostatectomy . . . . . . . . . . . . . 82.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . 102.2.1 TRUS Robot . . . . . . . . . . . . . . . . . . . . . . . 102.2.2 TRUS Robot Calibration to the da Vinci . . . . . . . 112.3 In-vivo Studies . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3.1 Canine Study . . . . . . . . . . . . . . . . . . . . . . 142.3.2 Discussion of the Results . . . . . . . . . . . . . . . . 20xTable of Contents2.3.3 Animal Study Conclusions . . . . . . . . . . . . . . . 222.4 Human Patient Studies . . . . . . . . . . . . . . . . . . . . . 222.4.1 Patient Population . . . . . . . . . . . . . . . . . . . 222.4.2 Procedure Description . . . . . . . . . . . . . . . . . . 232.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 252.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 313 A System for MR-Guidance During Robot-Assisted Laparo-scopic Radical Prostatectomy . . . . . . . . . . . . . . . . . . 343.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . 373.2.1 TRUS Imaging System . . . . . . . . . . . . . . . . . 373.2.2 TRUS-MR Registration . . . . . . . . . . . . . . . . . 393.2.3 In-vivo Patient Studies and Results . . . . . . . . . . 403.3 Discussions and Conclusions . . . . . . . . . . . . . . . . . . 434 Automatic 3-D Transrectal Ultrasound to the da Vinci Sur-gical System Registration . . . . . . . . . . . . . . . . . . . . . 474.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . 504.2.1 Automatic Detection Algorithm . . . . . . . . . . . . 514.2.2 Experimental Setup for TRUS Data Collection . . . . 544.2.3 System Interface . . . . . . . . . . . . . . . . . . . . . 544.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 544.3.1 Phantom and Ex-vivo Results . . . . . . . . . . . . . 544.3.2 In-vivo Results . . . . . . . . . . . . . . . . . . . . . . 614.4 Discussion of Results . . . . . . . . . . . . . . . . . . . . . . 624.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 655 Tracked Robotic Ultrasound Palpation . . . . . . . . . . . . 675.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 685.1.1 Background Review . . . . . . . . . . . . . . . . . . . 695.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . 705.2.1 Real-Time Strain Imaging System . . . . . . . . . . . 72xiTable of Contents5.3 Phantom and Ex-vivo User Study and Results . . . . . . . . 735.4 Human Patient Studies . . . . . . . . . . . . . . . . . . . . . 765.4.1 Patient Population . . . . . . . . . . . . . . . . . . . 785.4.2 Procedure Description . . . . . . . . . . . . . . . . . . 785.4.3 Results and Discussions . . . . . . . . . . . . . . . . . 806 Multi-parametric 3D Quantitative Ultrasound Vibro-ElastographyImaging for Detecting Palpable Prostate Tumors . . . . . 886.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 896.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 936.3.1 Registration . . . . . . . . . . . . . . . . . . . . . . . 936.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 986.5 Discussions and Conclusions . . . . . . . . . . . . . . . . . . 997 A Novel Force Feedback Control Structure for Robot-AssistedLaparoscopic Surgery . . . . . . . . . . . . . . . . . . . . . . . 1027.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1037.2 Asymmetric Control Architecture . . . . . . . . . . . . . . . 1087.2.1 Stability and Transparency . . . . . . . . . . . . . . . 1137.3 Controller Implementation on the da Vinci Surgical System . 1137.3.1 System Components . . . . . . . . . . . . . . . . . . . 1147.3.2 Bimanual Teleoperation Controller Implementation . 1167.3.3 Force Feedback Controller Implementation . . . . . . 1187.4 User Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . 1247.4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . 1247.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 1257.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 1288 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135xiiTable of ContentsAppendicesA Semi-Autonomous Ultrasound Guidance During Robot-AssistedLaparoscopic Surgery . . . . . . . . . . . . . . . . . . . . . . . 151A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 152A.2 Materials and Methods . . . . . . . . . . . . . . . . . . . . . 155A.2.1 System Overview . . . . . . . . . . . . . . . . . . . . 155A.2.2 3D Ultrasound and Robot Instrument Registration . 155A.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . 158A.3.1 da Vinci Surgical Instrument Motion Accuracy in theDVRK Setup . . . . . . . . . . . . . . . . . . . . . . . 158A.4 Discussions, Future Work and Conclusions . . . . . . . . . . 160B Clinical Study Protocols . . . . . . . . . . . . . . . . . . . . . 162B.1 Animal Study at Intuitive Surgical Inc (22 October 2012 ) . . 162B.1.1 Setup Preparation . . . . . . . . . . . . . . . . . . . . 162B.1.2 Pre-Operative Phase: . . . . . . . . . . . . . . . . . . 163B.1.3 Intra-Operative Phase: . . . . . . . . . . . . . . . . . 163B.1.4 System Evaluation Criterion: . . . . . . . . . . . . . . 168B.2 Transrectal Probe Disinfection at BC Cancer Agency . . . . 169C Elastography and the da Vinci . . . . . . . . . . . . . . . . . 171C.1 Tissue Exciter Design and Building for Intraoperative Elas-tography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172C.2 Clinical Integration Plan . . . . . . . . . . . . . . . . . . . . 174xiiiList of Tables2.1 3D TRUS to da Vinci registration accuracy . . . . . . . . . . 172.2 Automatic tracking accuracy . . . . . . . . . . . . . . . . . . 192.3 Demographic and preoperative data. . . . . . . . . . . . . . . 232.4 Intraoperative TRUS results. . . . . . . . . . . . . . . . . . . 273.1 Table of MR-guidance patient study results . . . . . . . . . . 424.1 Automatic registration accuracy . . . . . . . . . . . . . . . . 594.2 Localization accuracy results . . . . . . . . . . . . . . . . . . 615.1 Stiff inclusion exploration user study results. . . . . . . . . . 755.2 Demographic and preoperative data. . . . . . . . . . . . . . . 786.1 Table of features . . . . . . . . . . . . . . . . . . . . . . . . . 966.2 Classification results . . . . . . . . . . . . . . . . . . . . . . . 99C.1 Internal shaker components. . . . . . . . . . . . . . . . . . . . 174xivList of Figures1.1 Thesis outline. . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 The TRUS robot. . . . . . . . . . . . . . . . . . . . . . . . . . 102.2 Automatic instrument tip tracking concept. . . . . . . . . . . 122.3 Intraoperative TRUS to da Vinci calibration procedure. . . . 132.4 Animal clinical lab at Intuitive Surgical research labs. . . . . 152.5 TRUS to da Vinci registration data collection. . . . . . . . . 182.6 Registration accuracy vs number of fiducials. . . . . . . . . . 192.7 TilePro—images for the animal study. . . . . . . . . . . . . . . 202.8 TRUS guidance system components in the OR . . . . . . . . 242.9 Operating room scenario . . . . . . . . . . . . . . . . . . . . . 262.10 Bladder neck dissection . . . . . . . . . . . . . . . . . . . . . 282.11 Seminal vesicle removal . . . . . . . . . . . . . . . . . . . . . 292.12 Dissection at rectal wall and DVC suture . . . . . . . . . . . 302.13 Dissection at the apex . . . . . . . . . . . . . . . . . . . . . . 323.1 Instrument-based strain concept . . . . . . . . . . . . . . . . 383.2 MR-guidance system components . . . . . . . . . . . . . . . . 413.3 MR-guidance imaging interface . . . . . . . . . . . . . . . . . 433.4 MR-guidance clinical study results . . . . . . . . . . . . . . . 453.5 MR and TRUS guidance in Tilepro—results . . . . . . . . . . 463.6 MR and TRUS guidance imaging interface . . . . . . . . . . . 464.1 Surgical scene during RALRP . . . . . . . . . . . . . . . . . . 484.2 Air-tissue boundary . . . . . . . . . . . . . . . . . . . . . . . 504.3 Automatic detection algorithm . . . . . . . . . . . . . . . . . 52xvList of Figures4.4 Registration workflow . . . . . . . . . . . . . . . . . . . . . . 554.5 Sample detection results . . . . . . . . . . . . . . . . . . . . . 564.6 Automatic detection results . . . . . . . . . . . . . . . . . . . 584.7 Automatic registration accuracy . . . . . . . . . . . . . . . . 605.1 Instrument-based strain concept . . . . . . . . . . . . . . . . 715.2 Strain user studies with CIRS phantom and daVinci Si . . . . 745.3 CIRS phantom strain results . . . . . . . . . . . . . . . . . . 765.4 ex-vivo kidney strain results. . . . . . . . . . . . . . . . . . . 775.5 Porcine kidney experimental setup . . . . . . . . . . . . . . . 775.6 TRUS and strain system components . . . . . . . . . . . . . . 805.7 OR scenario during strain imaging study . . . . . . . . . . . . 815.8 Strain imaging interface and results . . . . . . . . . . . . . . . 825.9 Different instruments used for strain imaging . . . . . . . . . 835.10 The imaging system interface . . . . . . . . . . . . . . . . . . 845.11 Strain image Tilepro—images . . . . . . . . . . . . . . . . . . 855.12 Strain imaging useful information . . . . . . . . . . . . . . . . 865.13 Pulsatile motion . . . . . . . . . . . . . . . . . . . . . . . . . 865.14 Instrument-based strain patient results . . . . . . . . . . . . . 876.1 Quantitative elastography system components . . . . . . . . . 916.2 Clinical setting . . . . . . . . . . . . . . . . . . . . . . . . . . 926.3 Example pathology slices . . . . . . . . . . . . . . . . . . . . 936.4 Example absolute elastography images . . . . . . . . . . . . . 946.5 TRUS to pathology registration . . . . . . . . . . . . . . . . . 956.6 Classification results . . . . . . . . . . . . . . . . . . . . . . . 1007.1 Clinical example during RALRP . . . . . . . . . . . . . . . . 1057.2 The asymmetric control framework . . . . . . . . . . . . . . . 1077.3 Control structure comparison . . . . . . . . . . . . . . . . . . 1087.4 Asymmetric control structure block diagram . . . . . . . . . . 1117.5 da Vinci Research Kit platform in UBC RCL Laboratory . . 1157.6 The bimanual teleoperation framework on dVRK . . . . . . . 1187.7 Controller implementation results . . . . . . . . . . . . . . . . 119xviList of Figures7.8 Haptic exploration experiment . . . . . . . . . . . . . . . . . 1217.9 Knot tying experiment . . . . . . . . . . . . . . . . . . . . . . 1237.10 Knot tying experiment . . . . . . . . . . . . . . . . . . . . . . 1267.11 Tension spring experiment . . . . . . . . . . . . . . . . . . . . 1277.12 User study results . . . . . . . . . . . . . . . . . . . . . . . . 1287.13 User study results . . . . . . . . . . . . . . . . . . . . . . . . 129A.1 dVRK system for ultrasound guidance . . . . . . . . . . . . . 156A.2 Auxiliary arm registration concept . . . . . . . . . . . . . . . 157A.3 Motion accuracy testing of the dVRK system . . . . . . . . . 159A.4 Semi-autonomous guidance on ex-vivo kidney . . . . . . . . . 160A.5 TRUS images of the kidney experiment . . . . . . . . . . . . 160A.6 dVRK semi-autonomous guidance . . . . . . . . . . . . . . . 161C.1 Internal shaker design . . . . . . . . . . . . . . . . . . . . . . 173C.2 Internal shaker components . . . . . . . . . . . . . . . . . . . 173C.3 da Vinci adapter designs . . . . . . . . . . . . . . . . . . . . . 175C.4 Internal shaker grasped by da Vinci . . . . . . . . . . . . . . 175xviiAcknowledgementsI would like to thank my supervisor Prof. Tim Salcudean for giving me theopportunity to work on these meaningful and exciting research projects andhelping me become who I wanted to be in my professional life.xviiiDedicationI dedicate this work to my parents, Farzad Mohareri and Maryam Mirah-madi, for their endless love, support and encouragement.xixChapter 1Introduction1.1 Clinical MotivationProstate cancer (PCa) remains the most common solid organ malignancy inmen and is responsible for more deaths than any other cancer except lungcancer. Approximately 220,800 new cases will be diagnosed and 27,540 willdie from PCa in the United States in 2015 [2]. Radical prostatectomy (RP),the surgical removal of the prostate gland and surrounding tissue, is thestandard-of-care option if the cancer is not thought to have spread outsidethe gland (stage T1 or T2 cancers). Surgical options include the traditionalopen RP (ORP), minimally invasive laparoscopic RP (LRP), and robot-assisted laparoscopic RP (RALRP). The RALRP procedure involves the daVinci surgical system (Intuitive Surgical Inc., Sunnyvale, CA, USA), whichprovides the surgeon with enhanced three-dimensional (3-D) visualizationof the surgical site and improved dexterity over standard laparoscopic in-struments. The da Vinci surgical system is now used to perform as manyas four out of five RP procedures in the United States[81].Three important goals in RP are: 1) cancer control, 2) preservationof urinary continence, and 3) preservation of sexual function. Success inthese goals is believed to be associated with the accurate delineation of can-cerous lesions, prostate boundaries, and periprostatic anatomy such as theurethral sphincter muscle and neurovascular bundles. The difficulty in thisprocedure remains the fundamental tradeoff between the cancer contain-ment goal (achieving negative margins) and the quality of life goal (sparingof critical structures).In spite of the excellent, magnified, three-dimensional visualization, im-proved access and superior surgical dexterity to the operating surgeon dur-11.2. Backgrounding RALRP, there remains a distinct lack of guidance tools available forreal time in-field determination of the prostate boundary and of the criti-cal structures adjacent to it. The location of peripheral tumors and theirpossible extension in the proximity of such critical structures would alsohelp identify the surgical resection planes that would provide the absolutebest tradeoff between cancer containment and the sparing of critical struc-tures. Intraoperative medical image guidance is a tool that could potentiallyovercome this deficiency and improve outcomes from RALRP.1.2 BackgroundTransrectal ultrasound (TRUS) is the imaging modality most commonlyapplied for diagnostic and therapeutic purposes in PCa. Several studieshave shown the potential benefits of integrating TRUS in variants of RP,including the potential improvement of positive surgical margin rates andurinary and potency functional outcomes [78, 107, 108, 110]. Ukimura etal.[108] found TRUS useful for identifying the correct plane between thebladder neck and the prostate base, and for providing visualization of anyhypoechoic nodules abutting the prostate capsule. They also reported thatintraoperative TRUS decreased positive surgical margin rates from 29% to9% of the patients [110]. Van der Poel et al. [83] reported that intraoperativeTRUS during RALRP significantly decreased positive surgical margin ratesat the prostate base for an experienced laparoscopic surgeon who was newto the robotic procedure. The most significant limitation of all these studiesis that the TRUS imaging was performed by an assistant who manuallyadjusted the probe during surgery. This is problematic in RALRP, becausethe da Vincis patient-side cart is generally located adjacent to the operatingtable between the stirrups that hold the patients legs, limiting the assistantsaccess to the patient. In addition, the surgeon must guide the assistantverbally during the procedure, and their ability to coordinate effectivelywill strongly affect the usefulness of TRUS.Recently, robotic TRUS manipulators have been used for real-time guid-ance during RALRP procedures [38, 41, 58]. Hung et al. used a robotic21.2. BackgroundTRUS manipulator (ViKY system, EndoControl medical, Grenoble, France)for real-time monitoring of the prostate and periprostatic anatomy. Theyshowed that using robotic TRUS is feasible and safe, and it provided thesurgeon with valuable anatomic information [41]. Long et al. used the sameTRUS robot to visualize real-time bladder neck dissection, NVB release andapical dissection [58]. They showed that using robotic TRUS resulted inno positive surgical margins in five patients. Han et al. used their custom-made robotic TRUS manipulator for improved visualization of the NVB.This study demonstrated that the prostate can be safely scanned using theTRUS robot, to reconstruct the 3D images of the prostate gland and ad-jacent NVB, and the intra-abdominal da Vinci instruments can be clearlyvisualized in the TRUS images [38].In previous in-vivo studies, the TRUS manipulators have not been registeredto the da Vinci robot or camera, and therefore the ultrasound image couldnot be presented at the correct location in space relative to the console viewor the da Vinci instruments. The control of the TRUS image location fromwithin the da Vinci console has also not been demonstrated before in in-vivostudies.The work presented in this thesis is towards using a robotic TRUS systemthat has been integrated to the da Vinci surgical system in order to provideintuitive image guidance to the robotic surgeon. We present the roboticTRUS system and details of its clinical integration and studies.Our hypothesis is that advanced trans-rectal ultrasound imaging canbe deployed and used easily during surgery, can be registered to the robotcoordinate systems with high accuracy, and can be controlled from the sur-geons console, in order to improve the visualization of the prostate andperi-prostatic anatomy, and in order to produce a cancer probability mapthat can be used to make decisions on surgical margins. In addition to ul-trasound imaging as an intrinsic imaging modality, ultrasound can also beused to bring in pre-operative magnetic resonance imaging (MRI) to theoperating room by registering the pre-operative MRI to the intra-operativeultrasound and displaying the MRI image at the correct physical locationbased on the real-time ultrasound image.31.3. Thesis Outline and Contributions1.3 Thesis Outline and ContributionsThis thesis covers the background literature related to ultrasound and hap-tic guidance systems for robot-assisted surgery, the proposed system design,algorithms, and the validation results on phantom as well as in animal andhuman subjects. The summary of the thesis outline is illustrated in Fig-ure 1.1. A detailed outline of the thesis is as follows:Chapter 2: Intraoperative Transrectal Ultrasound Guidance forRobot-Assisted Laparoscopic Prostatectomy This chapter describesthe robotic TRUS guidance system, intraoperative TRUS to da Vinci sur-gical system registration, system integration for in-vivo animal and humanclinical studies, details and protocol of the clinical studies, study outcomedata analysis and evaluation of the results.Chapter 3: A System for MR-Guidance During Robot-AssistedLaparoscopic Radical Prostatectomy This chapter explains the ex-tension of the robotic TRUS system explained in Chapter 2 to be used forMRI guidance during da Vinci prostatectomy. The outline of the approachesused and our experience with the system in the first two patients are de-scribed here.Chapter 4: Automatic 3-D Transrectal Ultrasound to the da VinciSurgical System Registration This chapter describes a technique forautomatic instrument localization in 3D TRUS, automatic registration ex-periments using the proposed method (tissue phantoms, ex-vivo and in-vivo), registration accuracy analysis, system integration, evaluation of theresults and discussion of the limitations.Chapter 5: Tracked Robotic Ultrasound Palpation This chapterdescribes a method of using ultrasound imaging for tissue palpation us-ing a robotic ultrasound system that can track a surgical instrument. Themethod is implemented both on our robotic TRUS system and on the da41.3. Thesis Outline and ContributionsVinci system with a robotic drop-in ultrasound system. Details of the pro-posed remote palpation technique, its implementation, experimental resultson phantom, ex-vivo and clinical human studies are explained in this chap-ter.Chapter 6: Multi-parametric 3D Quantitative Ultrasound Vibro-Elastography Imaging for Detecting Palpable Prostate TumorsThis chapter describes a system for detecting dominant prostate tumors,based on a combination of features extracted from a novel multi-parametricquantitative ultrasound elastography technique. The quantitative elastogra-phy data acquisition system used to perform this study is built on top of therobotic TRUS system explained in chapter 2. This chapter will also explainthe details of our data collection in the operating room on 10 patients, aswell as a description of a propesed technique to integrate this system intothe da Vinci surgical system.Chapter 7: A Novel Force Feedback Control Structure for Robot-Assisted Laparoscopic Surgery This chapter describes a novel forcefeedback control framework to potentially enable haptic force feedback fortwo-handed tasks in teleoper-ated robot-assisted surgery. Details of theproposed control framework, the da Vinci Research Kit (dVRK) systemintegration in UBC RCL lab, implementation of the force feedback controlstructure on the dVRK system and user studies to evaluate the system.Chapter 8: Conclusion This chapter summarizes the goals, results andcontribution of the research, discusses potential applications of the results,and also suggests possible future directions for improving the work presentedin the thesis.Appendix A: Semi-Autonomous Ultrasound Guidance During Robot-Assisted Laparoscopic Surgery This chapter describes a semi-autonomousrobotic ultrasound guidance system for robot-assisted laparoscopic surgery,51.3. Thesis Outline and Contributionsimplemented on the da Vinci® surgical system using the dVRK system anda drop-in robotic ultrasound system.Appendix B: Clinical Study Protocols This chapter describes thedetails of the protocols of the clinical studies performed in this work.61.3. Thesis Outline and ContributionsFigure 1.1: Thesis outline.7Chapter 2Intraoperative TransrectalUltrasound Guidance forRobot-Assisted LaparoscopicProstatectomy2.1 IntroductionRobot-assisted laparoscopic radical prostatectomy (RALRP) using the daVinci Surgical System is now used to perform more than 80% of radicalprostatectomies in the United States [81]. While RALRP has enhanced thevisualization and dexterity over standard laparoscopic procedures, achieve-ment of the 3 main procedure outcomes of cancer control, urinary controland sexual function is still highly dependent on the surgeons intraopera-tive knowledge of prostate anatomy. It can be challenging for the surgeonto accurately identify critical structures such as the bladder neck, the neu-rovascular bundles and the apical prostate solely using visual cues providedby the da Vinci stereo vision system. Intraoperative imaging may aid sur-geons in localizing these structures.Transrectal ultrasound is the most common modality for imaging theprostate and is easy to integrate in a standard operating room. To be usefulthe TRUS transducer must be positioned and controlled by the surgeon inan intuitive way, and its images should be displayed at the correct locationrelative to the da Vinci vision system and instruments.Starting with the pioneering study by Ukimura et al. [108], research has82.1. Introductionshown the potential benefits of integrating TRUS into variants of RP, includ-ing the potential improvement in positive surgical margin rates, and urinaryand potency functional outcomes [78, 83, 107, 110]. Recently robotic TRUSsystems have been used for real-time guidance during RALRP [38, 41, 58].Hung et al. used a TRUS robot (ViKY®) for real-time monitoring of theprostate in a 10-patient trial [41]. Long et al. used the same TRUS robotto visualize real-time bladder neck dissection, NVB release and apical dis-section in a 5-patient study [58]. Han et al. used a custom-made roboticTRUS manipulator for improved visualization of the prostate gland, the sur-gical instruments and the NVB in 3 patients [38]. In all previous studiesthe TRUS manipulators have required manual readjustment using an addi-tional custom-made device such as a joystick or a foot pedal. Furthermore,3D TRUS was not registered to the da Vinci surgical system’s coordinateframes and hence, using 3D TRUS was not easy to interpret for the surgeonat the da Vinci console. Methods for registration of 3D ultrasound to theda Vinci surgical system has been previously described by a few researchgroups focusing on fusion of 3D ultrasound with stereoscopic video. Suchmethods were mainly based on electromagnetic (EM) or optical tracking sys-tems [19, 98]. Cheng et al also described a method based on photo-acousticmarkers to register da Vinci camera to 3D ultrasound [18]. While such pre-vious works show successful and accurate registration methods, there existno evidence of their clinical feasibility.In this chapter we describe a robotic TRUS guidance system calibratedto and controlled by the da Vinci Surgical System along with it’s first clinicalapplications. We also present a rapid and clinically feasible, intraoperativeregistration technique to calibrate 3D ultrasound to the da Vinci surgicalinstrument. The clinical study is a phase I-II clinical study showing that reg-istered robotic TRUS imaging can be deployed and used easily and safelyduring surgery with a short setup time, and that TRUS imaging can becontrolled in the registered coordinate system directly from within the sur-geon console. (Clinical trial registration: identifier:NCT02001597.)92.2. Materials and MethodsFigure 2.1: Robotic TRUS imaging setup installed on operating table underda Vinci arms as arranged in RALRP.2.2 Materials and Methods2.2.1 TRUS RobotOur research group recently presented a robotic system for TRUS imag-ing based on a modified brachytherapy stepper (Micro-Touch® 610-911)[4].The robot has three degrees of freedom and is designed compact enough tofit under the da Vinci patient side manipulators. This system, shown inFigure 2.1, was used for automatic and remote rotation (angle range ±45degrees) of the TRUS transducer during the procedure. The insertion depthof the transducer (insertion range ±60 mm) was also remotely controlled butonly manually, and it was limited to ±20 mm with a velocity of 0.5 mm persecond for safety during the human patient clinical studies. Ultrasound im-ages (2-dimensional) were captured using a Sonix ultrasound machine witha sagittal/transverse biplane TRUS transducer (Ultrasonix Medical Corp.,Richmond, British Columbia, Canada). Ultrasound volumes were acquiredby rotating the TRUS transducer about its long axis and collecting the 2-dimensional sagittal images at each angle increment. Software running onthe ultrasound console was responsible for directing the robot movements,ultrasound data acquisition and image processing.102.2. Materials and Methods2.2.2 TRUS Robot Calibration to the da VinciWe also recently described a method for using the TRUS robot to automat-ically track the da Vinci surgical instruments with the TRUS imaging plane[3]. This tracking allows the surgeon to control the TRUS imaging planeautomatically with the tip of a da Vinci surgical instrument. The track-ing method is based on a rigid registration of the kinematic frames of theda Vinci surgical manipulators and the robotic TRUS probe manipulator.This rigid registration can be accomplished by defining points on a tissuesurface (i.e., a boundary between air and tissue) in both coordinate frames.This requires the ability to localize surgical tool tips accurately in the 3-DTRUS data, since the da Vinci Application Programming Interface (API)[22] can be used to provide the locations of the tool tips in the da Vincikinematic frame. A schematic of the approach and the prostate anatomyduring RALRP is shown in Figure 2.2. In our implementations of the reg-istration method, da Vinci instrument tips have been localized manually inthe 3-D TRUS volumes. The approach requires the surgeon to press the tipof a da Vinci instrument (EndoWrist monopolar curved scissors) against theanterior surface of the prostate at 4 locations (Figure 2.3 and Figure 2.2).At each location the TRUS transducer was manually and remotely rotated,using a hand controller, to find the angle giving the highest intensity tooltip artifact in the B-mode image plane. Then the tool tip axial and lateralcoordinates were selected in this ultrasound plane. This process involvesscrolling through two-dimensional (2-D) slices of the volume, finding the 2-D slice that appears to contain the tool tip and selecting the approximatetip of each fiducial using a mouse. Chapter 3 will describes a method forautomatic da Vinci instrument tip localization in 3-D TRUS which is moreclinically realistic, since it would be less disruptive to the surgical workflow.The transformation relating the da Vinci coordinate system to that of theTRUS was computed by identifying the 4 tool tip locations with respect tothe TRUS and da Vinci coordinate systems [65]. The 3-dimensional locationof the da Vinci instrument tip was streamed out from the da Vinci systemusing its Application Programming Interface [22], with a reported accuracy112.2. Materials and MethodsOxyz TRUS O TRUS Oxyz TRUS (a) (b) TRUS transducer TRUS coordinate system da Vinci tool tip coordinate system 2D TRUS imaging plane Prostate Bladder Oxyz TRUS Oxyz TRUS Oxyz TRUS (a) (b) O TRUS O TRUS daVinci instrument Robotic TRUS  TRUS  sagittal image plane {𝑂 𝑑𝑉 , 𝐶  𝑑𝑉} {𝑂 𝑅𝑇 , 𝐶  𝑅𝑇} Roll angle 𝜃   +𝟑𝟎° 𝟎° −𝟏𝟓° Figure 3: Robotic TRUS to da Vinci instrument calibration and automatic tracking concepts. The registration is performed between da Vinci instrument tip coordinate frame 𝑂𝑑𝑉 , 𝐶  𝑑𝑉  and robotic TRUS coordinate frame 𝑂𝑅𝑇 , 𝐶  𝑅𝑇 . After the registration, the TRUS sagittal image automatically follows the da Vinci instrument tip.     (c) Figure 3: (a) TRUS to da Vinci instrumen  calibration concept, (b) Automatic instrument tip tracking in action. Figure 2.2: (a) TRUS to da Vinci instrument calibration concept. (b) au-tomatic instrument tip tracking in action. (c) robotic TRUS to da Vinciinstrument calibration and automatic tracking concepts. Registration isperformed between da Vinci instrument tip coordinate frame and roboticTRUS coordinate frame. After registration TRUS sagittal image automati-cally follows da Vinci instrument tip.122.3. In-vivo Studies(a)(b)Figure 2.3: (a) da Vinci instrument tip pressed at 4 locations across anteriorsurface of prostate for TRUS to da Vinci instrument calibration. (c) da Vinciscissor tips visible as hyperechoic focal point in B-mode images. Yellowarrows indicate location of instrument tip.better than 2 mm [53]. This 3-dimensional location provided the anglenecessary to rotate the TRUS transducer to automatically track the tip ofthe da Vinci instrument with the sagittal imaging plane (Figure 2.2). Thus,the surgeon at the console controlled the location of the TRUS imaging planesimply by moving the robotic instrument around, as if the TRUS image wasdragged to a given angle by the da Vinci instrument tip.2.3 In-vivo StudiesThis section describes the initial clinical evaluation of our ultrasound-basedguidance system for RALRP procedure. Both the TRUS robot system andthe calibration technique were tested and validated in lab experiments usingtissue phantoms before starting the clinical tests. Some of the phantomstudy results were published in [3] and some will be explained in Chapter 3.132.3. In-vivo Studies2.3.1 Canine StudyBefore starting the human patient clinical studies, we validated our methodsin a canine study approved by IACUC (Institutional Animal Care and UseCommittee) at Intuitive Surgical Inc. clinical research labs in Sunnyvale,California.Brief Research on the Animal ModelA live anesthetised animal is the most realistic, non-patient environmentthat has been utilized for decades as a method of educating, developing,and refining complex surgical techniques. Despite some anatomical differ-ences with the urinary tracts of humans, the porcine and canine model arethe most often used for various urologic procedures in the kidney, urethra,bladder, prostate and bowel [33]. The canine model is more often usedfor prostate surgery. The small and narrow pelvis in the canine model isan ideal environment to practice radical prostatectomy and robot-assistedlaparoscopic radical prostatectomy and it had been used in various relatedstudies and developments [32, 54, 85]. Canine neurovascular anatomy re-sembles that of humans and it is a suitable model in which to assess prosta-tectomy related erectile dysfunction. However, key differences, includingthe absence of discernible seminal vesicles, lateral placement of the pelvicplexus, the laterally placed dominant cavernous nerves and the circumfer-ential urethral distribution of cavernous nerves at the prostatic apex, mustbe considered during radical prostatectomy studies using the canine model[33]. According to the above brief review, the canine model seems to be thebest model for evaluation of our ultrasound-based image guidance systemfor da Vinci robot-assisted laparoscopic radical prostatectomy. TRUS imag-ing of the canine prostate has been done in various studies such as [29, 42].These previous studies show the feasibility of inserting the TRUS probe intothe dogs rectum and imaging its prostate.142.3. In-vivo StudiesStudy DescriptionA 10-month-old male hound weighing 27 kg was used in this study. Followinga lower bowel prep, the live anaesthetized animal was placed on the OR tablein a 40-degree Trendelenburg position. Before docking the da Vinci surgicalrobot, the TRUS robot was attached to the OR table using the MicroTouchBrachytherapy stabilizer passive arm (CIVCO Medical Solutions, Kalona,IA), which was adjusted for the TRUS to provide optimal transversal andsagittal images of the animal’s prostate as done in standard brachytherapyprocedures (Figure 2.4). A Sonix TABLET ultrasound machine (UltrasonixMedical Corp., Richmond, BC) with a bi-plane TRUS transducer was usedfor imaging. All TRUS volumes were captured using the 128-element 55mm long linear BPL9-5/55 array with transmit frequency of 6.6 MHz and(a)(b) (c)Figure 2.4: The clinical setup and TRUS images of the canine’s prostate:(a)TRUS robot attached to the OR table in Trendelenburg position withthe da Vinci robot docked to the table and da Vinci ports are placed as inRALRP. (b)Sagittal plane TRUS image of the prostate at elevational depthof 4 cm. (c)Transverse plane TRUS image.152.3. In-vivo Studiesimaging depth of 4.0 cm. They were acquired using an 80-degree rotarysweep about the probe axis, and contained 220 images at increments of 0.36degrees. Image capture time was 8.8 seconds per volume. The surgeonsplaced the da Vinci ports in the recommended pattern for RALRP, takinginto consideration the smaller size of the canine model. Three arms wereused for the procedure, with a Large Needle Driver, Prograsp and Mary-land Bi-polar forceps in the right, left and third arm respectively. A 12mm 0-degree stereo endoscope (3.8 mm disparity) was used throughout theprocedure. TilePro—was used in order for the surgeon to see the ultrasoundimage in the da Vinci console while performing the surgery. The surgeoncontinued with the RALRP procedure, with the TRUS transducer in po-sition, until the anterior surface of the prostate was visible in the stereocamera. A complete description of the animal study protocol is included inAppendix B.Registration Experiments and ResultsThe surgeon was asked to press the tool tip of a da Vinci instrument againstthe prostate surface while a full TRUS volume was being acquired (Fig-ure 2.5). The tool tip is visible as a hyperechoic focal point in the B-Modeimage. To manually find the tool tip, first the angle of the TRUS imagingplane is selected. Then the tool tip axial and lateral coordinates are selectedin this plane. The tip location relative to the TRUS coordinate system isobtained by transforming these cylindrical coordinates to Cartesian ones.The tool tip location relative to the robot coordinate system is also knownfrom the Research API provided by Intuitive Surgical, providing three con-straint equations for the homogeneous transformation relating the da Vincicoordinate system to that of the TRUS. Multiple constraints are obtainedby repeating the process. N = 12 different target locations and correspond-ing volumes were acquired. For n = 100 iterations, Nf = 4 point pairswere picked at random and a least squares problem was solved to find theregistration homogeneous transformation. The remaining Nt = N −Nf = 8162.3. In-vivo StudiesTable 2.1: 3D TRUS to da Vinci surgical tool registration accuracy (Manualtool tip localization in 3D TRUS). TRE and FRE are calculated for (n =100) iterations, (Nf = 4) tool tip points and (Nt = 8) target points with 4manual tool tip localization trials perfomed by 4 different users.TREA−P(mm)TRES−I(mm)TREM−L(mm)MeanTRE(mm)MeanFRE(mm)S 11.96 ±1.041.66 ±0.541.78 ±0.851.86 ±0.800.86 ±0.44S 21.93 ±0.521.62 ±0.581.72 ±0.701.76 ±0.610.97 ±0.97S 31.94 ±1.091.67 ±0.991.80 ±0.921.81 ±0.990.91 ±0.35S 42.19 ±1.312.07 ±1.172.07 ±0.972.11 ±1.151.02 ±0.38Average2.01 ±0.991.75 ±0.821.84 ±0.861.88 ±0.880.94 ±0.54target locations were used to calculate the TRE, defined as the error betweenthe location of the tool tips and the transformed points from the ultrasoundvolumes. To determine the inter-subject variation (ISV) in fiducial local-ization and analyze its effect on TRE, four different ultrasound users wereasked to localize the tool tip in each of the N = 12 B-mode TRUS volumeswe acquired. The TRE and Fiducial Registration Errors (FRE) in all threeanatomical directions and RMS values for each user, as well as the meanover all users, are reported in Table 2.1.Table 2.1 lists the mean values for TRE and FRE during TRUS robotto da Vinci instrument registration. A total of 12 TRUS volumes and daVinci API point-pairs were collected. Errors are represented in the anatom-ical frame of the patient (Anterior-Posterior (AP), Superior-Inferior (SI),Medial-Lateral (ML)). Mean values of FRE and TRE and their standarddeviations were calculated for each combination of (Nt,Nf ) for 100 itera-172.3. In-vivo Studies(a)(b)(c)Figure 2.5: (a) Camera image of the surgical site through the da Vinciconsole and spatial locations of the instrument tips scattered on the surfaceof the prostate. (b) The da Vinci instrument tip locations were spread onthe surface of the prostate to achieve an accurate registration across theentire prostate gland. (c) US images of the da Vinci instrument tip pressedon the anterior prostate surface at different points.182.3. In-vivo StudiesNf=3 Nf=4 Nf=5 Nf=6 Nf=7 Nf=800.511.522.533.54Number of tool tip points used for registrationError (mm)TRE and FRE values for different number of instument tip points for registration  FRETREFigure 2.6: TRE and FRE values for different number of tool tip pointsused for registration. As the number of fiducials increase, TRE decreases.We suggest using 6 fiducials in clinical applications.Table 2.2: Automatic da Vinci tool tip tracking accuracy.Tracking error (deg) Mean TRE (mm) TimeRegistration Trial 1 1.47 ± 0.83 1.78 ± 0.65 120sRegistration Trial 2 1.63 ± 1.22 2.00 ± 1.04 90sRegistration Trial 3 1.95 ± 1.28 2.11 ± 1.17 111sRegistration Trial 4 1.58 ± 1.63 1.83 ± 0.76 64sAverage 1.65 ± 1.24 1.93 ± 0.90 96stions and the results are plotted in Figure 2.6. As can be seen from thisfigure, as Nf increases, both the mean and the standard deviation of theTRE decreases. Based on this analysis, the number of fiducials suggestedfor this registration is Nf = 6.Registration Timing:To determine the ease with which the above registrations can be performed,we asked the surgeon to perform four timed registrations using four regis-tration points each. For each registration point, the tool tip location was192.3. In-vivo Studiesfound manually in the ultrasound volume. Often the surgeon would gentlymove the tool tip to confirm the correct tool tip location. After each regis-tration, the automatic tracking was activated and the surgeon was asked tomove the tool tip to an additional 10 points on the surface of the prostate.For each location, the corresponding TRUS angle was recorded, then thetracking was temporarily deactivated and the points were located manuallyby adjusting the TRUS angle. The error in this measurement is shown inTable 2.2.The tracking accuracy for the four timed registration trials are reportedin Table 2.2. TRE values were also calculated for each registration. Allregistrations were completed in under 2 minutes with an average registra-tion time of 96 seconds. Throughout the registration experiments and thesurgery, TRUS images were streamed into the da Vinci console for real-timeguidance. Figure 2.7 shows the TilePro—and camera images inside the daVinci console, when the automatic tool tracking is activated and the TRUSimage follows the da Vinci tool tip.2.3.2 Discussion of the ResultsIn this set of experiments, we tested and validated the intraoperative use ofa robotic TRUS manipulator for RALRP procedures. The TRUS robot is(a) (b) (c)Figure 2.7: TilePro—images inside the surgeon console while the automatictool tracking is activated. The da Vinci instrument tip is visible in bothcamera and ultrasound images.202.3. In-vivo Studiesbased on a small modification to a standard brachytherapy stabilizer which isavailable in almost any hospital where brachytherapy is performed. Hospitalstaff are familiar with the set-up and positioning of the transducer on thestabilizer with respect to the patient.During the TRUS to da Vinci tool registration, a TRE of 1.88 ± 0.88mm was achieved. This is on par with the results from [5] when using aPVC prostate phantom. It is pointed out by Ukimura et al. [109] that themean distance between the NVB and the lateral edge of the prostate rangedfrom 1.9 ± 0.8 mm at the prostate apex, to 2.5 ± 0.8 mm at the base. Thisis suggestive of the required accuracy of a guidance system since one majoraspect of the system is to accurately localize the NVB. Currently the errorin our TRUS to camera registration is slightly larger, but the error betweenthe da Vinci tools and the TRUS is within the range reported in [109].For this registration approach, some of the registration error may also bedue to the limited localization accuracy of the fiducials within the US images.While subjects were instructed on the best way of picking the fiducial edgesas described in [36], they had higher variance in localizing the fiducials thanthe automatic method. The use of an automated algorithm would also meanthat no additional personnel would be needed in the OR in order for thetracking to be activated. For the TRUS to da Vinci tool registration error,another contributing factor is the tool tip localization error from the daVinci API, which has been reported to be within 2mm. Another source oferror is instrument shaft deflection, as pointed out in [100].Timing results have shown that the da Vinci instrument to TRUS regis-tration could be completed very quickly and would be valid throughout thesurgery since neither the TRUS nor the da Vinci coordinate systems willbe moving. We determined that using six tool tip positions gives the bestTRE with minimal added benefit derived from further measurements. Thiswould increase registration time by approximately 20 seconds. Camera toTRUS registration tools should be similar, not counting the time requiredfor camera calibration.Although the canine model was chosen, there are key differences fromhumans which actually made the study somewhat more difficult. Positioning212.4. Human Patient Studieswith a human patient does not usually put extensive pressure on the distalend of the transducer, but in the canine case, there was a larger amount offorce on the transducer which could cause errors in TRUS rotation duringTRUS volume acquisition and also in fiducial localization in TRUS images.2.3.3 Animal Study ConclusionsWe have presented the validation of our intraoperative registration methodthat can be used during RALRP for image guidance and surgical navigation.Using the kinematics of the robot, we were able to register the da Vincicoordinate system with that of the TRUS robot. This was achieved quicklyand efficiently with surgeons new to this concept. All registration errors werewithin the scope of the clinical setting and the constraints of the ultrasoundimaging system. Surgeons even suggested approaches on how to distributethe registration points (2 points at the prostate base, 2 points at mid-glandand 2 points at the apex) to make the process more efficient and maintainregistration accuracy across the prostate. We have demonstrated that theseregistration methods work effectively in an in-vivo environment. The daVinci kinematic registration was ready for clinical testing after this set ofexperiments and we submitted our application to human ethics and startedpatient studies.2.4 Human Patient StudiesAfter an evaluation in a canine model and approval of our clinical researchethics application (H11-02267), 20 patients were enrolled in a clinical study.2.4.1 Patient PopulationA total of 20 patients with clinically organ confined prostate cancer under-going RALRP at our institution agreed to participate in this study betweenMarch and November 2013. Median patient age was 63 years (range 52 to70) and median baseline prostate specific antigen was 5.9 ng/ml (range 0.84to 36.4). Patient demographic and preoperative data are listed in Table Human Patient StudiesThe study was approved by the University of British Columbia Clinical Re-search Ethics Board.Table 2.3: Demographic and preoperative data.No. of patients 20Age, years ; median (range) 63 (52-67)Preoperative PSA, ng/ml ; median (range) 5.9 (0.84-36.4)Preoperative Gleason score (index lesion), no.(%):3+3=6 5 (25)3+4=7 8 (40)4+3=7 2(10)4+4=8 2 (10)3+5=8 1(5)4+5=9 2 (10)Clinical stage, no. (%):T1c 8 (40)T2a 5 (25)T2b 3 (15)T3a 3 (15)T3b 1 (5)IPSS value ; median (range) 7 (1-24)Prostate volume, cc ; median (range) 33.5 (11-55)Nerve sparing plan, no. (%)Bilateral nerve sparing 10 (50)Unilateral nerve sparing 8 (40)No nerve sparing 2 (10)PSA: Prostate-Specific Antigen ; IPSS: International Prostate SymptomScore2.4.2 Procedure DescriptionTwo surgeons performed the procedures for this study using a da VinciS system. All patients undergoing robotic prostatectomy received a Fleet(R) enema the evening before the surgery. Our robotic TRUS system wasattached to the foot of the operating table and was placed to provide opti-mal transverse/sagittal images of the prostate as in standard brachytherapy232.4. Human Patient Studiesprostate volume studies. The TRUS sagittal plane was placed underneaththe prostate midline so that the prostate could be imaged from the baseto the apex. The system configuration setup inside the OR and during theprocedure are shown in Figure 2.9 and Figure 2.8. In the majority of cases,standard dissection of the retropubic space was performed until the anteriorsurface of the prostate was visible to the surgeon and the endopelvic fasciaincised. In a few early cases the Montsouris approach (initial posterior dis-section of the prostate and seminal vesicles) was used, with some negativeeffects on subsequent TRUS image quality due to insufflation gas posteriorto the prostate.The calibration procedure was performed at this stage with the 4 in-Figure 2.8: TRUS guidance system components and connection in the op-erating room.242.4. Human Patient Studiesstrument tip locations spread across both sides of the dorsal prostate. Theautomatic TRUS tracking of the da Vinci working instrument tip was en-abled and used during several points of the procedure, including transectingthe anterior and posterior bladder neck, lifting the prostate off the rectumand dissecting the prostatic apex. In selected cases the TRUS robot was alsoused for placement of the suture through the DVC, and for identification ofthe NVBs and seminal vesicles. As in the canine study we used TilePro—todisplay real-time TRUS images on the da Vinci console.The primary aim of this study was to investigate the feasibility and safetyof real-time TRUS tracking and imaging in RALRP. The secondary aim wasto investigate whether the surgeons found this technology useful at criticalstages of RALRP.2.4.3 ResultsIntraoperative TRUS with automatic tracking was performed successfullyin all 20 patients and there were no complications related to the addition ofthis procedure. We did not note any postoperative complications, includinginfection. Specifically, patients did not complain of anal pain or bleeding.The TRUS transducer was in a condom with ultrasound coupling gel and itwas free to rotate without resistance.The set-up of the TRUS robot at the foot of the bed was performedin a median of 7 minutes (range 5 to 14) with minimal delay to the startof surgery. Indeed the TRUS robot could be positioned while the anesthe-siologists gained additional vascular access and the patient was secured tothe table with arms tucked. The calibration process was performed afterincision of the endopelvic fascia and could be completed in approximately2 minutes. Table 2.4 lists the critical steps in RALRP where we judged theTRUS images to be useful. At each step it was noted whether the surgeonused or commented on the real-time TRUS images.At the anterior bladder neck TRUS images can be helpful in confirmingthe plane of dissection between the bladder and the prostate (Figure 2.10).Experienced surgeons use the shape of the prostate and the distinctive ap-252.4. Human Patient StudiesFigure 2.9: (a) TRUS robot attached to operating table in Trendelenburgposition. (b) TRUS transducer and robot attached to foot of operating tableusing Micro-Touch brachytherapy stabilizer passive arm. (c) OR setup ofTRUS robot and da Vinci system.262.4. Human Patient StudiesTable 2.4: Intraoperative TRUS results.Setup time for TRUS robot, min ; median (range) 7 (5-14)TRUS to da Vinci calibration time, sec ; median(range)120 (110-214)No. of cases in which TRUS was foundhelpful in:DVC suture 2 of 20 (10%)Anterior bladder-neck dissection 15 of 20 (75%)Posterior bladder-neck dissection 13 of 20 (65%)Seminal vesicle removal 9 of 20 (45%)Close to rectal wall 10 of 20 (50%)NVB preservation 1 of 20 (5%)Apical dissection 13 of 20 (65%)No. of intraoperative complications 0Surgical margins1, no. (%)Negative 17 (85)Positive2mm left anterior 1 (5)2mm right posterior 1 (5)2.3mm right inferior and apex 1 (5)1TRUS was used for all of the listed tasks during each case,but it was noted useful by the surgeon in the specified number of cases.pearance of the relatively bloodless plane as the detrusor muscle fibers dis-sect off the prostate. This is one of the key steps in the learning curve of thisprocedure that may be accelerated with TRUS images. The TRUS robotalso facilitates bladder neck transection by allowing early recognition of anintravesical median lobe.The posterior bladder neck is a common site of positive margins as itcan be difficult to accurately dissect a plane that stays out of the prostateand avoids buttonholing the bladder [9].In Figure 2.10b, the base of theprostate is clearly delineated and the surgeon can be seen dissecting towardthe seminal vesicles.TRUS imaging may be helpful in ascertaining the extent of the dissectionrequired to reach the tips of the seminal vesicles (Figure 2.11). If the seminalvesicles are dissected out from a posterior access before dropping the bladder272.4. Human Patient Studies(a)(b)Figure 2.10: TRUS use at anterior (A) and posterior (B) bladder neck dis-section. Yellow arrows indicate location of instrument tip.282.4. Human Patient StudiesFigure 2.11: TRUS use at removal of seminal vesicles. Yellow arrows indicatelocation of instrument tip.from the anterior abdominal wall, the insufflation gas space that is createdimpairs subsequent TRUS imaging of the bladder neck.One of the rare but major potential complications of RP is rectal injurythat is usually the result of a difficult posterior dissection. If there is residualinflammation or there are adhesions from the prostate biopsy, for example,the anterior wall of the rectum can easily be drawn up with the anteriorretraction of the prostate. With the TRUS robot automatically trackingthe tip of the surgeons operating instrument, the rectal wall can be easilyvisualized. In Figure 2.12(a) the rectum can be seen being tented up, andthe plane between the rectum and the prostate is clearly demarcated.In open RP and RALRP it can be difficult to assess the distal-most ex-tent of the prostatic apex, especially posteriorly, and this is a common siteof positive surgical margins[95]. TRUS imaging seems promising in delineat-ing the prostatic apex. For example, in Figure 2.12(b) the hyperechoic tipof the scissors can be seen dissecting just beyond the prostatic apex whilepreserving the maximal amount of urethra.292.4. Human Patient Studies(a)(b)Figure 2.12: TRUS use in dissection at rectal wall (A) and DVC suture (B).Yellow arrows indicate location of instrument tip.302.4. Human Patient StudiesFigure 2.13(a), shows the NVBs running along the posterior lateral as-pect of the prostate close to the rectum, visualized by TRUS in color Dopplermode. After removal of the specimen, preservation of the NVBs could beverified using color Doppler in appropriate cases (Figure 2.13(b)).2.4.4 DiscussionWe have demonstrated for the first time in patients an intraoperative roboticTRUS system that precisely tracks the movements of the da Vinci instru-ments with an accuracy of the same order as the da Vinci robot instrumentpositioning. This system takes little time to install, is easy to use and ispotentially helpful to surgeons in completing specific steps of the surgery, asnoted in Table 2.4.A concern before the study was that the TRUS probe would push therectum anteriorly and into the surgical field, thereby narrowing the spacearound the prostate and compromising instrument maneuverability, as wellas increasing the risk of rectal injury. However, by keeping the TRUS probeparallel to the operating bed, the presence of the TRUS probe in the rectumwas not even noticeable to the surgeon and did not interfere with any partof the prostatectomy.Some surgeons use the Montsouris approach to RALRP, which involvesposterior dissection of the prostate and seminal vesicles before dissecting thespace of Retzius and the prostate anteriorly [35]. We found that this tech-nique introduced gas into the space posterior to the prostate and interferedwith optimal TRUS image quality for subsequent stages of the procedure.However, we found it particularly easy with ultrasound guidance to makethe transition from routinely using the Montsouris approach to an anteriorapproach, as the ultrasound clearly delineated the bladder neck and theunderlying seminal vesicles.While the use of the TilePro—feature did reduce the main video imagesize, it only needed to be on during brief TRUS imaging sessions, at pointsin the procedure when the anatomy needed to be confirmed.The TRUS robot is based on a small modification to a standard brachyther-312.4. Human Patient StudiesFigure 2.13: TRUS use in dissection at prostatic apex (A) and Dopplerevaluation of NVBs (B). Yellow arrows indicate location of instrument tip.322.4. Human Patient Studiesapy stabilizer that is widely available and familiar to hospital staff. TheTRUS automatic positioning approach has safety features built into it. In-deed, if the TRUS tracking is working with the required accuracy, touchingthe tissue with the da Vinci tool tip will provide a confirmatory imagingartefact.While high volume surgeons are less likely to benefit from the use ofTRUS, the majority of RALRPs in the United States are performed by sur-geons who do fewer than 12 such procedures a year [27]. These surgeons,as well as surgeons in training and those early in their learning curve, maybenefit from the added information provided by the TRUS robot. Our qual-itative evaluation indicates that the greatest benefit would be expected fordissection of the bladder neck, the prostatic apex and the plane posteriorto the prostate with the aim of reducing positive margins, optimizing post-operative continence and reducing the risk of rare but potentially seriousrectal injuries, respectively. With higher resolution ultrasound systems andthe use of flow imaging, the localization of the NVB may become possible.Thus, intraoperative TRUS might be able to contribute to all 3 aspects ofthe trifecta of prostate cancer surgery, namely cancer control, continenceand potency.In the future we anticipate enhancing the usefulness of our system. Inparticular, we believe that image overlay with TRUS to preoperative mag-netic resonance imaging registration will be advantageous as it will allowthe surgeon to be guided intraoperatively by the preoperative magnetic res-onance imaging. While there is currently debate about the true advantagesof RALRP compared to other RP approaches, such technological advance-ments with integration of imaging into the robotic surgeons repertoire havethe potential to clearly delineate the superiority of robotic surgery in thefuture.33Chapter 3A System for MR-GuidanceDuring Robot-AssistedLaparoscopic RadicalProstatectomyIn this chapter, we describe a new ultrasound and magnetic resonance imageguidance system for robot assisted radical prostatectomy and its first use inpatients. This system integrates previously developed and new componentsand presents to the surgeon preoperative magnetic resonance images (MRI)registered to real-time 2D ultrasound to inform the surgeon of anatomy andcancer location. At the start of surgery, a trans-rectal ultrasound (TRUS)is manually positioned for prostate imaging using a standard brachytherapystepper. When the anterior prostate surface is exposed, the TRUS, whichcan be rotated under computer control, is registered to one of the da Vincipatient-side manipulators by recognizing the tip of the da Vinci instrumentat multiple locations on the tissue surface. A 3D TRUS volume is then taken,which is segmented semi-automatically. A segmentation-based, biomechan-ically regularized deformable registration algorithm is used to register the3D TRUS image to preoperatively acquired and annotated T2-weighted im-ages, which are deformed to the patient. MRI and TRUS images can thenbe pointed at and examined by the surgeon at the da Vinci console. Weoutline the approaches used and present our experience with the system inthe first two patients. While this work is preliminary, the feasibility of fusedMRI and TRUS during radical prostatectomy has not been demonstrated343.1. Introductionbefore. Given the significant rates of positive surgical margins still reportedin the literature, such a system has potentially significant clinical benefits.3.1 IntroductionA standard treatment of prostate cancer is radical prostatectomy, or thesurgical removal of the prostate gland, via open, laparoscopic or robot as-sisted surgery. Studies seem to indicate that robot-assisted RP has the bestoutcomes, yet the rates of positive surgical margins (cancer left behind aftersurgery) still range between 9% and 30%, depending on the center [26].The goal of the surgery to remove all the prostate and the cancer within itand extending from it, while attempting to spare as much as possible theadjacent critical structure responsible for continence (sphincter muscle) andpotency (neurovascular bundles and the cavernosal nerves). The main rea-son why the best trade off between achieving oncological success (cancer re-moval) and functional success (continence and potency) cannot be achievedis the inability to localiz, intra-opratively, the location and extent of can-cer. Advances in magnetic resonance imaging (MRI) may provide spatiallylocalized information to fill this void and aid surgical planning, particularlyfor robotic surgeons [99]. The combination of conventional anatomical MRIand functional MR sequences, known as multiparametric MRI (mp-MRI),is emerging as an accurate tool for identifying clinically relevant tumours.This is due to the additional information provided by functional MRI se-quences, such as diffusion-weighted (DW) MRI, dynamic contrast-enhanced(DCE) MRI, which significantly improve the ability of mp-MRI to predictthe behaviour of tumours [43]. mp-MRI is becoming an integral diagnos-tic tool in pre-operative planning of RALRP; specifically for prediction ofpathologic stage in extracapsular extension (ECE), neuro-vascular bundle(NVB) and seminal vesicle invasion. This capability of mp-MRI to gener-ate the most accurate characterization of prostate cancer [50], has led tothe development of methods for MRI-guided treatment, mainly biopsy andbrachytherapy [102].Direct MRI-guided methods have been reported for prostate biopsy and353.1. Introductionbrachytherapy [96], but intraoperative MRI is still cumbersome, time con-suming, resource costly and not widely used. Cognitive fusion, in which theclinician estimates the lesion’s location in the intraoperative TRUS based ona preoperative MRI, varies greatly with expertise. A more feasible approachto allow integration of MRI data in the operating room involves registrationof the preoperative MRI to the intraoperative TRUS, and visualization ofthe corresponding images to assist the clinician during treatment. Previ-ously, such an approach was successfully demonstrated for prostate biopsy[37, 61, 63, 82, 106], and for prostate brachytherapy [87]. However, thereexists no report on integration of such an approach for real-time surgicalguidance using the da Vinci surgical system which is currently being usedto perform more than 80% of radical prostatectomies in North America.In this work, we present a novel MR-guidance system for the da Vinciwhich involves an intraoperative segmentation-based MR-TRUS registrationmethod that is integrated into a clinically used robotic TRUS imaging sys-tem which in turn can be registered to the da Vinci system’s coordinateframe. In this way a 3D MR volume and the preoperative surgical plancan be mapped to the da Vinci coordinate frame. The surgical instrumentcan then be visualized, in real-time, with respect to the pre-operative MRvolume. In addition, since the TRUS imaging system is robotic, it can trackthe tip of a da Vinci surgical instrument automatically, and is registered tothe pre-operative MR volume, the surgical instrument itself can be used asan intuitive and easy to use control device for the surgeon to manipulateboth MR and TRUS images in real-time during the procedure. The systemwas initially tested and validated on a prostate phantom in a lab setting.Next, it was used with a clinical da Vinci surgical system inside a roboticoperating room and tested on two patients undergoing RALRP using theda Vinci. Both patients have undergone clinical pre-operative MR imagingfor diagnosis. The description of the system components and methods alongwith the clinical study results and discussions are presented in this chapter.363.2. Materials and Methods3.2 Materials and MethodsThe components of our image-guided system are illustrated in Figure 3.1.The system comprises an ultrasound system with a motorized TRUS trans-ducer mounted on a brachytherapy setup, an external PC used for imageregistration and display on the da Vinci console, and registration and track-ing software. These components will be described next.3.2.1 TRUS Imaging SystemA previously designed and clinically used robotic TRUS manipulator [64]was used for automatic and remote rotation (angle range ±45 degrees) ofthe TRUS transducer during the procedure [64]. The sagittal TRUS imagingplane is automatically repositioned using the robot so that a 2D ultrasoundimage continuously contains the tip of a specified da Vinci surgical ma-nipulator. Automatic instrument tracking is achieved by means of a rapidand clinically feasible intraoperative registration technique that solves forthe rigid homogeneous transformation between the da vinci and the TRUSrobot coordinate systems (Figure 3.1). The registration involves definingpoints on tissue surface using the da Vinci instrument tip and accurate andautomatic localization of the points in both da Vinci and Robotic TRUScoordinate frames [3]. This tracking method allows the surgeon at the con-sole to control the TRUS imaging plane automatically with the tip of a daVinci surgical instrument with an accuracy of less than 2 mm [65].A BK ultrasound machine (BK Medical, Herlev, Denmark) with a 88484-12 MHz biplane transducer was used for imaging the prostate. Raw inphase quadrature (IQ) data was captured at 43.07 Hz sampling rate andsaved into an external PC through a DALSA Xcelera-CL PX4 Full framegrabber card (Teledyne DALSA, Waterloo, ON). The TRUS robot was usedfor 3D TRUS volume acquisition by automatically controlling the rotationangle of the TRUS transducer and saving the location information of eachimage. All TRUS volumes were captured using the 214-element 6 cm longsagittal array with a transmit frequency of 9 MHz and an imaging depth of5.6 cm. Volumes were obtained using a 90-degree rotary sweep about the373.2. Materials and Methodsprobe axis with images acquired at increments of 0.2 degrees. The imagecapture time was 45 seconds per volume.Figure 3.1: A schematic of the system. da Vinci {OdV , CdV } and Robotic-TRUS {ORT , CRT } coordinate frames are registered in order to enableautomatic tracking of the instrument with ultrasound. {ORT , CRT } and{OUS , CUS} is computed using ultrasound calibration. Images are thenconverted from cylindrical coordinates to cartesian spatial coordinates forMR-TRUS registration.383.2. Materials and Methods3.2.2 TRUS-MR RegistrationThe prostate gland on each transverse slice in the preoperative T2w MRvolume is segmented manually by an expert radiologist before surgery. Afterinterpolating the intraoperative TRUS B-mode images into a 3D grid inorder to obtain transverse slices from the sagittal volume, we employed areal-time semi-automatic algorithm for 3-D segmentation of the prostatein the ultrasound volume. This algorithm, described in [59], has beenroutinely employed during brachytherapy, and found to be a fast, consistentand accurate tool for the delineation of the prostate gland in TRUS images.Based on the segmented surfaces of the prostate in the TRUS and MRvolumes, we construct the binary volumes. We may then register the twobinary volumes and apply the resulting displacement map back to the MRvolume. First, the two binary volumes are rigidly aligned (and scaled) usingthe principal axes transformation. This is a fast, one-step registration thatwas found to provide a good initial alignment for the followed deformableregistration. To this end, the binary volumes are treated as pdfs, and thecorresponding centers of mass, and covariances are calculated. The alignedMR binary volume can be then computed using a linear coordinate transfor-mation. Next we do the deformable registration using a method developedin our lab [70] .The proposed method was tested on n = 6 data sets of RALRP patientswho underwent preoperative MR and intra-operative US. The MR volumeswere acquired by a 3-Tesla system (Achieva 3.0T, Philips, The Netherlands)using a standard 6-channel cardiac coil with acceleration factor (SENSE)2. Both of the MR and TRUS transverse volumes were segmented andthe surfaces of the prostate were generated with medium resolution andsmoothing strength settings. The urethra on both modalities was segmentedmanually for evaluation as well. In order to maintain consistent processingtimes, we ran the deformable registration algorithm a fixed number of 30iterations that takes about 184 sec, and was found to converge sufficiently.Including the times for the semi-automated segmentation (46± 15sec), thetotal run-time of the entire segmentation-based registration process is about393.2. Materials and Methods4 minutes. We evaluated the registration performance using the volumeoverlap (VO), in the sense of Dice’s coefficient between the MR and theTRUS volumes after rigid and deformable registrations. The mean distancebetween splines on both modalities that were fitted through the center pointsof the urethra was also measured for the rigid and deformable registeredvolumes. The mean VO was computed to be (97.7± 0.3%) and mean errorwas computed to be (1.44± 0.42mm).3.2.3 In-vivo Patient Studies and ResultsTwo patients (ages 55 and 72; prostate specific antigen 13.5 and 28.5 ng/ml)with clinically organ confined prostate cancer undergoing RALRP agreed toparticipate in this institutional review board approved study. The maincomponents and configuration of our system during this clinical study in-side the operating room are shown in Figure 3.2. Both patients underwentpreoperative mp-MR and a radiologist was asked to examine their MR vol-ume and segment the prostate and the lesions. The lesions were identifiedin DCE-MR images and then marked on T2w images.Once the patient is placed on the operating table and before dockingthe da Vinci robot, we attached the Robotic-TRUS system to the foot ofthe operating table and placed the TRUS transducer to provide optimaltransverse/sagittal images of the patient’s prostate as performed in stan-dard brachytherapy volume studies. After docking the da Vinci system andstart of the procedure, once the anterior surface of the prostate is visibleto the surgeon, we performed the TRUS to da Vinci calibration to enableautomatic tracking of the surgical instrument with ultrasound. The cali-bration process was performed after incision of the endopelvic fascia with4 instrument tip locations spread across both sides of the dorsal prostateand could be completed in approximately 2 minutes. Before activating theautomatic instrument tracking control mode, we acquired 3D TRUS vol-umes (45 seconds each) to be used by the MR-TRUS registration algorithm.The TRUS volume used for the registration was the one that was acquiredright before the part of the procedure when the surgeon wanted to use MR403.2. Materials and Methods(a)(b)Figure 3.2: (a) MR-guidance system components, (b) The operating roomscenario.413.2. Materials and MethodsTable 3.1: Table of patient study results. treg is the duration of each reg-istration process, Nr is the number of times registration was repeated forthat patient, Nf is the number of fiducial points selected for instrumentregistration, and tT is the total duration of TilePro usage during each case.MR-TRUS reg. TRUS-dV reg.tT (min)VO (%) ereg (mm) treg (s) Nr Nf treg (s)P1 97.7± 0.3 1.5± 0.3 240 1 4 96 16p2 96.6± 0.2 1.8± 0.3 235 2 4 95 28images to localize tumors. This was done to make sure that prostate defor-mations were minimal and to achieve a more accurate registration resultsand deformation field.After the tracking was activated, the surgeon could examine the prostateanatomy and tumor locations by moving the registered surgical instrumentaround, slightly placing it on the tissue surface in the area of interest and lo-calizing the instrument tip with respect to anatomy seen in real-time TRUSimages and then use it’s corresponding MR slice. As it is shown in Fig-ure 3.3, both real-time TRUS and the corresponding MR images are be-ing sent to the da Vinci console and displayed to the surgeon using theTilePro—feature of the da Vinci system. The surgical view is divided intothree tiles with adjustable sizes, one for endoscopic view, one for real-timeTRUS and one for corresponding deformed MR slice. Our graphical userinterface for MR imaging display (shown in Figure 3.3) shows instrumenttip location in cylindrical coordinate frame of the TRUS system (roll angle)with a superimposed red cursor line on the 2D deformed MR slice. Thiscursor line was used by the surgeon for localization of the registered surgicalinstrument with respect to the segmented and annotated lesions displayedin MR images. Snapshots of the surgeon console images at the stages whenthe MR-guidance system was being used to localize tumors are shown inFigure 6.6. A summary of the registration results for both patients is listedin Table 3.1.423.3. Discussions and ConclusionsImaging interface inside the daVinci®S surgeon console ( Tilepro™ ) Registered transverse  MR slice Sagittal TRUS image Surgeon console of the daVinci®S system Marked up Tumor region Instrument tip Figure 3.3: Real-time imaging system interface using the da Vinci SiTilePro—feature.3.3 Discussions and ConclusionsThe performed surgical procedures were the first in which a surgeon was ableto see registered MR images while performing the operation and refining thesurgical planes to achieve a better surgical margin and functional outcome.During the first case, a lesion on the right side, stretching from the mid-glandstretching anteriorly and superiorly was seen on MR imaging system (shownin Figure 6.6). Based on this information, the surgeon attempted to leaveas much of the nerves intact on the left (“high” or more anterior dissectionapproach in order to get closer to the prostate) and on the right, forget aboutsparing or try to get closer to the prostate only posteriorly. The surgeondecided to perform a bilateral nerve sparing but more conservatively becauseof the anterior lesion on the right seen in MR images. If the lesion wasposterior, then he would not have done nerve sparing on that side. Duringthe second case, a lesion on left posterior side was seen on MR images andsurgeon avoided sparing the nerve on that side. The intra-operative MRimaging allows surgeons to fine tune based on anatomy. without MRI, theywould just have a certain number and proportion of cores involved on one433.3. Discussions and Conclusionsside or the other, but no other info on location - hence a guessing gameat best. Given the number of papers on the “best” surgical approach, thisMR-guidance system could be a powerful tool to trade off positive surgicalmargins against potency.Intraoperative TRUS to da Vinci and TRUS-MR registrations were per-formed successfully for 2 patients. The imaging was examined at the anteriorsurface of the prostate, after the surgeon placed the dorsal venus complex(DVC) suture at the prostate apex, and before bladder-neck dissection forlocalization of the tumor and for finding the best surgical plane to achieveboth better oncological and functional outcomes before performing NVB re-lease. TRUS volume acquisition and MR-TRUS registration processes wereredone before any stage of the surgery when the surgeon wanted to use thesystem. The mean volume acquisition and registration time was 235 sec-onds. A fresh TRUS volume would deform the preoperative MR imagesbased on the current deformation state of prostate gland.It is important and intuitive to the surgeon to have control over theimaging system. Previous work showed that surgeon-assistant coordinationis inefficient. The other proposed method to interact with the imaging sys-tem would be adding another control device to the daVinci console. Devicessuch as another foot-pedal or a hand controller was added to the daVinciconsole. Input device such as the concept of “master as mice on da Vinciplatform to change viewpoints [55] was also presented. The advantage ofour system is that once we perform our 2 minutes intraoperative calibration,the surgeon can use the daVinci surgical instrument to move through the 3Dvolume of both TRUS and MRI and manipulate 3D data. The registrationmoves the 3D images coordinates to the davinci coordinate system. Hence,surgeon could easily move the tool tip around and find the location of theinstrument tip with respenct to 3D vision, 3D TRUS and 3D MRI.This chapter demonstrates a preliminary study on a framework basedon a combination of an MR-TRUS registration and TRUS to da Vinci reg-istration. We believe that the MR-TRUS registration appraoch could beimproved in future to better deal with deformations in the prostate glandduring the procedure. Furthermore, in order to achieve more consistent and443.3. Discussions and ConclusionsFigure 3.4: Clinical study results. Both deformed MR and realtime TRUSare shown to the surgeon at the console along with the surgical endoscopicimaging.thorough registration accuracy results, target registration error should becomputed for all regions of the prostate gland. Such an study requires iden-tification of common anatomical landmarks in both 3D TRUS and 3D MRvolumes of the prostate. Another alternative to perform such an analysis isto perform an animal study during which clearly visible fiducials could beinjected into the prostate gland and imaged in both modalities.453.3. Discussions and ConclusionsFigure 3.5: Surgical tool motion in medial-lateral direction and the corre-sponding TRUS and MR images.Figure 3.6: View of the surgical console when both MRI and TRUS are inuse.46Chapter 4Automatic 3-D TransrectalUltrasound to the da VinciSurgical System Registration4.1 IntroductionGiven the limited field of view of the surgical site in RALRP, several groupshave proposed the integration of transrectal ultrasound (TRUS) imagingin the surgical workflow to assist with accurate resection of the prostateand the sparing of the neurovascular bundles (NVBs). In chapter 2, weintroduced a robotic TRUS manipulator and a method for automaticallytracking da Vinci surgical instruments with the TRUS imaging plane, inorder to facilitate the integration of intraoperative TRUS in RALRP. Rapidand automatic registration of the kinematic frames of the daVinci surgicalsystem and the robotic TRUS probe manipulator is a critical componentof the instrument tracking system. In this chapter, we describe a fullyautomatic registration technique based on automatic 3-D TRUS localizationof robot instrument tips pressed against the airtissue boundary anterior tothe prostate.In the manual implementations of the registration method, da Vinciinstrument tips or other surface fiducials have been localized manually in the3-D TRUS volumes. This process involves scrolling through two-dimensional(2-D) slices of the volume, finding the 2-D slice that appears to contain thetool tip and selecting the approximate tip of each fiducial using a mouse.While the overall registration method has been shown to work well [3], the474.1. Introductionmanual fiducial selection procedure is time consuming and the interpretationof the fiducial centers in a sequence of 2-D TRUS images in the volume issubjective and varies from one user to another. A method for automaticallylocalizing the surface fiducials is more clinically realistic, since it would beless disruptive to the surgical workflow. This chapter describes a method forautomatic da Vinci instrument tip localization in 3-D TRUS. The detectionalgorithm could also be applied to other types of surface fiducials, such as thesteel spherical fiducials we previously used for registration of 3-D ultrasoundto the da Vinci stereoscopic camera [114].Localization and real-time tracking of surgical tools in 3-D ultrasoundhas been addressed in previous studies [24, 30, 45]. In our work, da Vincisurgical instruments are working within the carbon dioxide filled abdominalcavity, as depicted in Figure 4.1, and are not generally visible by ultrasound.As a result, the instrument tip must be pressed against the tissue surfaceto become visible, and allow the registration to occur. Previous work hasdemonstrated real-time intraoperative registration of 3-D US to surgicalrobot coordinate frames, but in these cases, the instruments were constantlyvisible in the ultrasound [72, 73]. To the best of our knowledge, there areno reports of automatic instrument tip or surface fiducial localization in 3-Dultrasound.Figure 4.1: da Vinci surgical instruments and the prostate surface in aclinical scenario during the RALRP procedure.484.1. IntroductionThe localization problem can be divided into two sub-problems: 1) au-tomatically detecting the presence of the instrument tip in the ultrasoundvolume and finding the 2-D plane that best contains it, and 2) automati-cally locating the (x, y) coordinate of the center of each detected tool tipin the 2-D frame. Poon and Rohling [84], in research on calibration of 3-D ultrasound probes, used the centroid of an image region around a usersupplied location to semi-automatically detect the center of each fiducial.This method solves the second subproblem, but not the first. In this study,our aim was to create a method for solving both of the problems mentionedearlier, making da Vinci instrument tip localization fully automatic, whileachieving registration accuracy comparable to the 0.95 ± 0.38 mm that wepreviously achieved using manual fiducial localization [5].We propose a multi-scale filtering technique for this 3-D TRUS instru-ment tip localization problem, which is a combination of a second-orderGaussian derivative, a circular Hough transform, and a hierarchical cluster-ing technique. A 3-D mask is first created based on a background ultrasoundvolume (a volume that is acquired before pressing the instrument tip on thetissue surface) and is applied to the ultrasound volume that includes theinstrument tip. Next, the masked ultrasound volume is further filtered tofind the edges representing the candidate tip locations in the remaining partof the image.Finally, the tip of the instrument is found using a circular Hough trans-form. Hence, the tool location is both identified in the 3-D volume and insideits corresponding 2-D frame. The same technique could also potentially beused to localize any surface fiducial pressed against the airtissue boundaryfor registration or other purposes.As described in this section, we have tested our automatic fiducial lo-calization method in experiments using several tissue models and also ina clinical scenario where the RALRP surgical procedure was performed onan anesthetized dog using a da Vinci Si surgical system. In each case, thefiducial localization error (FLE) and the target registration error (TRE)were evaluated and compared with the manual method. Complete detailsof the methods, further analysis of the results and discussions and in vivo494.2. Materials and Methodsvalidations are presented in this chapter.4.2 Materials and MethodsSeveral key factors have to be considered in the selection of a detection al-gorithm for this problem. Using our TRUS probe manipulator, 3-D TRUSdata are acquired as a series of 2-D images during the motion of the probe.While this produces 3-D ultrasound data, actual volumes are constructedby an off-line scan conversion algorithm. The detection algorithm must thusbe applied to the raw sequence of 2-D images if it is to run in real time. Theappearance of the target fiducials is fairly consistent. As depicted in Fig-ure 4.2, ultrasound images of the da Vinci instrument tips pressed againsttissue surface all contain similar features: strong horizontal lines showingthe airtissue boundary, approximately circular areas of high-intensity andvertical reflection lines from the instrument tips themselves. The scale ofthe ultrasound volumes is fixed by the spatial resolution of the ultrasoundtransducer, so scale invariance is not required. The RALRP procedure in-volves fairly consistent relative orientations of the TRUS transducer and daVinci instruments, so rotational invariance is likewise not required. Finally,the detection must be very rapid and a detection algorithm that can scan anentire volume in a few seconds is necessary to avoid disrupting the surgicalworkflow.Figure 4.2: (a) Example of an airtissue boundary in an ex vivo liver tissue.(b) da Vinci instrument tip pressed against an airtissue boundary.504.2. Materials and Methods4.2.1 Automatic Detection AlgorithmFigure 4.2 shows an example image of da Vinci instrument tip pressedagainst the airtissue boundary of ex vivo liver tissue. In our method,the automatic localization of da Vinci instrument tips is performed in foursteps: masking, filtering, circle detection, and removal of false positives. Aschematic of the process is shown in Figure 4.3.We first apply a 3-D mask, primarily to remove ultrasound data thatlies beyond the airtissue boundary. To create the 3-D mask, series of 2-Dimages of the background volume have been filtered first using a Hessian-based Frangi vesselness filter [28]. This vessel-enhancing diffusion filter ischosen to find the airtissue boundary lines that are shaped similar to vesselsin lung and cardiac images [16, 20]. In this filtering approach, the principaldirections in which the maximum changes occur in the gradient vector ofthe underlying intensity in a small neighborhood are identified. Eigenvaluedecomposition of the Hessian matrix is used to calculate these directions.Hence, the Hessian matrix at each pixel of the image for a particular scale(σ) of the Gaussian derivative operator is computed.Then, the eigenvalue decomposition is applied to extract two orthonor-mal directions. Based on the computed eigenvalues (λ1, λ2) at each scale(Σ = 1, 3, 5), a measure is defined as follows:S = maxσ∈ΣS(σ) ={0 if λ2 > 0exp(R2b2β2)(1− exp(−R2n2c2)) if λ2 < 0(4.1)Rb =λ1λ2is the blobness measure in 2-D and and Rn =√λ21 + λ22 isa background or noise reduction term.Rb shows deviation from blob-likestructures and Rn shows the presence of the structure using the fact thatthe magnitude of the derivatives (eigen values) is small in the backgroundpixels. In this equation β is chosen as 0.5 and c is chosen to be the maxx,y Rn.Such values are chosen according to the application of the Frangi filter invessel detection in cardiac US images [28]. Further image processing is doneto remove the small areas of high intensity, which are randomly scattered514.2. Materials and Methodsin the filtered images due to speckles. Each image is first thresholded bya value obtained from the mean of the intensities of pixels in the images.Figure 4.3: Automatic tool detection framework. The background ultra-sound volume and the volume with the tool inside are the inputs. Slicenumber and the (x, y) position of the tool tip in the detected slice are theoutputs.524.2. Materials and MethodsNext, parts that have less than M connected components are removed. Thevalue M = 10 is obtained by a trial-and-error process for our TRUS images.The remaining components, which represent the high intensity and rel-atively large areas corresponding to the tissue structures in the image, aredilated using morphological operators. To find the airtissue boundary, aline detection algorithm using a Hough transform is applied to the images.Lines with the minimum length of 30 pixels and minimum gap of 15 pixelsbetween them are extracted in 260 180 pixel images. As it can be seen fromFig. 3, the airtissue boundary region is shaped like a line which could bestraight or curved depending on the tissue surface shape. The line detectionalgorithm detects lines which construct this region regardless of its curva-ture. A mask is created using the detected airtissue boundary and artifactand is applied to remove regions corresponding to the artifacts and regionsoutside the tissue boundary.After applying the mask to the image set, the Frangi filter is applied tothe series of 2-D images and the relatively large components are extracted.A circular Hough transform is applied to the obtained components to findcircles with radius of approximately 5 pixels and minimum pixel count of20. The mean location of these circles (the circular Hough transform findsa number of circles in the vicinity of the tool tip region) in each 2-D im-age is computed as a candidate for the tool location. Next, a hierarchicalclustering algorithm [44] is performed to find the group of candidates thatare in adjacent slices, considering the fact that the tool tip can be seenin a multiple consecutive slices. Candidates that are close to each other(Euclidean distance) are linked into the same cluster. The linkage func-tion continues until an inconsistency coefficient reaches a threshold. Thisapproach removes false positive detections that do not necessarily occur inadjacent slices. Once the clusters of 2-D images that have the tool insidethem are found, the slice containing the maximum tool tip intensity insidethe largest cluster is chosen to be the output slice for the detection algo-rithm. Results shown in Figure 4.5 and Figure 4.6 clearly demonstrate theearlier methodology implemented in different experiments.534.3. Experimental Results4.2.2 Experimental Setup for TRUS Data CollectionThe robotic system shown in Figure 2.1 (a), which is designed for intra-operative TRUS imaging during RALRP, was used for image acquisitionin this study. This system consists of a robotic ultrasound probe ma-nipulator (robot), a Sonix TABLET ultrasound machine (UltrasonixMed-ical Corp., Richmond, VA, Canada) with a parasagittal/transverse biplaneTRUS probe, and control and image processing software. A standard brachyther-apy stabilizer arm (Micro-Touch 610-911; CIVCO Medical Solutions, Kalona,IA, USA) was mounted to the operating table, and the robot was installed onthe stabilizer, as shown in Figure 2.1 (a). Volumes of 3-D TRUS data wereacquired by rotating the 2-D imaging planes and automatically recording theencoder positions for each image. Software running on the ultrasound con-sole is responsible for controlling the robot movements and the ultrasounddata acquisition.4.2.3 System InterfaceA simple graphical user interface (GUI) was designed for the surgical teamto allow the user to position the probe, automatically collect 2-D B-modeimages and radio-frequency (RF) data while rotating from −40° to +40°,and perform this automatic registration during RALRP procedure. Theregistration workflow during this surgical procedure is explained in the dia-gram depicted in Figure 4.4. The surgeon or his/her assistant will use theGUI installed on the Sonix TABLET ultrasound machine to perform theregistration.4.3 Experimental Results4.3.1 Phantom and Ex-vivo ResultsThe proposed automatic detection method has been initially tested on threedifferent tissuemodels: a custom-made polyvinyl chloride (PVC) prostatephantom of appropriate size and shape to model the imaging conditions544.3. Experimental ResultsFigure 4.4: TRUS to da Vinci registration workflow during RALRP proce-dure.554.3. Experimental Results(a) (b) (c)Figure 4.5: Sample detection results and their corresponding imaging setups.The da Vinci large needle driver is pressed against the tissue surface ondifferent locations and the location of its tip is found using our detectionalgorithm in 3-D.during RARLP, an ex vivo liver and ex vivo bovine tissue. A surgical ma-nipulator commonly used in RARLP (Large Needle Driver, Intuitive SurgicalInc.) was used to perform the registration. Sample results of the tool detec-tion algorithm along with the testing configuration for each of the datasetsare depicted in Figure 4.5.Instrument tip intensity in the US images versus probe roll angle isplotted for TRUS volumes in each dataset and sample results are shownin Figure 4.6. As seen in these figures, the detection algorithm looks atthe area under the tool intensity versus probe roll angle curves, selects thecluster as the group with maximum area, removes all false positives, andfinally, selects the point with the maximum intensity inside the cluster asthe final detection. Scatter plots of the candidate detected points along with564.3. Experimental Resultsthe selected cluster, false positive detections, and the selected 2-D slices arealso depicted in Figure 4.6 to further clarify the robustness of the proposedalgorithm. The rigid point registration technique proposed by Umeyama[111] was used in this study to evaluate the automatic detection algorithmsperformance.The da Vinci instrument tip was pressed against the airtissue boundaryat 12 locations in each of the three collected datasets (Nt = 12) and itsposition (x0) in the ultrasound robot frame {O0, C0} is calculated using ourautomatic detection algorithm. The da Vinci API was used to provide thelocation of the tool tip (x1) in the da Vinci frame {O1, C1}. There is a 9 mmoffset between the visible edge of the instrument tip in US and the distalelement of the kinematic chain provided by the da Vinci API. This distancewas added to the vertical component of the point locations in ultrasoundimages before matching the point pairs in the two frames and calculating theregistration transformation. This offset distance might contribute to someof the registration error, since the tool tip might be slightly angled when itis pressed against the tissue.For each registration experiment, Nf ≥ 3 points were randomly pickedfrom the total points collected for each tissue type (3 is the minimum numberof points for finding the transformation), and the rigid point registrationmethod was used to compute the transformation between the frames T 10to minimize the fiducial registration error (FRE). FRE is computed as theroot mean square of the distance between corresponding fiducials after theregistration. Next, the remaining points in each dataset were assumed to bethe target points and the calculated transformation was used to transformthem from the TRUS robot frame to the da Vinci frame and the TRE isestimated. Estimated TRE includes the real TRE and the target localizationerror (TLE).TRE is defined as the root mean square of distances between corre-sponding fiducials after registration (i.e., the distance between the localizedposition of each tool tip as transformed from the ultrasound robot space toda Vinci space and the position of that corresponding tool tip localized inthe da Vinci space provided by the API).574.3. Experimental Results(a)(b)(c)Figure 4.6: Sample results of tool intensity-probe angle, detected points inthe TRUS volume, and 2-D slide and tool tip inside for (a) liver dataset, (b)PVC phantom dataset, and (c) Bovine tissue dataset. Each group of figuresis associated with one experiment in each dataset.584.3. Experimental ResultsValues of FRE and TRE were computed for different number of fiducialsNf chosen between the total points for each dataset Nt. Mean values ofFRE and TRE and their standard deviations were then calculated for eachcombination of (Nf ,Nt) for 100 iterations (n = 100) and the results areplotted for each dataset in Figure 2.6.As it can be seen from Figure 4.7, as Nf increases, both mean andstandard deviation of TRE decreases. Based on this analysis, the number offiducials suggested for this registration is (Nf = 6). The mean and standarddeviation of the TRE for this number of fiducials and 100 iterations arereported in the anatomical frames of the patient in Table 4.1.Tool tip segmentation error (FLE)was also calculated in (x, y, theta) di-rections to further show the accuracy of the method. Three human subjectswere asked to independently identify the location of (n = 10) fiducial pointsin the TRUS volumes and their detections were considered to be the goldstandard detections. The manual tool detection method involved scrollingthrough 2-D slices of the volume, finding the 2-D slice with the tool tipinside and selecting the approximate center of each fiducial. The differencebetween the average user-defined locations and the result of our algorithm,in addition to the inter-subject variations (ISVs) were calculated and arereported in Table 4.2.Table 4.1: Mean errors (n = 100) between tool tip location and predictedlocation based on registration. Errors are presented in the anatomical frameof the patient, along the superior-inferior (eS−I), medial-lateral (eM−L) andanterior-posterior (eA−P ) axes.TREA−P(mm)TRES−I(mm)TREM−L(mm)Mean TRE(mm)Phantom 0.62 ± 0.31 1.29 ± 0.22 0.82 ± 0.24 1.80 ± 0.32Liver 0.72 ± 0.83 2.65 ± 0.51 1.18 ± 0.54 3.33 ± 0.81Bovine 1.90 ± 0.79 1.24 ± 0.34 3.51 ± 0.70 4.54 ± 0.88594.3. Experimental ResultsNf=3, Nt=9 Nf=4, Nt=8 Nf=5, Nt=7 Nf=6, Nt=6 Nf=7, Nt=5 Nf=8, Nt=4 Nf=9, Nt=30246810121416  FRETRE(a)Nf=3, Nt=9 Nf=4, Nt=8 Nf=5, Nt=7 Nf=6, Nt=6 Nf=7, Nt=5 Nf=8, Nt=4 Nf=9, Nt=30246810121416  FRETRE(b)Nf=3, Nt=9 Nf=4, Nt=8 Nf=5, Nt=7 Nf=6, Nt=6 Nf=7, Nt=5 Nf=8, Nt=4 Nf=9, Nt=3024681012  FRETRE(c)Figure 4.7: FRE and TRE and their standard deviations for different num-ber of fiducials (Nf ) and target points (Nt) for (a) Steak dataset, (b)Liverdataset and (c)Phantom dataset. As the number of fiducials increases, TREdecreases. We suggest using six fiducials in clinical applications.604.3. Experimental Results4.3.2 In-vivo ResultsIn order to show the clinical feasibility of the proposed methods, we furthertested the algorithm on datasets collected during our in vivo canine study.The surgeon was asked to press the tool tip of the da Vinci instrumentagainst the prostate surface while a full TRUS volume was acquired. Thetool tip was visible as a hyperechoic point with strong vertical reflectionlines in the B-mode images, as shown in Figure 2.5. (N = 12) differenttarget locations on the prostate surface (also shown in Figure 2.5) and cor-responding volumes were acquired. Tool tips were automatically located ineach volume, and for n = 100 iterations, Nf = 4 point pairs were pickedat random and a least squares problem was solved to find the registrationhomogeneous transformation. The remaining Nt = N − Nf = 8 target lo-cations were used to calculate the TRE. To determine the ISVs in fiduciallocalization and analyze its effect on TRE, four different ultrasound userswere asked to localize the tool tip in each of the N = 12 TRUS volumes.Manual fiducial localizations were compared with the automatic detectionsto calculate FLE.The TRE, FRE, FLE, and ISV obtained with the automatic localizationmethod compared to manual localization are listed in Table III. The tableincludes the TRUS imaging plane θ localization error, and the localizationerror (x, y)=(lateral,axial), in the plane at θ.Table 4.2: Mean errors (n = 10) between manual tool tip location and au-tomatic tool tip location (x, y, theta) directions and inter-subject variationsin (x, y, theta) directions.FLE(x,y)(mm)FLE(theta)(deg)ISV(x,y)(mm)ISV(theta)(deg)Phantom 2.13 ± 0.97 0.67 ± 0.34 3.65 ± 0.88 1.55 ± 0.39Liver 3.11 ± 0.88 0.79 ± 0.39 4.25 ± 1.45 2.05 ± 0.8Bovine 2.42 ± 1.19 0.68 ± 0.48 3.13 ± 0.88 1.43 ± 0.44614.4. Discussion of Results4.4 Discussion of ResultsThe overall average TRE previously reported in [5] using the manual local-ization of three fiducials on the PVC tissue phantom was 0.95 ± 0.38 mm.The average TRE using the automatic detection technique in this study is1.80±0.32 mm for the recommended number of fiducial points Nf = 5 on thePVC tissue phantom. This error could be further reduced to 1.61±0.39 mmby increasing the number of fiducial points to 9. To further evaluate the algo-rithm robustness before clinical applications, experiments have been done ondifferent tissue types (ex vivo liver and bovine tissue), which are more similarto the human tissue. The minimum calculated TRE value is 2.86±1.40 mmfor the liver tissue and 4.15 ± 0.61 mm for the bovine tissue, for pickingnine fiducials on the airtissue boundary. Our initial clinical study shows thealgorithms accuracy and applicability in a real surgical scenario with aver-age TRE of 2.68± 0.98 mm, while four points were chosen to calculate theTRE. The overall average TRE using the manual localization of four pointsis 1.88±0.88 mm. Using more fiducial points will further decrease the regis-tration error, as shown in Figure 4.7. We propose choosing six tool positionson the tissue surface, Nf = 6, both to achieve an acceptable registration er-ror and to have a reasonable number of tissue indentation repetitions duringthe surgical procedure. It is pointed out by Ukimura et al. [108] that themean distance between the neurovascular bundles (NVBs) and the lateraledge of the prostate ranged between 2 and 3 mm from apex to base. Thisis suggestive of the required accuracy of a guidance system since one majoraspect of the system is to accurately localize the NVB. According to theseresults, the automatic detection algorithm could yield comparable registra-tion accuracy with the manual detection method. It could also compensatefor errors resulting from misinterpretation of tool locations in the TRUSvolume. The goal of instrument tracking is to have the TRUS parasagittalimaging plane contain the tip of the da Vinci instrument at all times. Hence,TRE values in the axial and lateral ultrasound directions are irrelevant aslong as the tool tips are within the image boundaries. Because the thicknessof the TRUS beam at the anterior surface of the prostate is on the order624.4. Discussion of Resultsof millimeters at moderate imaging depths, small errors in the elevationaldirection are not critical. The appearance of surface fiducials in TRUS im-ages, while regular, is not necessarily exactly that of a spheres outline (orthe outline of another shape in the case of da Vinci tools).It is possible that reverberation artifacts, or other issues,might introduceconstant offsets in the fiducial localization that would make our registrationless accurate. This inaccuracy would not be revealed in the testing describedin this work, since we use the same surface fiducials as targets to validatethe registration. While this could theoretically be a factor in the valida-tion presented here, we believe it is not significant for two reasons. First,previous work on ultrasound surface fiducials [36] has shown that at worstthe artifacts result in a small constant offset. Second, we have performedvalidation on a related method for 3-D TRUS to camera registration thatdid not use surface fiducials for validation, resulting in roughly equivalentaccuracy [5].Our focus for this registration technique is the RARLP procedure, andwe believe it would be effective to choose two points on the prostate apex,two points on the midgland, and two points on the prostate base. As it is alsoshown in Figure 4.7, choosing six fiducials results in satisfactory registrationerror in our in vivo study. In this process, the da Vinci instrument tip isslightly pressed on the surface of the tissue at different locations. The daVinci instruments should not be deflected during the registration process.This is because we rely upon the joint angle measurements and the tool tippositions, computed via the da Vinci API by solving the robot kinematics,to localize the tool tip relative to the base of the da Vinci patient side cart.The bending deflection of the da Vinci instrument is not taken into ac-count by the robot API, and is not measured in this work. Such deflectionscould be estimated by methods explained in [100]. We assume that thebending stress on the instrument resulting from careful soft tissue indenta-tion in this application is negligible and will not affect da Vinci API readouts. Furthermore, the actual TRUS transducer remains fixed relative tothe da Vinci during the entire surgery. It is in a fixed correspondence withrespect to the robot and held so by a brachytherapy stepper as shown in634.4. Discussion of ResultsFigure 6.1(a). Brachytherapy steppers have been used in many proceduresand do not move.Replacing the process of manual detection of tool tips in the TRUS vol-ume with the proposed automatic technique will accelerate the registrationand reduce the disruption to surgical workflow. The processing of each USvolume consisting of 150 2-D US images takes approximately 55 seconds.The algorithm is currently implemented in MATLAB, but future implemen-tations of the algorithm in GPU will further accelerate the processing. Aspreviously mentioned in [3], performing a registration by manually identi-fying three common points takes an experienced operator approximately 2min, although a surgeon working with actual tissues would likely take longer.Using the automatic method will approximately take the same amount oftime; while there will be no need for a sonographer to attend the surgeryto manually find the tool tip in the US volume. In addition, the resultswill not be user-dependent, as is the case in the manual detection. Com-parison between the ISV results and automatic localization errors reportedin Table 4.2 further proves the consistency and accuracy of the proposedautomatic method with respect to the manual method.The overall goal of this work is to provide an easy to use TRUS guidancesystem to the surgeons. The images will be streamed to the surgeon consoleand will be displayed side-byside to the camera view of the surgical site usingthe TilePro—feature of the robot. Because the surgical tool itself is used topoint to the ultrasound plane, if the pointing is correct, touching the tissuewill provide an ultrasound image of the tool. This in itself is a reasonablecheck of the tool tracking accuracy. If more than two noncollinear touchesare performed, then the problem is constrained fully. Thus, there is a certainsafety built in the method that identifies if the tracking is working withinthe required accuracy. At critical stages of the procedure such as the bladderneck and NVB dissections, the surgeon could use the TRUS images at thedesired locations by easily placing the tool at that spot. In case of algorithmfailure, the manual registration method could be performed instead. Even ifthe manual registration fails in this process, manual remote manipulation ofthe TRUS robot will be used by the operator to provide TRUS guidance to644.5. Conclusionsthe surgeon. Hence, there will not be any patient safety concerns involvedin this process. The only invasive component of our system is a TRUSprobe, which will be placed inside the patients rectum during the surgeryand its motion will be restricted only to roll motion, which is harmless tothe patient.We performed a simulated RALRP procedure in our initial animal studyand tested the effectiveness and suitability of all the methods mentionedearlier. We will proceed with patient studies, for which we have alreadyobtained ethics approval.An alternative method to the one presented earlier would be to performa registration of the laparoscopic stereo camera to ultrasound, as done in[114]. That registration relies on the accurate localization of targets in bothultrasound and in the camera views, by using specially designed fiducialsthat can be seen in both ultrasound and the camera views.However, thereare several technology hurdles to the quick translation of the camera to ul-trasound registration method into clinical practice. The da Vinci cameracalibration changes with shifts in the cameras focal distance, which is aparameter that is unfortunately not available via the da Vinci API. Recal-ibration within the body is not straightforward with the present methodsand tools. The da Vinci API provides us with the location of the instrumenttips in a simple and reliable way. Having the registration working with thisapproach will enable the easy integration with the camera as soon as a groupdevelops a reliable da Vinci-based stereo tracking algorithm.4.5 ConclusionsIn this study, we have addressed the problem of detecting da Vinci instru-ment tips pressed against an airtissue boundary, in 3-D TRUS data. Amethod based on multiscale filtering and circle detection has been proposed.The tool tip localization accuracy is evaluated by analyzing the registrationerror between the TRUS robot frame and the da Vinci frame. Overall, themethod decreases the complexity of the registration procedure while keepingits accuracy at the same order of the manual method. The proposed tech-654.5. Conclusionsnique will further facilitate integration of TRUS in RALRP with minimuminterruption to the surgical procedure. After an automatic registration,which could be completed in less than 5 min using a simple GUI, the sur-geon can use TRUS imaging in many steps of the RALRP procedure byeasily moving the surgical tool around to control the TRUS imaging plane.Future work will include clinical human trials to verify the accuracy andreliability of the proposed technique. Furthermore, GPU implementation ofthe algorithm to reduce the processing time to approximately 2 min is alsoplanned in the future work.66Chapter 5Tracked Robotic UltrasoundPalpationIn this chapter, we propose a method of using ultrasound imaging for tissuepalpation using a robotic ultrasound system that can track a surgical instru-ment. When the surgical instrument presses against the tissue thus causingan indentation, the resulting sub-surface strain image is computed and dis-played to the user. The sub-surface strain is indicative of tissue stiffness,and could be used as a contrast mechanism to better outline boundaries ofpalpable tumors. The automatic tracking of the instrument tip by the ultra-sound transducer can be used to regain a sense of palpation by displaying,along or instead of the conventional ultrasound image, the strain field pro-duced by the compression of tissue by the instrument tip near or adjacentto the tip.We implemented the method on two robotic ultrasound platforms, each de-signed for specific clinical applications: (i) a robotic transrectal ultrasound(R-TRUS) system (explained in Chapter 2) for robot-assisted Prostatectomyprocedure, and (ii) da Vinci surgical system auxiliary patient-side manip-ulator (PSM) used with a robotic ultrasound transducer for robot-assistedNephrectomy procedure. Implementation on the da Vinci surgical system(fully explained in Appendix A) was done using the da Vinci research kit(dVRK) controllers that enable complete access to all control levels of theda Vinci manipulators. Initially, experiments have been performed on tissuephantom and ex-vivo models in a lab setting and promising results have beenobtained across n = 5 users testing the system. Furthermore, we tested theR-TRUS system in a clinical study involving n = 5 patients undergoing the675.1. Introductionda Vinci Prostatectomy procedure. The clinical study, being the first of it’skind, was conducted to first evaluate the feasibility of using this palpationsystem in a real human procedure, and second, to evaluate whether such animaging system would provide surgeons with useful information during theprocedure.5.1 IntroductionRobotic surgical systems provide a platform for integrating surgical guidancewith imaging and automation of particular tasks, while enhancing the abilityof surgeons to perform delicate and precise minimally invasive surgery. Onelimitation of many robotic surgical systems is the inability of the operatingsurgeon to feel the tissues on which they are operating. During open surgery,surgeons are able to physically palpate the tissue and feel for stiff tumorsunder the surface, which enables them to tailor their resection margins toremove only the cancerous tissue and preserve as much healthy tissue as pos-sible. Unfortunately during minimally invasive or robot-assisted procedures,the physical cues provided by open surgery “hands in” palpation are lost.This is because either the surgeon needs to manipulate long laparoscopic in-struments through small incisions, with palpation becoming equivalent withprobing tissue with a long stick, or because the robotic system that copiesthe surgeons hand motion at the master console to the instruments withinthe body provides limited haptic feedback. The loss of haptic feedback dur-ing surgery has been lamented by surgeons [8, 34, 60, 76, 79]. Proceduressuch as radical Prostatectomy or partial Nephrectomy could become less in-vasive if the precise location of the tumors could be found through palpationof sub-surface tissue not seen with the normal laparoscopic camera.Mechanical properties of tissue that are felt by palpation are importantindicators of disease potential. Indeed, palpation techniques are commonlyused by medical doctors to determine the potential for disease, for example,stiffer tissue regions that can be felt as harder objects can indicate thepresence of cancer. This is the basis for a number of clinical examinationssuch as the digital rectal examination for prostate cancer.685.1. Introduction5.1.1 Background ReviewSeveral techniques have been proposed to enable medical robots to detectlumps in soft tissue. Howe et al. introduced a teleoperation method withhaptic force feedback to convey the biomechanics of a palpated artificialtissue. They found that use of their tactile display resulted in increasedsuccess of tumor identification and localization [40]. Trejos et al. showedthat autonomous robotic tumor localization resulted in increased tumor de-tection accuracy and a significant decrease in the maximum force appliedcompared to teleoperated human palpation [104]. Ahn and Kim teleoper-ated a robot that mimicked the geometry and motions of a doctors hand;allowing a robotic finger to sweep across the prostate and use the resultantforce profile to assess the likelihood of tumors [7]. Sangpradit et al. observeddiscrepancies between a finite element model approximating a wheel-tissueinteraction and force data taken during rolling contact experimentation toidentify simulated tumors of diverse shapes and depths [57]. Liu et al. useda force-sensitive wheeled probe to gather what they referred to as a “rollingmechanical image” and found the continuous measurement approach to besensitive to differences in force profiles caused by simulated tumors [57].The imaging of mechanical properties of tissue is called elastography.One approach to elastography for medical imaging is strain imaging, whichinvolves the acquisition of tissue images under different states of compres-sion, and computing the relative tissue displacement field using speckletracking [115]. Relative tissue displacement or strain is indicative of tissueelasticity. Indeed, stiffer tissue will undergo less strain than soft tissue, soa strain image of tissue provides elasticity or tissue stiffness contrast. Theunderlying assumption is that the strain is linearly related to the stress andthat this relationship is described mathematically by a linear scale factorcalled the Young’s modulus, or simply elasticity. Ultrasound is a commonimaging modality for this method because it is ubiquitous, non-invasive,safe, inexpensive and portable.We propose a method of using ultrasound for tissue palpation. Themethod employs a motorized or beam-steered ultrasound beam that can695.2. Materials and Methodstrack a surgical instrument. When the surgical instrument presses againstthe tissue thus causing an indentation, the resulting sub-surface strain imageis displayed. The sub-surface strain is indicative of tissue stiffness, worksat the depth of the ultrasound beam and can be used as a more objectiveform of palpation. Compared with manual palpation, this method has theadvantage of evaluating deeper lying lesions, and furthermore it is semi-quantifiable.A competitive approach would be to use a laparoscopic ultrasound toprobe tissue depth by strain imaging, using the ultrasound transducer itselfto compress the tissue [10]. This approach has the disadvantage that atransducer has to be picked up and used, which slows down the procedureand is much more cumbersome than using the instrument for compression.Using the instrument for compression requires a mechanical movement ofthe transducer, preferably rotation, or electronic beam steering.In this chapter, we describe systems and methods for creating high-quality real-time strain images using registered robotic ultrasound and man-ual surgical instrument indentation.5.2 Materials and MethodsTwo experimental systems were used to validate instrument-based strainimaging for robotic radical prostatectomy and partial nephrectomy usingthe da Vinci surgical system. In both cases, the ultrasound transducerswere robotically controlled and 3D ultrasound was registered to the da Vincisurgical instrument. In our proposed method for control of the robotic ul-trasound, the ultrasound imaging plane is automatically repositioned usingthe robot so that a 2D ultrasound image continuously contains the tip of aspecified da Vinci surgical manipulatoras demonstrated in Figure 5.1. Theautomatic instrument tracking is achieved using an air-tissue boundary reg-istration method explained in Chapter 2 and 3. The details of the methodimplementation for the pick-up ultrasound with the dVRK system is ex-plained in Appendix A and [66].705.2. Materials and MethodsdaVinci instrument Robotic TRUS  TRUS  sagittal image plane {𝑂 𝑑𝑉 , 𝐶  𝑑𝑉} {𝑂 𝑅𝑇 , 𝐶  𝑅𝑇} Motion for strain 𝑑 Roll angle 𝜃   +𝟑𝟎° 𝟎° −𝟏𝟓° (a)𝑑 US-PSM PSM1 da Vinci Tool  and  Ultrasound Probe Intra-operative  Ultrasound PSM Intraoperative pick-up Ultrasound {𝑂1, 𝐶 1} {𝑂2, 𝐶 2} {𝑂𝑤1, 𝐶  𝑤1} {𝑂𝑤2, 𝐶  𝑤2} {𝑂𝑢𝑠 , 𝐶  𝑢𝑠} (b)Figure 5.1: (a) Instrument-based strain concept with robotic registeredtransrectal ultrasound for radical prostatectomy procedure, (b) Instrument-based strain concept with a pick-up ultrasound transducer grasped by theda Vinci auxiliary manipulator for partial nephrectomy procedure.715.2. Materials and Methods5.2.1 Real-Time Strain Imaging SystemReal-time strain imaging is achieved using a speckle tracking motion estima-tion method based on a time domain cross correlation with prior estimates(TDPE) technique [115]. We implemented TDPE on a SonixTablet ultra-sound machine (Analogic Ultrasound, Richmond, Canada) and it runs at25 fps for strain images of 16 000 pixels. 40% of the CPU is used by themain ultrasound program and R-TRUS control software and the remaining60% of the CPU is used for RF data acquisition, motion tracking, least-square strain estimation and display. 3D ultrasound images were capturedusing a sagittal/transverse biplane TRUS transducer (Transrectal BiplaneCurved BPL9-5/55, 9-5 MHz) by rotating the TRUS transducer about itslong axis and collecting the 2-dimensional sagittal images at each angleincrement. Software running on the ultrasound console is capable of com-puting high frame rate strain images which are tracked in the coordinateframe of R-TRUS (Figure 5.1), controlling the robot movements in real-time and ultrasound data acquisition. Mechanical excitation is applied tothe tissue manually using the surgical instruments. A sequence of nf framesof RF-data is acquired for each 2D plane, processed using TDPE to cre-ate a series of real-time displacements per pixel, and then processed by aleast-square strain estimation. Temporal averaging of the displacements isperformed to reduce the noise-to-signal ratio. Histogram equalization is alsoperformed to unify the scale and intensity of the strain images. In order toevaluate the quality of the strain images in real-time, average correlationcoefficient (ACC) is being computed and shown in the imaging interface.The strain images are displayed to the surgeon within the console usingthe TilePro—feature of the da Vinci S surgical system to provide real timeguidance.While it does not provide an absolute value of elasticity, strain imagingprovides an idea of tissue compliance. If in response to an excitation, acertain region of interest exhibits higher strain than the background, it islikely to be softer.For validation of the method in the case of radical prostatectomy, a CIRS725.3. Phantom and Ex-vivo User Study and Resultsprostate phantom (Model 066) was used with a parasagittal/transverse bi-plane TRUS transducer shown in Figure 5.3. This phantom contains the theprostate, along with structures simulating the rectal wall, seminal vesiclesand urethra in a 11.5 x 7.0 x 9.5 cm clear plastic enclosure. A 5mm thicksimulated perineal membrane enables various probes and surgical tools tobe inserted into the prostate. This phantom contains 2 isoechoic randomlyplaced lesions that are at least two times stiffer than the simulated prostatetissue. The ProGrasp robotic instrument was used to touch the surface ofthe simulated perineal membrane gently to produce the strain images.When considering the case of partial nephrectomy, a different set-up wasused as shown in Figure 5.4. An ex-vivo kidney was implanted with stiffPVC inclusions. These inclusions were then imaged using a robotically heldultrasound transducer [92].5.3 Phantom and Ex-vivo User Study andResultsInitial validation of our palpation method was performed on a CIRS prostatephantom (Model 066). The phantom contains the prostate anatomy and twoisoechoic randomly placed lesions that are at least two times stiffer than thesimulated prostate tissue. The ProGrasp—robotic surgical instrument wasused to excite the surface of the phantom’s simulated perineal membranegently to produce the strain images. In the first set of experiments per-formed, high-quality strain images were produced in the prostate phantom.The light pressure of the robotic tool on the phantom surface was enough tocreate a strain image of the entire phantom. The lesions within the phantomcould easily be located along with the other anatomical features, such as thesimulated urethra (Figure 5.3).Next, a user study was conducted in which n = 6 users (2 Urologysurgeons, 4 training residents) tested the system in a mock up operatingroom with a da Vinci Si surgical robot and the R-TRUS system. The userswere asked to sit at the da Vinci surgical console and use the system to735.3. Phantom and Ex-vivo User Study and ResultsUser study imaging interface inside the daVinci®Si surgeon console ( Tilepro™ ) TRUS images Real-time strain images Prostate boundary Stiff inclusion Surgeon console of the daVinci®Si system CIRS 066 prostate phantom Figure 5.2: User study imaging interface using the da Vinci Si system,TilePro—and a CIRS 066 prostate phantom. Under normal B-Mode ul-trasound the stiff inclusions cannot be detected but are readily visible onstrain images.palpate the prostate phantom and localize the two stiff inclusions. The real-time strain and TRUS images were being relayed to the surgeon consoleand visualized to the users using the TilePro—feature of the da Vinci Sisystem as depicted in Figure 5.2. R-TRUS to da Vinci surgical instrumentregistration was performed before users start testing the system. All sixusers could successfully localize the stiff inclusions inside the phantom withthe results listed in Table 5.1. In this set of experiments, we have shown thefeasibility and potential use of our system in a scenario identical to a realrobotic surgical operation and with only the tools that would be normallyavailable to a robotic surgeon.Using an ex-vivo porcine kidney, the lesions could also be identified on745.3. Phantom and Ex-vivo User Study and ResultsTable 5.1: Phantom st..iff inclusion exploration user study results. Strainimaging parameters: LSE neighbors: 7, Temporal averaging: 90, Frame-rate: 25 HzInclusion-I roll angle(deg) Inclusion-II roll angle(deg) Timesurgeon 1 11.47 ± 1.83 -21.78 ± 1.65 320ssurgeon 2 12.63 ± 1.22 -22.00 ± 1.04 290sresident 1 11.63 ± 1.22 -22.00 ± 1.04 270sresident 2 10.95 ± 1.28 -20.11 ± 1.17 311sresident 3 12.58 ± 1.63 -21.83 ± 1.76 264sresident 4 11.47 ± 1.83 -21.78 ± 1.65 320sAverage 11.65 ± 1.24 -21.93 ± 1.90 291sthe strain and displacement images from the small intraoperative transducer.We touched several different locations around the transducer to estimatethe effect on the strain image, but in all cases the strain image clearlydepicted the embedded stiffer region (Figure 5.4 and Figure 5.5). In thisset of experiments, we have shown the feasibility of creating strain imageswith only the tools that would be normally available to a robotic surgeon.In the next stage of this study, we have tested the method in real humanpatient prostatectomy cases. This method is entirely feasible and easy toimplement in a real surgical setting with a minimum of added equipment.Although strain imaging is sensitive to boundary conditions and operatortechnique and, in the absence of measured forces, produces only relativemeasurements, it is a simple but highly effective method for determiningthe stiffness under the surgeons tool. Because the proposed integration ofstrain imaging uses a tool tracking technique, the surgeon can easy andintuitively determine the stiffness directly in the area where they are goingto work. This would potentially improve the surgical margins and allow formore precise dissections.755.4. Human Patient StudiesFigure 5.3: (a) Experimental setup for imaging the CIRS 066 prostate phan-tom using robotic TRUS and da Vinci ProGrasp instrument, (b) Real-timeB-Mode and displacement image example results produced by the users forthe phantoms inclusions and urethra. Under normal B-Mode ultrasound thestiff inclusions cannot be detected but are readily visible on strain images.5.4 Human Patient StudiesAfter achieving promising results in phantom and ex-vivo studies, 5 patientswere enrolled in a clinical study (CREB: H11-02267) similar to the TRUS-guidance study described in Chapter 2. Our main goal in performing thisclinical study was: (i) to evaluate the technical feasibility of using such animaging system in a real da Vinci surgery scenario, (ii) to get some feedbackfrom the surgeons about the intuitiveness of the approach and our imaginguser interface and (iii) to evaluate its clinical usefulness to the surgeons(Phase I clinical study).765.4. Human Patient StudiesReal-time strain image  Real-time Bmode image  Porcine Kidney with PVC inclusions Robotic ultrasound of the kidney with instrument-based strain as substitution for remote palpation 23 mm (a) (b) Figure 5.4: (a) Experimental setup for imaging the ex-vivo kidney, (b) real-time strain and B-mode images produced using instrument-based strain andthe pick-up transducer.23 mm (a) Figure 5.5: TilePro—images from the da Vinci S surgical console. Bothstrain and B-mode images could be provided to the surgeon at the consolein real-time.775.4. Human Patient Studies5.4.1 Patient PopulationA total of n = 5 patients with clinically organ confined prostate cancer un-dergoing RALRP at our institution (Vancouver General Hospital) agreed toparticipate in this study between November 2014 and February 2015. Me-dian patient age was 63 years (range 52 to 70) and median baseline prostatespecific antigen was 5.9 ng/ml (range 0.84 to 36.4). Patient demographicand preoperative data are listed in Table 5.2.Table 5.2: Demographic and preoperative data.No. of patients 5Age, years ; median (range) 63 (55-72)Preoperative PSA, ng/ml ; median (range) 18.6 (13.5-28.5)Preoperative Gleason score (index lesion), no.(%):3+4=7 3 (60)4+4=8 1 (20)4+5=9 1 (20)Clinical stage, no. (%):T1c 1 (20)T2a 1 (20)T2b 3 (60)IPSS value ; median (range) 5 (2-9)Prostate volume, cc ; median (range) 35 (20-46)PSA: Prostate-Specific Antigen ; IPSS: International Prostate SymptomScore5.4.2 Procedure DescriptionThe main components and configuration of our system during this clinicalstudy inside the operating room are shown in Figure 5.7 and Figure 5.6.We usually setup our system in the operating room before the patientis placed on the operating table. This stage of the study includes settingup the video-capture PC with the da Vinci system, setting up and test-ing the da Vinci research API and TilePro software, and placing the steriletransducer cover (NeoGuard Transducer Cover CV610-843) on the TRUS785.4. Human Patient Studiestransducer. We sterilize the TRUS transducer before doing every clinicalstudy according to the guidelines provided to us by BC cancer agency (in-cluded in Appendix B.2). Once the patient in placed on the operating table,after he is anesthetized and while the OR staff are preparing the patient forport placement, we attach our robotic TRUS system to the foot of the op-erating table. Then we ask the attending Urology fellow to place the TRUStransducer inside patient’s endocavity to provide optimal transverse/sagittalimages of the prostate as in standard brachytherapy prostate volume stud-ies. The TRUS sagittal plane was placed underneath the prostate midlineso that the prostate could be imaged from the base to the apex. Our roboticTRUS system was draped using sterile disposable drapes (Sterile flat 81.3X 91.4cm polyethylene drape, CV610-870) to follow reprocessing guidelinesset by VCH operation research committee. Once the TRUS system is setup,the OR staff continue with preparing the patient and docking the da Vincirobot.Once the anterior surface of the prostate is visible to the surgeon, weperform the TRUS to da Vinci calibration (explained in Chapter 2) to en-able automatic tracking of the surgical instrument with ultrasound. Afterthe tracking is activated, the surgeon could examine the prostate anatomyand different regions by moving the registered surgical instrument around,pressing on the tissue surface of the area of interest, slightly indent the tis-sue and look at both TRUS and real-time strain images. As it is shownin Figure 5.10 and Figure 5.8, both real-time strain and TRUS images arebeing sent to the da Vinci console and visualized to the surgeon using theTilePro—feature of the da Vinci S system. Because the ultrasound imagesare always underneath the tip of the surgical instrument, using our imag-ing system was intuitive for the surgeon, especially in terms of 3D spacialorientation. Similar to the case of open surgery where they simply touchand press the surface of tissue in their area of interest and feel the tissuemechanical properties, in this case they touch and press with their surgicalinstrument and look at the strain image which conveys tissue mechanicalproperties.795.4. Human Patient StudiesFigure 5.6: TRUS and real-time strain guidance system components andconnection in the operating room.5.4.3 Results and DiscussionsThe calibration process was performed after incision of the endopelvic fasciawith the 4 instrument tip locations spread across both sides of the dorsalprostate and could be completed in approximately 2 minutes.Intraoperative registered TRUS and instrument-based strain imagingwas performed successfully for all 5 patients. The strain imaging was ex-amined at the anterior surface of the prostate, after the surgeon placed thedorsal venus complex (DVC) suture at the prostate apex, and before hewanted to start finding the bladder-prostate boundary and dissecting thebladder neck. Strain volumetric data was collected successfully (2D strain805.4. Human Patient StudiesFigure 5.7: Operating room configuration during the TRUS+Strain guid-ance study in OR no. 10 in Vancouver General Hospital.+ roll angle) for all patients. The software architecture used in this studyallowed us to use both automatic tracking control of the TRUS robot andreal-time volumetric strain imaging at the same time. Once tool tracking isactivated, real-time roll angle of the TRUS sagittal image are streamed fromthe TRUS control software to the strain imaging software. This allowed usto acquire 3D strain imaging as the surgeon moves the instrument to dif-ferent locations on the prostate surface to examine. As an example, afterthe calibration stage and activation of automatic tracking, for two cases weasked the surgeon to start examining the prostate volume by placing hissurgical instrument on the surface of the prostate from left-lateral bound-ary through to the right-lateral boundary. Our goal in collecting volumetricstrain images was to collect a 3D strain volume that could be compare topost-operative whole mount whole mount histopathology results. Becauseof the limitations in the number of whole mount pathology procedures ourresearch group could order at the time of performing this study, we couldnot get the pathology results for any of the patients in this dataset. Hence,we only relied on qualitative evaluation and feedback from our collaboratingclinicians and quantitative evaluation of our strain imaging system. Threedifferent surgical instruments (Prograsp forceps, Large needle driver and815.4. Human Patient StudiesMaryland Bipolar Forceps) were tested to see the effect of instrument tipsharpness, size and shape on the quality of the strain images. Using thesides of the ProGrasp instrument gave us more smooth and consistent im-ages (shown in Figure 5.9). The reason for this is hypothesized to be thelarger area over which the displacements are applied on the tissue surface.When the sharp tip of the instrument is used, displacements are appliedover a small surface area and would not result in a large and consistentdisplacement (strain) field.In order to evaluate the strain image quality in real-time, to make surethat our strain imaging system is working with high quality, we computedthe average correlation coefficient (ACC) and showed it inside our imaginggraphical user interface. The average correlation coefficient is the index toevaluate how well the tissue motion tracking algorithm is estimating dis-Imaging interface inside the daVinci®S surgeon console ( Tilepro™ ) TRUS image real-time strain image Prostate Calcification area Surgeon console of the daVinci®S system Figure 5.8: Strain imaging interface and results825.4. Human Patient Studiesplacement and strain.Since generating the strain images are being performed by surgical toolmotion, and surgical tool motion is in surgeon’s control, there exist a learningcurve for the user to produce high ACC strain images. By pressing the tissuesurface too hard, the ACC and image quality will drop.Human-in-the-loop strain palpation is intuitive and easy to use by thesurgeons. Instrument based excitation is user subjective and requires somelearning. Strain image interpretation by the surgeon requires some experi-ence and learning.In terms of the evidence for clinical usefulness, there were a few casesthat we could detect tissue structures that were not obvious in the endoscopeor the BMode TRUS images. Some use cases of instrument-based strainimaging for detecting a clinically proven calcification, and prostate boundaryand urethra are shown in Figure 5.12. An imaging sequence inside the daMaryland Bipolar Forceps ProGrasp𝑇𝑀 Forceps Large needle driver Figure 5.9: Three different da Vinci instrument tested for the instrument-based strain imaging. The surgeon was asked to use the longer sides of theinstruments to enable applying displacement over a larger area on the tissuesurface. All of the above pictures are taken when the instrument is pressedon the anterior surface of the prostate.835.4. Human Patient Studies(a)Instrument tip Surgical scene view Registered TRUS Registered Strain (b)Figure 5.10: The imaging system interface inside the daVinci consolein two different configurations in two different patient cases using theTilePro—feature.Vinci surgeon’s console while the surgeon is trying the system is shown inFigure 5.14.After testing the system in this pilot clinical study, we concluded thatwhile this imaging system has some potential clinical benefit to augmentsurgeon’s view of the anatomy with bio-mechanical information, producinghigh quality and useful strain images required a large amount of learning and845.4. Human Patient Studies(a)(b)Figure 5.11: Strain imaging using the surgical instrument tip palpation. (a)intrument tip in pressing motion, (b) instrument tip by the surgeon. Because surgical operation time is an extremelyimportant factor for patient health and surgical performance, surgeons arereluctant to spend a huge amount of time practicing with an imaging systemto learn how to use it. Furthermore, unlike BMode ultrasound and MRIimaging, which are clinically accepted and practiced imaging modalities,strain imaging is a pretty new imaging modality and lacks evidence forcancer detection.With further processing to simulate tooltissue interaction, elastic modelscan be used to synthesize haptic feedback at the da Vinci masters withoutcompromising system stability. The next step for this system is to estimate855.4. Human Patient Studies𝐴𝐶𝐶:  0.931, roll angle: −29°, patient #1 𝐴𝐶𝐶:  0.856, roll angle: 2°, patient #2 𝐴𝐶𝐶:  0.812, roll angle: 15°, patient #5 (𝑎) (𝑏) (𝑐) Figure 5.12: Strain imaging useful information in different cases and differentstructures.tissue stiffness in the prostate volume and present stiffness to the surgeonshands in the form of haptic force feedback.𝐴𝐶𝐶:  0.978, roll angle: −29°, patient #2 𝐴𝐶𝐶:  0.943, roll angle: −29°, patient #2 𝐴𝐶𝐶:  0.984, roll angle: −29°, patient #2 𝐴𝐶𝐶:  0.975, roll angle: −29°, patient #2 Figure 5.13: Detecting blood vessels and pulsatile motion in strain imaging.865.4. Human Patient StudiesFigure 5.14: Strain imaging on the prostate anterior surface using the needledriver surgical instrument.87Chapter 6Multi-parametric 3DQuantitative UltrasoundVibro-Elastography Imagingfor Detecting PalpableProstate TumorsIn this chapter, we describe a system for detecting dominant prostate tu-mors, based on a combination of features extracted from a novel multi-parametric quantitative ultrasound elastography technique. The perfor-mance of the system was validated on a data-set acquired from n = 10patients undergoing radical prostatectomy. Multi-frequency steady-statemechanical excitations were applied to each patient’s prostate through theperineum and prostate tissue displacements were captured by a transrectalultrasound system. 3D volumetric data including absolute value of tissueelasticity, strain and frequency-response were computed for each patient.Based on the combination of all extracted features, a random forest clas-sification algorithm was used to separate cancerous regions from normaltissue, and to compute a measure of cancer probability. Registered wholemount histopathology images of the excised prostate gland were used asa ground truth of cancer distribution for classifier training. An area un-der receiver operating characteristic curve of 0.82 ± 0.01 was achieved ina leave-one-patient-out cross validation. Our results show the potential ofmulti-parametric quantitative elastography for prostate cancer detection for886.1. Introductionthe first time in a clinical setting, and justify further studies to establishwhether the approach can have clinical use.6.1 IntroductionThe use of tissue elasticity as a contrast mechanism to detect prostate tumorshas been suggested in many previous studies, in the area of elastographyimaging [13, 90, 116, 117]. However, most clinical ultrasound elastographysystems are based on a quasi-static tissue excitation, with major drawbackssuch as dependency on operator skill and lack of reproducibility [6]. Hence,an absolute, quantitative elastography technique is highly desirable. Fur-thermore, the majority of tested real-time elastography systems are shownto have a high rate of false-positives [13, 14]. One major reason for this poordetection performance is hypothesized to be the fact that the current clinicalelastography systems are only capable of producing an image that visual-izes a single tissue physical parameter, such as stiffness or compliance, whilecancerous tissues are complex and non-uniform and cannot be characterizedusing only one parameter.Multi-parametric imaging is an emerging technology that combines infor-mation from different techniques, to improve detection rates beyond whatcan be achieved using any single imaging method. Brock et al. assesseda combination approach of ultrasound elastography and contrast enhancedultrasound and showed that the multi-parametric approach decreased thefalse-positive value of real-time elastography alone from 34.9% to 10.3%[14]. Vibro-elastography - the multi-frequency tissue response over a wideexcitation bandwidth [90, 105], as well as tissue nonlinear response as afunction of applied displacements [117], are also shown to contain addi-tional information that may increase the accuracy of cancer detection basedon elastography.In this work, in vivo 3D volumetric data acquired from multi-frequencyquantitative vibro-elastography imaging is analyzed. This is the first re-port of such clinical data. We propose a novel set of features that combinethe B-mode, strain, absolute elasticity, along with the frequency-dependent896.2. Methodsparameters that reveal tissue relaxation time and visco-elastic properties.A supervised classification framework is constructed and used to combinethe multi-parametric features to separate cancerous and normal tissue andcompute a cancer probability map.6.2 MethodsAbsolute Vibro-Elastography:A multi-frequency steady-state mechanical excitation is applied externally togenerate tissue motion. A sequence of nf frames of RF-data is acquired foreach plane in an imaging volume by the ultrasound machine, and processedusing a speckle tracking algorithm [115] to create a series of displacementsper pixel as a function of time. With a linearity assumption, motion at eachpixel has the same temporal frequency content as the input excitation, andtherefore the tissue response can be described using complex exponentials(phasor: pi = Aiexp(jφi)) at each pixel for each frequency fi. A single pha-sor displacement image is generated from nf frames for each plane at eachfrequency and any traveling wave inside the tissue could be revealed fromthis image at each plane. Tissue strain could also be computed from thisphasor image. The waves seen in phasor displacement images are only 2Dprojections of the actual traveling waves created by the steady state externalexcitations. Therefore, 2D phasor images are computed for a series of neplanes creating a 3D volume. The Local Frequency Estimation (LFE) inver-sion algorithm [68] was used here for elasticity computation. This processis repeated for an entire volume producing NE elastograms from Np planes(NE = Np − ne + 1).System Implementation for Prostate Imaging:The main components of our prostate imaging system are depicted in Fig-ure 6.1. A BK ultrasound machine (BK Medical, Herlev, Denmark) with a8848 4-12 MHz biplane transducer was used for imaging the prostate andtissue displacement measurements. Raw In-phase Quadrature (IQ) data was906.2. Methodscaptured at 42.66 Hz sampling rate and saved into an external PC through aDALSA Xcelera-CL PX4 Full frame grabber card (Teledyne DALSA, Water-loo, ON). A previously designed TRUS robot [4] was used to automaticallycontrol the rotation angle of the TRUS transducer and save location infor-mation of each image.To ensure good wave penetration into the prostate in a noninvasive man-ner, we used transperineal excitation similar to the approach used in Mag-netic Resonance elastography (MRE) [88]. An electromagnetic exciter incombination with an Agilent U2761A function generator (Agilent Technolo-gies, Santa Clara, CA) was used to generate desired excitation frequencies.The excitation frequencies used for tissue motion generation in this studyvaried between 58 Hz to 180 Hz. Since we did not have external access to theimage acquisition parameters of the BK ultrasound machine, a band-passsampling algorithm described in [25] was used here for phase and amplitudereconstruction with sampling frequencies that are lower than the excitationFigure 6.1: Main components of the quantitative elastography system withtransperineal excitations916.2. Methodsfrequencies.Patient Data Collection:Ten patients with clinically organ-confined prostate cancer (median patientage: 61 years, range: 52-70 and median baseline PSA: 6.4 ng/ml, range: 4.6-36.4) undergoing robotic radical prostatectomy at our institution agreed toparticipate in this study. Ethics approval for this clinical study was obtainedfrom the UBC Research Ethics Board (H08-02696). For each patient, four tosix volumes of multi-parametric data including time displacements, phasordisplacement and elasticity data were acquired, for a variety of excitationfrequencies. One of the acquired volumes for all patients was at an excitationfrequency of 75 Hz and single frequency features were extracted from it.Data from other frequencies were used to compute frequency dependentparameters.Whole-mount histopathology images of the excised prostate were used asground truth for cancer detection validation. Depending on the size of theTRUS robot BK ProFocus US machine CIVCO Brachytherapy setup TRUS robot Patient in Lithotomy Image processing  and TRUS and excitation control PC Transperineal excitation Figure 6.2: Clinical setting for data acquisition926.3. Data AnalysisFigure 6.3: Example pathology slicesgland, 8-13 slides were obtained, each of which were processed by a patholo-gist who marked the gland boundary, cancer regions and prostate anatomicalzone boundaries as shown in Figure 6.3. Approximately 75% of the canceroccurs in the peripheral zone (PZ) [62]. This zone was also segmented onthe histopathology slides. Figure 6.4(a) shows an example pathology slide,and its corresponding acquired B-mode and absolute elastography imagefrom one patient.6.3 Data Analysis6.3.1 RegistrationTo define regions of interests (ROI) for the classifier data, the acquiredimages should be registered to the pathology images. Bilinear interpolationis performed first to convert the volume in the probe’s curved grid into a 3DCartesian grid, ensuring that orientation of the axial (transverse) imagesmatches the orientation of the pathology slides (Figure 6.5). A slice-to-surface, particle-filter-based registration technique [71] was used to registerthe stack of equispaced 2D pathology contours to the 3D surface extractedfrom the volumetric ultrasound images as depicted in Figure 6.5. To selectareas of interest, a re-slicing tool was used which allows the user to selectan area on the pathology slides, and using the transformation by which theregistration is performed, the corresponding area in the ultrasound volumeis identified.936.3. Data AnalysisFeature Extraction:To define regions of interests (ROI) for the classifier data, the acquiredimages were registered to the pathology images. A slice-to-surface, particle-filter-based registration technique [71] was used to register the stack of eq-Absolute elasticity image Bmode TRUS image Histopathology slice  (a)𝐵𝑖 𝐸𝑖 𝐴𝑖 𝜙𝑖 (b)Figure 6.4: (a) Example pathology images, and their corresponding recon-structed B-Mode and absolute elastography, (b) example slices of four typesof volumetric images available for feature extraction: B-mode (Bi), displace-ment phasor magnitude (Ai) and phase (φi), and absolute elasticity (Ei).(fi = 75Hz)946.3. Data AnalysisBilinear  interpolation Sagittal B-mode image. Sweep angle: −20.7° Sagittal B-mode image. Sweep angle: −4.5° Absolute elasticity image overlaid on Bmode  Absolute elasticity image overlaid on Bmode  Transverse absolute elasticity image overlaid on Bmode  Transverse absolute elasticity image overlaid on Bmode  PZ 3+4 3+3 PZ PZ 3+3 3+4 3+4 3+3 3+3 NPZ NPZ NPZ Stack of Histopathology images in a prostate volume Slice to surface registration Figure 6.5: TRUS to pathology registration for comparison to pathologyimages.uispaced 2D pathology contours to the 3D surface extracted from the volu-metric ultrasound images.For each plane in each volume, four types of images are available forfeature extraction: B-mode (Bi), displacement phasor magnitude (Ai) andphase (φi), and absolute elasticity (Ei). To identify ROIs, regions of inter-est were specified for both Class 1 (malignant cancer) and Class 0 (benignlesion) using the pathology markings which were registered to ultrasounddata. A feature vector was created for each ROI corresponding to a wholetumor or a non-cancerous area. For each of the four data types (Bi, Ai,φi, Ei), seven statistical parameters of the pixel intensities within the ROIwere calculated and used as features. These included the mean, standarddeviation, maximum, minimum, median, kurtosis and skewness. Before ex-tracting the features, histogram normalization was performed on the data956.3. Data AnalysisTable 6.1: Table of features. Physical meaning of the features: Bi :Brightness, Ei : Stiffness, (Ai and φi): strain, FR(FrequencyResponse) :Relaxation-time and ViscosityDatatypeFeatues per ROI IndexBi µB σB MaxB MinB MedB KurtB SkewB 1-7Ei µE σE MaxE MinE MedE KurtE SkewE 8-14Ai µA σA MaxA MinA MedA KurtA SkewA 15-21φi µφ σφ Maxφ Minφ Medφ Kurtφ Skewφ 22-28FRmφ 29mE 30across the data-set to map the intensities to the same dynamic rage for allcases. A feature vector with n = 28 components (described in Table 1) wascreated per ROI, all calculated from images with excitation at 75 Hz.In order to leverage the multi-frequency data for each patient, two fre-quency dependent features were computed for each ROI and added to thefeature vector. The displacement phasor phase (φi) and elasticity (Ei)frequency-response were analyzed for the range of frequencies available foreach patient.Assuming linearity, the tissue displacement transfer function could beformulated as: G(jω) ∼= X(jω)U(jω) = 11+Tjω , where X(jω) is the displacementmeasured at each pixel, U(jω) is the input displacement from external ex-citation, and T is a time-constant. The phase of this transfer function is∠G(jω)=arctan(Tω), which is the same as the computed phasor phase φi ateach frequency. Hence, the slope of a line fitted to the φi frequency-responsewill be T , an estimate of the tissue relaxation time. Such analysis was per-formed for each ROI in our data-set, and the slope of a line fitted to the φifrequency response was computed (mφ) and added to the feature vector.Tissue visco-elastic properties are reported to vary with the input ex-citation frequency [90] and the rate of such variations may yield more in-formation about tissue characteristics. The frequency response of Ei wascomputed for each ROI and the slope of a line fitted to the curve (mE), rateof change of elasticity with frequency, was included as another feature for966.3. Data Analysiseach ROI. Using the combination of all described features, we incorporatedthe texture and intensity features from one common frequency and also usedthe multi-frequency content of the data.Classification:Binary classification between malignant and benign lesions was performedusing random forests [12]. To generate the trees, a random feature vectorθk is selected from all the available features for each tree k. This vectoris independent of the feature vectors for the other trees, but has the samedistribution. The decision tree is grown using the training set and θk; eachnode in the tree is partitioned using the best binary split. Once grown, eachdecision tree yields a classifier h(x; θk) , where x is the input vector. Afterall the independent trees have been grown, to classify an input example,each tree casts a unit vote and the final classification is based on the mostpopular class.Bootstrapping of features was always performed, and the Gini index wasused as the criterion for determining the best binary split. Various otherparameters are available for design and a number of these were optimizedusing a grid-search method: (i) Number of estimators (Ne): the number oftrees in the forest. (ii) Maximum number of features (Maxnf ): these arethe number of features to consider when looking for the best binary split.(iii) Maximum depth of the tree Maxnt: tested values for this parameterranged from no limit to a depth of 15. This parameter is also tree-specific.To perform classifications consistent with the tissue types in the prostategland, features were extracted twice: (i) only from the peripheral zone (PZ)of the prostate, (ii) from the whole gland (WG), since different regions inthe prostate have inherently different elastic properties [62].In order to demonstrate the performance of each group of features, fourclassification experiments with different feature vectors were performed: (i)multi-parametric and multi-frequency experiment (n = 30, feature index:1-30 in Table 1), (ii) multi-parametric single-frequency experiment (n = 28,feature index: 1-28), (iii) multi-parametric and multi-frequency experiment976.4. Resultswithout B-Mode features (n = 23, feature index: 8-30), (iv) single-frequencysingle-parametric experiment (n = 7, feature index: 8-14).For each combination of these parameters, a leave-one-patient-out cross-validation was performed. Benign lesions were the negative class (class 0)and malignant ones were positive (class 1). The accuracy, sensitivity, speci-ficity, precision and area under the receiver operating characteristic (ROC)curve (AUC) was recorded for each of the ten leave-one-patient-out clas-sifications, and then averaged to find the classifier performance for thatparticular set of parameters. Further, this grid search was performed threetimes for each of the three feature vectors described in the Feature Extrac-tion section. A total of three experiments were completed using randomforests, with a unique parameter optimization done for each.6.4 ResultsThe classification results in terms of sensitivity, specificity, accuracy andarea under the ROC curve (AUC) for each experiments are presented inTable 6.2. In plotting the receiver operating characteristic (ROC) curve, avalue of probability=0.5 was used as the cutoff between classes.Comparison of Results in Each Prostate Region:For each of the experiments with different feature groups, the classificationalgorithms were tested once on ROIs extracted from the PZ, and once onROIs from the WG. Results suggest that limiting the analysis to the PZwould consistently lead to better results in terms of AUC, specificity andaccuracy (AUC changes from 0.79± 0.01 to 0.82± 0.01 in experiment (i)).Comparison of Results for Different Feature Groups:Comparing the results of the multi-frequency multi-parametric experimentwith single-frequency single-parameter elasticity imaging shows ≈ 10% im-provement in AUC and specificity in the PZ. Single-frequency single-parameterexperiments represent the traditional single parameter elasticity imaging.986.5. Discussions and ConclusionsComparison between the results of experiment (i) and (ii) shows 4% im-provement in an AUC and 7% improvement in the specificity in the PZ,when multi-frequency features are added to the feature vector. Without us-ing features from B-Mode, the multi-frequency multi-parametric elasticityimaging could yield an AUC of 0.77 (compare the results of experiments (i),(iii)).6.5 Discussions and ConclusionsOn the basis of the receiver operating characteristic curve, a value of prob-ability=0.5 was used as the cutoff between benign and malignant.Table 6.2: Classification results. n: number of features, f-index: featureindex, Zone: prostate region features extracted from, NROI : number ofROIs extracted, Param.: [Ne, Maxnf , Maxnt] for random forest, AUC:area under ROC curve. Results corresponding to PZ are colored in gray foreasier comparison.Random forest classification resultsEx n f-index Zone NROI Param. Acc. Sens. Spec. AUCi 301-30PZ 164 [19, 19, 2]0.72±0.010.61±0.020.82±0.040.82±0.01WG 231 [18, 14, 2]0.67±0.010.63±0.030.74±0.020.79±0.01ii 281-28PZ 164 [17, 18, 2]0.69±0.020.62±0.020.75±0.030.78±0.01WG 231 [17, 19, 2]0.66±0.010.61±0.030.72±0.020.77±0.01iii 238-30PZ 164 [16, 17, 2]0.65±0.020.61±0.020.72±0.030.77±0.01WG 231 [13, 4, 7]0.64±0.020.60±0.040.68±0.020.75±0.02iv 78-14PZ 164 [19, 2, 2]0.64±0.010.63±0.010.69±0.010.73±0.02WG 231 [18, 3, 2]0.64±0.020.64±0.010.63±0.010.70±0.02996.5. Discussions and Conclusions0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 ROC Random Forest over 10 patients 1−SpecificitySensitivity  PZTotal(a)27 6 20 13 1 2 15 16 0 23Feature0. ImportanceFeature Importances Over all Patients(b)Figure 6.6: (a) RF results (b) feature importance from RFPrevious reports on the clinical application of elastography for prostatecancer detection all confirm its usefulness, but also agree on the fact thatsingle parametric elasticity imaging alone is not sufficiently accurate to en-able guidance for diagnosis and treatment. In the published clinical studieswith whole mount histopathology validation, Brock et al., reported an over-all sensitivity and specificity of 49% and 73.6% using shear-wave elastogra-phy [13]. Salomon et al. reported sensitivity and specificity of 75.4% and76.6%, using quasi-static elastography [91]. We show that the combinationof all features (experiment (i)) provides sensitivity of 0.61% and specificityof 0.82% and has a more efficient cancer detection performance than eachmethod individually.Nodular prostatic hyperplasia was observed inside the prostate transitionzone for 80% of the cases. It causes changes in the tissue mechanical prop-erties and could contribute to some of the false positive detections outsidethe PZ. Hence, the elasticity reconstruction is expected to perform betterin the PZ region, leading to more accurate results.Feature importance analysis from random forest classification shows thatKurtφ, KurtA, KurtE have the highest importance rank in the classificationresults. Kurtosis is a measure of “peakedness”of the distribution of theparameters in each ROI. Such results reveal that a dominant tumor typicallyhas a consistent intensity contrast with respect to the surrounding healthy1006.5. Discussions and Conclusionstissue in elasticity and displacement phasor images.A multi-parametric cancer detection framework based on quantitativevibro-elastography imaging was proposed in this chapter. A unique set offeatures were computed based on the acquired data from 10 patients ina feasibility clinical study, and their detection performance was comparedwith traditional single parameter elastography. Promising detection resultsjustify further clinical studies to prove the clinical usability of the system.According to the pathology reports for each patient, there are varyingnumbers of small tumor spots detected inside each prostate which weremarked by the pathologist. According to [62], minimum tumor diameterfor it to be considered as palpable is 7mm. We believe that considering thesmall tumors inside this data-set contributed to some of the false positivedetections in our system.In most of the cases, symmetrical nodular prostatic hyperplasia (showingsecondary cystic atrophy) was observed inside the prostate transition zone.This biological phenomenon causes change in the tissue mechanical prop-erties and could contribute to some of the false positive detections. Also,according to the pathologist, not most of the cancers they marked in theslides were stiffer than the surrounding tissue. They mentioned the softtumor phenomenon is one of the cases.101Chapter 7A Novel Force FeedbackControl Structure forRobot-Assisted LaparoscopicSurgeryDespite the recent widespread success of robot-assisted minimally invasivesurgery, haptic feedback is not yet available in current clinical surgicalrobotic systems. The absence of haptic feedback is a concern for novice sur-geons in the field, especially for execution of force-sensitive surgical tasks.A novel force feedback control framework for two-handed tasks in teleoper-ated robot-assisted surgery (RAS) is presented in this chapter. In bimanualtasks such as suturing and palpation that involve an action and a reactionforce, the applied forces on the environment by the dominant action handare not transferred back to the same hand, but rather to the non-dominanthand using our proposed asymmetric control framework. Such a techniqueprovides an intuitive way of feeling the force, while avoiding destabilizingeffects, since the control loop is not closed from the slave to the master ofthe same hand. The framework has been initially evaluated on an experi-mental setup consisting of four force feedback haptic devices, then on the daVinci surgical system (Standard version) using the da Vinci Research Kit(dVRK) controllers. The da Vinci implementation involves a teleoperationcontroller based on kinematic correspondence, motion scaling and gravitycompensation, as well as torque control for force rendering at the mastermanipulators. A series of user studies involving a small group of both sur-1027.1. Introductiongeons (n = 3) and novices (n = 6) were conducted to evaluate the systemin suture knot tying and haptic exploration tasks. The results show thatthe proposed technique has some promise when implemented on a realisticsurgical robot, but further work is necessary to make the system fully usable.7.1 IntroductionTele-robotic surgical systems have provided significant contributions to thegrowth of minimally invasive surgery (MIS), due to their enhanced precision,range of motion, dexterity and 3D endoscopic visualization offered to thesurgeon. Clinical advantages such as smaller incisions and minimized tissuetrauma translate into reduced wound complications, patient discomfort andrecovery times, making these systems also attractive to patients and thehealthcare industry [48].Despite these successes, current commercially available surgical roboticsystems have limited haptic (tactile and kinesthetic) feedback. This defi-ciency is hypothesized to be a limiting factor in the performance of suchsystems in robot-assisted MIS [77]. Force feedback conveys medically rel-evant information that could improve the performance of a surgical taskin different ways. The surgeon could estimate tissue mechanical propertiessuch as stiffness and texture using a combination of force and motion data,allowing him/her to identify pathologic conditions [60, 76, 113]. Force feed-back can also prevent trauma and incidental tissue damage as it providessurgical tool-tissue interaction forces to the operator [34]. In manipula-tion tasks such as blunt dissection [112] and suturing [86], force feedback isimportant to prevent excessive retraction forces or puncture of the tissue.Furthermore, force feedback can help to reduce the likelihood of breakingsutures from unintentionally large applied forces. Bethea et al. reportedthat even experienced surgeons training with robot-assisted surgery oftenbreak sutures and damage delicate tissue [8]. This is attributed to the factthat high enough forces could not be applied to create knots that are firmenough without breaking the suture or damaging the tissue. Kitagawa etal., in a study done on the da Vinci surgical system (Intuitive Surgical Inc.,1037.1. IntroductionSunnyvale, CA) showed that resolved force feedback would improve robot-assisted performance during complex surgical tasks such as knot tying withfine suture [49]. In a work studying the role of haptic feedback in a bluntdissection task using a telerobotic system, Wagner et al. discovered thatforce feedback reduces the number of errors that damage tissue by a factorof three [112].Recently, there have been a few studies on the effect of alternative formsof feedback on surgical knot tying practice. Visual sensory substitution is re-ported to help surgeons apply more consistent, precise, and greater tensionsto fine suture materials without breakage during robot-assisted procedures[21, 48, 75, 86]. Audio feedback is also proven to be a valuable sensorysubstitution method [21]. Furthermore, some groups have reported integra-tion of real-time ultrasound elastography to the da Vinci surgical system,to compensate for the loss of haptic feedback for palpation purposes [15].However, while such alternative methods are shown to improve the surgi-cal performance of existing robotic systems, they might cause mental orvisual overload for surgeons. Providing realistic haptic feedback will makesurgical robots more intuitive, would naturally facilitate surgical tasks andwould decrease mental workload, leading to better performance. The tech-nical challenges in the application of force feedback in a clinically practicaltelesurgical system could be divided into three categories: (i) integrationof force sensors into surgical instruments (haptic sensing) [31, 34, 47, 79],(ii) techniques to display the sensed force back to the surgeons hands (hap-tic display) [31, 47, 76, 86] and (iii) stable and robust force feedback con-trol architectures. This work is concerned with offering a novel solutionto the third problem. Force feedback teleoperation has been extensivelyaddressed in previous works on bilateral controllers [39], cooperative multi-master multi-slave teleoperation systems [80, 93, 94] and bimanual hapticteleoperation systems [51, 52, 103]. Due to safety and stability requirementsfor surgical telerobots, most of the previous reports either provide a scaledforce feedback or require additional hardware for indirect haptic feedbacknot to the controlling hand.This work introduces a new force feedback control solution for a class of1047.1. Introductioncooperative two-handed tasks in telerobotic surgery. While executing surgi-cal tasks, surgeons commonly use both their hands cooperatively to achieveprecise control. Often, tasks are divided between the two hands. As anexample, in knot tying and remote palpation tasks, one hand would be usedto hold the suture knot or a part of a tissue or an organ, and the otherhand is used to exert tensile/compressive forces. A clinical example of sucha scenario is robot-assisted knot tying performed with the da Vinci surgicalsystem as depicted in Figure 7.1. The proposed control approach takes ad-vantage of the task division between the right and left hands, suggesting thatthe human operators use one hand as the action hand and the other as thefixed/reaction hand, only while performing such tasks. Then the controllerFigure 7.1: Two-handed robot-assisted knot tying in da Vinci RadicalProstatectomy, suture ligation of dorsal veins at prostate apex (departmentof Urological Sciences, University of British Columbia). The division of thetasks between the two hands is demonstrated in this figure.1057.1. Introductionperforms an asymmetric transmission of velocity/force signals from the sur-gical environment to the operator’s hands. The action hand applies forcesto the surgical environment through its corresponding master-slave pair;force/impedance is sensed at the contact point and transmitted back to theother hand, which is holding the fixed master-slave pair. A schematic of thiscontrol framework is shown in Figure 7.2. By implementing such an indi-rect haptic force feedback technique, the action hand will safely perform thesensitive surgical task and is protected from any time-delayed position/forcesignal, while the operator is able to feel the applied forces on his/her otherhand (fixed hand), and adjust them accordingly. Controller instability willnot occur because one hand performs the action and the other receives thefeedback signal.There are many ways to implement this basic idea; in this work wepresent a variable structure controller to implement this concept and testits applicability. The user can switch between three control modes dependingon the task: one structure for normal bimanual unilateral control, and theother two for the asymmetric control for tasks in which either the right orleft hand could be used as a fixed hand while the other is performing theaction.To illustrate the feasibility and force-reflecting performance of the pro-posed controller, initial experiments have been performed using a two-master/two-slave setup composed of four 3-DOF haptic devices, and the preliminaryresults were presented in [4].In this work, we address a much more realistic scenario, involving a daVinci surgical system. The da Vinci Research Kit (dVRK) controllers wereused to implement this control framework on the first generation of theda Vinci surgical system (da Vinci standard). Such controllers are ”opensource mechatronics” platforms consisting of hardware, firmware and soft-ware components [17, 46] and are being installed at multiple centers formedical robotics research (see dVRK wiki: [1]). We integrated the con-trollers with the da Vinci classic system in our lab, established a biman-ual teleoperation control structure based on the components from the opensource cisst/SAW libraries [17] and implemented the asymmetric force feed-1067.1. Introduction(a) (b)Figure 7.2: The asymmetric control framework schematic described for atwo-master two-slave platform performing common two-handed tasks: (a)suture knot tying: tensile force being applied by the action hand and thereaction force being transferred back to the reaction hand, (b) palpation:compressive force being applied by the action hand and the reaction forcebeing transferred back to the reaction hand.back control framework. Using the dVRK controller with the da Vinci clas-sic enabled us to conduct user studies in a reasonably realistic da Vincisurgery environment, where users sit at the surgeon’s console and work un-der stereo-vision through a high resolution stereo-viewer, while using ourcustom control algorithms.The rest of this chapter is organized as follows. The architecture of theproposed controller in terms of general master-slave network parameters isdescribed in Section II. In Section III, the method implementation on theda Vinci surgical system is described. Description of the user studies, theresults and discussions are presented in Section IV.1077.2. Asymmetric Control Architecture7.2 Asymmetric Control ArchitectureThe force feedback teleoperation controller design problem poses stringentrequirements for performance, stability and robustness. Figure 7.3(a) andFigure 7.3(b) show a typical control block diagram and two-port networkmodel of a straightforward and commonly used method for creating unilat-(a)(b)(c)Figure 7.3: (a) Uni-lateral control structure for bimanual teleoperation,analogous to a simplified version of the da Vinci robot control structure,(b) Two-port network model of the uni-lateral bimanual teleoperation con-trol with a shared environment.1087.2. Asymmetric Control Architectureeral teleoperation. It is assumed that the operator’s right and left handsare holding the right and left master manipulators, and their correspondingslave robots are in contact with an environment/object that is shared be-tween them. In these block diagrams, the left and right hand dynamics, theenvironment, and the left and right master and slave dynamics are lumpedinto linear time-invariant (LTI) impedances ZhL, ZhR, Ze, ZML, ZMR, ZSL,ZSR, respectively. FhL, FhR, FSL, FSR are the forces at the left and righthands and left and right slave robots, F ∗hR and F∗hL are the forces appliedby the human hand, Fe is the force between the right and left slaves andthe environment, VML, VMR, VSL, VSR, VhL and VhR are the velocities ofthe left and right master and slave robots and hands respectively, all in theLaplace domain. The master devices continually send their positions to theircorresponding slaves, and each slave robot uses its local PD controller (CSRand CSL) to ensure position/velocity tracking to achieve kinematic corre-spondence both in free and in contact motions. In such a control structure,energy exchange is only uni-lateral, from human hands to master devices,E(h−m)R and E(h−m)L, from master devices to their corresponding slave ma-nipulators, E(m−s)R and E(m−s)L, and from slave robots to the environmentin contact, E(s−e)R and E(s−e)L (shown in the block diagram of Figure 7.3).Realistic scenarios include sources of energy leaks (ELoss) or an out-of-phaseenergy transfer because of the delayed communication channel. In the uni-lateral teleoperation case, since the overall control loop is not closed overthe communication channel, there is no energy exchange between masterand slave robots, and these manipulators are not subject to instability. Ifimpedance control is used to cause the slave robots to track the masters andvice versa, it will be position exchange (P-P) bilateral control and the hu-man operator will receive haptic force feedback. If the remote slave robotstrack the masters, but force sensors on the remote robots are used to deter-mine the displayed forces, it will be position forward/force feedback (P-F)control. The P-P control structure is well suited for the master and remoterobots that move freely when the user or environment pushes on the end-effector (similar to the da Vinci surgical robot). However, the user mightfeel some forces during free space motions because of the inertia and fric-1097.2. Asymmetric Control Architecturetion in the slave robots, especially for large surgical robots. If the dynamicslave forces are too strong, one could suggest lowering the controller gains,but this softens the feel when the slave robot encounters contact surfaces.Nevertheless, due to the cost and fragility issues with force/torque sensing,the P-P controllers are the preferred structure if either the master or theremote robot does not have large friction or inertia (very large or very smallrobots) [76].Both the P-P and P-F control structures are also subject to low stabil-ity margins in common cooperative surgical tasks, especially when there isscaling involved. This is due to a variety of issues such as unmodeled dy-namics, transport or time-delay or changing environment conditions. Hence,they are difficult to integrate in the currently available telerobotic surgicalsystems. The controller proposed in this work is an alternative and safeapproach to enable force feedback for a category of common two-handedsurgical tasks such as robotic knot tying and remote palpation (Figure 7.2and Figure 7.4).Since human operators use both hands cooperatively to perform suchtasks, we suggest using one hand to apply the tensile/compressive forces(action hand) and the other to hold the suture knot or the object undercompression/tension (fixed/reaction hand). The control strategy is struc-tured in such a way that the forces applied to the surgical environment bythe action hand, through the action master-slave robot pair, get transferredback to the other hand, which is holding the other master-slave robot pair.The control loop is closed through the human operators body, that couldisolate the action hand from the forces that are fed back to the fixed hand.The method is applicable to both position exchange bilateral controland position forward/force feedback control as shown in Figure 7.4(a) andFigure 7.4(b). Both position and force signals could be transferred backasymmetrically using this method. In this work, the asymmetric positionexchange strategy (Figure 7.4(a)) is implemented.1107.2. Asymmetric Control Architecture(a)(b)(c)Figure 7.4: (a) Asymmetric position forward force feedback control struc-ture, (b) Asymmetric position forward position back control structure, (c)Two port network model of the bimanual teleoperation control showing theasymmetric structure.1117.2. Asymmetric Control ArchitectureIn the case when the left hand is fixed and the right hand is perform-ing the suture/tissue manipulation and applying the force (Figure 7.4(b)),the right (action) master robot’s velocity VMR is transmitted to its cor-responding slave robot, and the controller at the slave side CSR ensuresvelocity/position tracking for this master-slave pair. The velocity trackingerror is fed back by the controller of the other master robot to reproduce thesame force and mechanical impedance that the moving slave is experiencingin the environment. In addition, the controller at the fixed slave robot isresponsible to force this manipulator to stay fixed at the surgeon’s desiredposition, i.e. the position it was at the time of the controller switching fromunilateral to asymmetric. The same scenario happens when the right handis executing the action and the left hand is fixed.To ensure position and force tracking and allow asymptotic stability asit is proved in [74], PD controllers with damping injection are used forcontrollers CSL, CSR, CMR and CML. The energy exchange between themoving slave robot and the slave environment (Es−e) gets transmitted backto the fixed master manipulator, and then back to the fixed hand Eh. Hence,the human operators will be able to control the energy/force they exert byone hand through feeling it on their other hand. In many cases, this is avery intuitive approach in which the user feels the reaction as opposed to theaction forces. For example, when tying a knot, one hand pulls the suturewhile the other holds the thread. Tissue could be palpated in the samemanner. The metaphor used by the user is that of holding an object withone hand while feeling it being probed by the other hand.The energy flow and control loop is closed through the human opera-tor’s body; one hand is the energy sink (fixed hand), the other hand is theenergy source (action hand), and the human brain acts as the controllerof this loop to adjust the energy balance. In addition, the possible out ofphase energy loss due to the delay in the communication channel could becompensated by the human operator’s brain. Currently, most expert robot-assisted surgeons only rely on visual cues such as tissue deformation causedby tension/compression forces, and use their brain to correlate forces anddeformations. Addition of our indirect force feedback system will poten-1127.3. Controller Implementation on the da Vinci Surgical Systemtially help them perform a more precise and realistic correlation and forcereconstruction, leading to better overall performance.7.2.1 Stability and TransparencyThe performance objective in force-feedback teleoperation is to have a highstiffness robot to feel material properties such as surface texture while hav-ing low inertia and low friction to reflect small environment forces [76].The application of our proposed control framework will allow using highgain controllers to achieve the above performance objective without losingstability.To learn about transparency, the conditions of kinematic correspondence(Vh≡Ve), and impedance matching (Zth≡Ze) have been examined in [67].In two-handed teleoperation systems and in cooperative tasks such as knottying and palpation, the goal would be to match the mechanical impedancebetween the two slave robots with the mechanical impedance that the hu-man operator is feeling between his/her two hands. As an example, whenthe operator is pulling the suture to tie the knot, the same tensile forceand compliance between the slave robots should be transferred back andfelt between the two master devices. Furthermore, when the operator iscompressing tissue between the two slave robots to get their mechanicalproperties, they should perceive the mechanical impedance of the tissue.7.3 Controller Implementation on the da VinciSurgical SystemAn overview of the dVRK platform integrated with the da Vinci classic sur-gical system is shown in Figure 7.5. Descriptions of the system architecture,various components and their installation on the da Vinci system will beexplained in the next sections.1137.3. Controller Implementation on the da Vinci Surgical System7.3.1 System ComponentsHardwareThe mechanical hardware is the first-generation da Vinci robot with two8-DOF Master Tool Manipulators (MTMs), three 7-DOF Patient Side Ma-nipulators (PSMs), one Endoscopic Camera Manipulator (ECM), a highresolution stereo viewer with a stereo endoscope, and a foot pedal tray. Thefoot pedal tray includes five switches accessed using the foot (Figure 7.5b) and is employed in our custom control architecture as a user-controlledswitch between different control modes. Control electronics include fourcontroller boxes (developed at Johns Hopkins University in collaborationwith Intuitive Surgical Inc. [17, 46]) each containing two pairs of customdesigned boards: (1) an IEEE-1394 FPGA Control board, and (2) a QuadLinear Amplifier (QLA) board. Using FPGAs as the control processing unitsat the hardware side enables a centralized computation and distributed I/Oarchitecture in which high level control algorithms could be implementedon a Linux PC, while high-speed I/O is performed through an IEEE-1394aserial network (max 400 Mbits/sec). The Quad Linear Amplifier is designedto interface with the FPGA board and provides the ability to control up tofour DC motors. Control boxes shown in Figure 7.5c contain two sets ofFPGA and QLA boards each, and are capable of controlling up to eight DCservo-motors which makes them suitable to control da Vinci PSM/MTMmanipulators. The interface to all the electronics, joint level motion sen-sors and actuators in MTMs and PSMs is through DL156 pin connectors(Figure 7.5 d).Software ArchitectureBased on the centralized computation and distributed I/O architecture [46],all computations including data read/write, joint level servo loop controland high level robot control are implemented on a Linux PC. Interface be-tween the FPGA controllers and high level robot control software (throughthe IEEE-1394a protocol) is established with a low-level C++ API. For1147.3. Controller Implementation on the da Vinci Surgical SystemFigure 7.5: (a) The overview of the da Vinci Research Kit platform installedin UBC RCL Laboratory. (b) The da Vinci classic console with a high reso-lution stereo viewer, two MTMs and a foot pedal tray. (c) 8-axis controllerpackages each responsible for controlling one da Vinci MTM or PSM manip-ulator. (d) The connections from the da Vinci system to the control boxesthrough zero insertion force DL156 pin connections and break-out boardsprovided by Intuitive Surgical. Controllers are connected in a daisy-chainstructure using IEEE-1394a bus external and internal to the boxes. (e) Twoda Vinci PSMs and the Endoscopic Camera Manipulator arm with the stereoendoscop e.the high level robot control and algorithm implementations, a component-1157.3. Controller Implementation on the da Vinci Surgical Systembased control architecture using the open-source cisst/SAW libraries wasemployed [17].7.3.2 Bimanual Teleoperation Controller ImplementationIn order to establish a teleoperation system with one MTM-PSM robot pair,software modules for I/O level data read/write, joint level servo PID controland logical robot control have been defined and a teleoperation componentbased on kinematic correspondence has been implemented. The I/O levelmodules communicate with the robot sensors and actuators directly throughthe IEEE-1394 connection to provide PSM I/O level interface, MTM I/Olevel interface and an interface for foot pedal events. Upper level control isprovided by the PSM/MTM PID servo controllers that connect to the PSMand the MTM I/O level interfaces, respectively. The teleoperation modulecommunicates to each of the arms’ joint level PID control routines as wellas to the foot pedal interface for control state switching based on the footpedal events.Two types of robot objects, one for each MTM and one for each PSM,have been defined to handle forward and inverse kinematics, and robot-specific higher level control. Forward and inverse kinematics as well as therobot specific control is achieved through manipulator calibration files (con-taining DH parameters, joint range and limits, etc.) provided by IntuitiveSurgical Inc.Furthermore, a gravity compensation component has been implementedon the MTMs using the Recursive Newton Euler Algorithm (RNEA) [71].Given the joint position and velocity variables, RNEA calculates the setof joint torques to produce the required acceleration to compensate for themanipulator dynamics. RNEA is a model-based approach to start with andis subject to modeling uncertainty, especially due to friction in the cabledriven mechanisms of the MTMs.The two-MTM two-PSM teleoperation control structure shown in Fig-ure 7.6 was established by expanding the single pair teleoperation moduleto interface with I/O and PID level control components of four arms and1167.3. Controller Implementation on the da Vinci Surgical Systemimplementing one to one kinematic correspondence between each robot pair.In this block diagram, the left and right hand dynamics, the environment,and the left and right master and slave dynamics are lumped into lineartime-invariant (LTI) impedances ZhL, ZhR, Ze, ZMTML , ZMTMR , ZPSM1 ,ZPSM2 , respectively for all seven actuated degrees of freedom of the manipu-lators (i.e. ZMTM = {ZJ1, ZJ2, ZJ3, ZJ4, ZJ5, ZJ6, ZJ7}). τhL, τhR, τMTML,τMTMR, τPSM1, τPSM2 are torques at the left and right hands, left and rightrobot arms, τ∗hR and τ∗hL are the torques applied by the human hands andτGC is the torque on MTMs from the gravity compensation component, allin 7-DOF (i.e. τGC = {τJ1, τJ2, τJ3, τJ4, τJ5, τJ6, τJ7}). VMTML , VMTMR ,VPSMR and VPSML are the velocities of each manipulator in 7-DOF com-puted by joint motion sensors. CGC is the gravity compensation controllerat each MTM and CPSM1 , CPSM2 are the joint level PID controllers at eachPSM responsible to ensure position (rotation/translation) tracking. All ofthe joint motions and torques are transmitted to their corresponding localcontroller through DL156 pin connectors and then to the central controlprocessing unit through the IEEE-1394a bus. All of the controllers are im-plemented in the centralized computation and distributed I/O architectureas shown in Figure 7.6. Control loop frequencies of 1 kHz for the I/O level,333 Hz for the kinematics and robot logic control, and 200 Hz for the tele-operation loop could be achieved while having stable data synchronizationbetween various components of a two-MTM two-PSM teleoperation con-troller.The foot pedal buttons were employed to switch between various controlstates. The COAG button is used as a toggle switch to activate and de-activate the teleoperation with gravity compensation mode. Furthermore,the clutch button is used to activate the clutching mode in which MTMs’movements are not reflected to the their PSM side, to reposition the MTMswhen needed. Motion scaling (PSM:MTM: 1:5, 1:3, 1:2) for each MTM/PSMpair is also implemented to increase the accuracy of the task performance,similar to the original da Vinci robot controller.Figures 7.7 shows the Cartesian positions of each MTM/PSM pair indifferent control states with a motion scaling of 1:5. The master devices1177.3. Controller Implementation on the da Vinci Surgical Systemcontinually send their positions (rotation/translation) to their correspondingslaves, and each slave robot uses its local PID controller CPSM1 and CPSM2to ensure position/velocity tracking to achieve kinematic correspondenceboth in free and in-contact motions.7.3.3 Force Feedback Controller ImplementationThe asymmetric control framework depicted in Figure 7.2 is implementedusing the control structure shown in Figure 7.6. CMTM is the joint level con-troller at the MTM side that is responsible to produce the feedback torqueτFB on the MTMs based on the position tracking errors on the PSM side, inFigure 7.6: The bimanual teleoperation framework. Solid lines show thecontrol structure between the components and the dashed lines show thecommunication structure for signals transmitted between the local hardwarecontrollers, centralized control unit (Linux PC) and robot manipulators.Red lines and controllers CMTM at each MTM are responsible to generatethe feedback torque on each hand according to the control state in the forcefeedback structure (based on the foot pedal events). The IEEE-1394 daisy-chain configuration between the central PC and four dVRK controllers isalso shown in the center boxes.1187.3. Controller Implementation on the da Vinci Surgical System0 50 100 150−0.2− tracking MTML−PSM2, Motion scale (1:5)Position X(m)  MTMPSM0 50 100 150− Y(m)  0 50 100 150−0.4−0.3−0.2−0.10Time(s)Position Z(m)  Start Teleop Mode(COAG pressed)ClutchModeClutchMode In−Contact(Spring pulling)In−Contact(Spring pulling)(a)0 50 100 150−0.1−0.0500.050.1Position tracking MTMR−PSM1, Motion scale (1:5)Position X(m)  MTMPSM0 50 100 150−0.500.5Position Y(m)  0 50 100 150−0.4−0.3−0.2−0.10Position Z(m)  Start Teleop Mode(COAG pressed)ClutchModeClutchMode In−Contact(Spring pulling)In−Contact(Spring pulling)Time(s)(b)Figure 7.7: The bimanual teleoepration system performance: (a) Cartesianposition tracking for the left MTM-PSM pair, (b) Cartesian position track-ing for the right MTM-PSM pair. Different control states (teleoperation,clutched, fixed) based on the foot pedal events could be seen in the specifiedtime intervals. PSM-MTM Kinematic correspondence with motion scalingcould be seen in the teleoperation intervals. PSM-fixed MTM-moving con-trol state could be seen in the clutch mode intervals. A tension spring wasbeing manipulated between the surgical tools in the in-contact intervals.1197.3. Controller Implementation on the da Vinci Surgical System7-DOF (τFB = {τ1, τ2, τ3, τ4, τ5, τ6, τ7}). The control strategy is structuredin such a way that the forces applied to the surgical environment by the ac-tion hand, through the action master-slave robot pair, get transferred backto the other hand, which is holding the other master-slave robot pair. Inthis work, the asymmetric position exchange strategy (position forward foraction MTM-PSM pair, position backward for reaction MTM-PSM pair) isimplemented. In the case when the left hand is fixed and the right handis performing the suture/tissue manipulation and applying the force, theright (action) MTM motion (translation/rotation) is transmitted to its cor-responding PSM, and the controller at the PSM side CPSM ensures positiontracking for this MTM-PSM pair. The position tracking error is fed backto the controller of the other master robot to reproduce the same force andmechanical impedance that the moving slave is experiencing in the environ-ment. In addition, the controller at the fixed PSM is responsible to force thismanipulator to stay fixed at the surgeon’s desired position, i.e. the positionit was at the time of the controller switching from unilateral to asymmetric.Figures 7.8a and b show Cartesian positions of both MTM/PSM pairsdescribing the system behavior for the complete force feedback and nor-mal teleoperation scenarios in which the user holds a tension spring withone robot tool (fixed MTM/PSM pair), and applies a pulling action withthe other hand (Figure 7.11b) to explore the spring stiffness while receivingforce feedback on the fixed hand. Figure 7.8(c) demonstrates the energyexchange in the system for the same spring pulling experiments. The totalcontrol effort or energy at each manipulator was computed by a summationof the control efforts at each joint. Three springs were tested in this exper-iment (KSP1 < KSP2 < KSP3). The action and reaction motions and theircorresponding control effort on all four arms could be seen in each time in-terval specified on these figures. Figure (c) shows the proportionality of thecontrol effort generated on the reaction MTM with the spring stiffness at thePSM side. MTML SP1 means that the MTML is the receiver of the forcefeedback while PSM2 is fixed for holding spring 1 (SP1), and MTMR-PSM1is performing the pulling action. The same scenario happens for other inter-vals. The user switches the force feedback control state between right and1207.3. Controller Implementation on the da Vinci Surgical Systemleft MTMs for each spring in this experiment. The surgeon operators areable to control the energy/force that they exert on the surgical environment0 50 100 150 200 250−0.3−0.2−0.100.1Position tracking MTML−PSM2, Motion Scale: (1:5), Feedback gain: 100Position X(m)  MTMPSM0 50 100 150 200 250− Y(m)  0 50 100 150 200 250−0.3−0.2−0.100.1Time(s)Position Y(m)  MTML SP2MTMR SP2MTML SP3MTMLSP1 MTMRSP1MTMR SP3(a)0 50 100 150 200 250−0.1−0.0500.050.10.15Position tracking MTMR−PSM1, Motion Scale (1:5), Feedback gain: 100Position X(m)  0 50 100 150 200 250− Y(m)  0 50 100 150 200 250−0.25−0.2−0.15−0.1−0.050Time(s)Position Z(m)  MTMPSMMTMLSP1MTMRSP1MTML SP2MTMR SP2MTML SP3 MTMR SP3(b)0 50 100 150 200 250−250−200−150−100−50050100150Total Control effort MTMR−PSM1Time(s)Effort (N.m)  MTMRPSM10 50 100 150 200 250−200−150−100−50050100Total Control effort MTML−PSM2Effort (N.m)  MTMLPSM2MTMLSP1MTMRSP1MTML SP2MTMR SP3MTML SP3MTMR SP2MTMLSP1MTMR SP3MTML SP3MTMR SP2MTMRSP1MTML SP2(c)Figure 7.8: The Cartesian positions of both MTM-PSM pairs and totalcontrol effort on each manipulator in a spring stiffness exploration task.1217.3. Controller Implementation on the da Vinci Surgical Systembecause of the energy transfer from the moving slave gets transferred backto the fixed master manipulator and thus the fixed hand of the surgeon. Theaction and reaction pairing can be intuitive during such tasks as knot tying,when the action hand pulls the suture and the reaction hold the thread tofeel the strength of the pull. Tissue can be manipulated as well, where thereaction hand holds the tissue steady and the action hand applies pressureto gauge the tissue stiffness. With the action (moving) hand as the energysource and the reaction (fixed) hand as the energy sink, the energy flow andcontrol loops are closed through the human operators body. This means thatthe human brain can be used to adjust the system in order to maintain theenergy balance. Currently, most expert robot-assisted surgeons only relyon visual cues such as tissue deformation caused by tension/compressionforces, and use their brain to correlate forces and deformations. Additionof our indirect force feedback system will potentially help them perform amore precise and realistic correlation and force reconstruction, leading tobetter overall performance. The force feedback controller CMTM gain wasadjusted based on the user experience and stiffness of the materials beingmanipulated. Higher gains will produce more torque τFB on the user’s handsbased on the mechanical impedance at the PSM side. The control structureswitching is implemented using the buttons on the foot pedal tray. Once theuser is in the teleoperation mode, she can use the foot pedal buttons to haveher right or left hand as the receiver of the force feedback while the otherhand is in the normal teleoperation mode performing the pulling/pushingaction. Motion and impedance scaling could be implemented in such a con-trol structure without instability issues, because the control loops are notclosed from one slave robot to its corresponding master.1227.3. Controller Implementation on the da Vinci Surgical System0 20 40 60 80 100 120 140 160 180 200−0.15−0.1−0.0500.05Position tracking MTML−PSM2, Motion Scale (1:5), Feedback gain: 100Position X(m)  MTMPSM0 20 40 60 80 100 120 140 160 180 200− Y(m)  0 20 40 60 80 100 120 140 160 180 200−0.2−0.15−0.1−0.050Position Z(m)  Time(s)Force feedback on MTML(a)0 20 40 60 80 100 120 140 160 180 200−0.0500.050.10.15Position tracking MTMR−PSM1, Motion Scale (1:5), Feedback gain: 100Position X(m)  0 20 40 60 80 100 120 140 160 180 200− Y(m)  0 20 40 60 80 100 120 140 160 180 200−0.25−0.2−0.15−0.1−0.050Time(s)Position Z(m)  MTMPSMForce feedback on MTML(b)100 120 140 160 180−25−20−15−10−50510152025Total PID effort MTMR−PSM1 (Action pair)Time(s)Effort (N.m)  100 120 140 160 180−6−4−202468Total PID effort MTML−PSM2 (Fixed pair)Time(s)Effort (N.m)  MTMRPSM1MTMLPSM2(c)Figure 7.9: Four robot manipulator motions and the control effort on eacharm in a knot tying experiment.1237.4. User StudiesFigures 7.9a and b show Cartesian positions of both MTM/PSM pairsdescribing a two handed knot tying experiment in which the users used thenormal teleoperation control to manipulate the suture to make the knotand used the force feedback controller to tie the knot (Figure 7.10). Theenergy exchange plot (Figure 7.8 c) shows the amount and pattern of theenergy transferred back to the fixed hand while the other hand is performingthe pull action on the knot. Figure 7.8 c shows the knot tying with forcefeedback section of the experiment, in which MTMR-PSM1 is performing thepulling action, while MTML-PSM2 is being fixed to hold the knot. MTMLis receiving the feedback while PSM2 is trying to be fixed but is pulled bythe suture between the surgical tools. The same pulling motion pattern inPSM1 (consecutive pull and release motion) could be seen in the feedbackpattern on MTML, proving the impedance matching property of the system.MTMR is working under gravity compensation in a very low energy state.Tasks were performed under the stereo endoscope where the location ofthe arm and ECM setup joints were manually adjusted. Tool to cameraregistration was using the Read-API of da Vinci system by reading theforward kinematics of the camera tip and each PSM setup joints with respectto the robot base, transforming the PSM tool end frames first to the robotbase and then back to the camera tip.7.4 User Studies7.4.1 MethodsTwo tests were undertaken to test how this new system could be used duringsurgical tasks. Nine users (3 surgeons and 6 novices) participated in twodifferent tests that utilize the force feedback system developed for the daVinci system. A variety of users were used such that we could see if thehaptic feedback would be useful to an experienced surgeon, and if it wouldchange the way a novice user performed the task. Each user was giveninstruction on how to use this force feedback system, and given time tobecome familiar with how the feedback feels when manipulating suture and1247.4. User Studiestesting springs. All user tests were completed using a motion scaling factorof 1:5 and force feedback gain of 100. Large needle drivers were used duringthe course of the tests as these instruments are most likely to be used whilesuturing. This system is independent of the tool type and can be used withany type of da Vinci tool. The first test required the users to differentiatethree springs of varying stiffness to simulate a haptic exploration task. Thesprings were covered such that the users could only see a string tied to eachend, minimizing the possible use of visual cues. They were instructed to pullon each spring using the force feedback to determine the relative ranking,from stiffest to weakest. The three springs have constants of 1.2, 0.2 and0.03 N/m respectively. The springs were then reordered and the users askedto repeat the experiment. Each correct relationship between the springs wasgiven one point, for a total of 12 points.The second test required the users to tighten the first throw of a sutureknot (Figure 7.10). They were first asked to tighten the knots by hand,then using the da Vinci without force feedback and finally using the forcefeedback system. During each trial, the users were asked to tighten the knotand hold this force for 3 seconds, and this was repeated 5 times. Load cellswere attached to the ends of both suture sections, such that the force appliedwith each hand could be measured during the test. These 5 sections weresegmented and used during the data analysis. During the force feedbackvariant, the left hand was fixed and the right hand moving to tighten thesuture.7.4.2 ResultsThe average score for all users during the spring differentiation was 98%,106/108 correct relationships. Only a single user made one mistake duringthe task. All users felt that they were able to differentiate the springs withconfidence.The standard deviations during the knot tightening exercise were ex-amined, and there is some evidence that tightening with the robot is lessconsistent than tying by hand. We looked at how consistent the force was1257.4. User StudiesFigure 7.10: Tightening the first throw of a knot. Two load cells at bothsides were used to measure the force exerted by each hand/robot. The userstudy was done under stereo-vision system of da Vinci.between pulls on the suture, and how well the users could hold a given force.We were interested in seeing if the users force would drift over time if there1267.4. User Studieswas no feedback. Without force feedback, users must rely only on visualcues to determine if the knot is tight, leading to a less consistent force ap-plied since it is difficult to differentiate degrees of force once the knot isreasonably tight.The difference between hand tying and robot tying shows that the im-plementation of force feedback system is warranted. Figure 7.13 shows thestandard deviation of the average force applied during the five tighteningsessions. In all but one user, the force applied by hand had a smaller stan-dard deviation than that applied by the robot. In one case, user #3, usingthe robot was more consistent than tying by hand, but this user has morehours tying sutures with the robot than by hand.Figure 7.12 shows the graphs of the one user during each of the three(a)(b)Figure 7.11: (a) A user practicing with a spring to get a feel for the forcefeedback, (b) springs with different stiffness for the haptics exploration test.1277.4. User Studies0 5 10 15 20 25 30 35 40 450123Knot tying by handForce (N)  0 5 10 15 20 25 30 35 40 45 500123Knot tying with the da Vinci robotForce (N)Time (s)  20 30 40 50 60 70 8000.511.52Knot tying with the da Vinci with Asymmetric force feedbackForce (N)  Right handLeft handFigure 7.12: The forces applied to the suture by the left (red) and right (blue)hand during the the variants of the suture tying task. The consistency ofthe applied forces were improved in the force feedback variant compared tothe robot only variant.variants, tightening by hand, with only the robot and with the force feed-back. The blue is the force applied by the right hand, and the red, thatapplied by the left hand. During the force feedback variants, the left handwas fixed and the right hand moving to tighten the suture.7.4.3 DiscussionAll users who participated in these tests, especially the surgeons, were ex-cited about the possible inclusion of force feedback into the da Vinci robot.We chose to use two tasks for our experiments that mimic tasks in a surgicalenvironment that would most benefit from the inclusion of force feedback,1287.4. User Studies0 1 2 3 4 5 6 7 8 9 1000.511.522.5Standard Deviation of applied force for each userUserForce(N)  handrobotFigure 7.13: The standard deviation of the mean forces applied during eachtightening of the suture for the hand tying and robot tying tasks. Resultsdemonstrate the decreased consistency of the applied forces in the robottying vs. hand tying, supporting the need for haptic force feedback.that of tightening sutures and testing tissue for stiffness. It is sometimesnoted that the 3D vision provided by the stereo camera can compensatefor the lack of force feedback, but this is not the case when dealing withstiff objects such as needles and suture where excessive forces do not have avisual feedback component.The force applied during the use of force feedback varied and the correctuse of the system depended heavily on the user’s familiarity with the useof the robot and understanding how the force feedback was applied. Theusers who were able to achieve the most consistent results were those whohad both used the da Vinci and had a good understanding of the feedbacksystem. This method of force feedback combines the force felt on one handand the movement made by the other and it takes some time to understand.We believe that with additional practice with the system the users’ accuracy1297.4. User Studiesand consistency would increase.We did see some drifting of the force the users applied when using the daVinci without force feedback but not sufficient to be significant comparedto that of a hand tie. We did notice that in some users, the difference inforce applied by the two hands when using the da Vinci, in comparison tothe hand tie, was very large. This could be due to the fact that once theknot reaches a certain tightness it is hard to discriminate visually furtherforce applied.During these tests we also became aware of some areas of improvement.The users felt the position of the master was not consistent, even after allforce was released. With repeated tests of the force, the master positiontended to drift. This could be due to the way that the position is measured.It was noted from several users during the knot trials that the systemchanges slightly the manner in which they would tie a knot. Typically theuser would pull with both hands with somewhat equal force. This is morenatural and also centers the knot as it is pulled tight. Since one hand isfixed during the use of force feedback, the knot needs to be placed in thecorrect location first, then the feedback only engaged to tighten the knotfully.Further user testing with a more realistic set up and additional usertraining will provide a better understanding of how the system can be usedduring surgical tasks and where additional improvements can be made.One improvement that we hope to make in the future is the way in whichthe applied force is measured. Currently, we look at how the position of theend effector changes as the force is applied. But due to the robot impedance,some directions of movement are much easier than others. In the future weplan to adapt the method of force sensing to be more versatile and accurate.The body wall and cannula disturbances could also be simulated in futurework to see how that affects force estimation using position tracking errors,as well as the effects of such disturbances on the user.The other implementation issue was using the foot pedal to change con-trol states. Some users suggest that this would be counter intuitive anda smart switch based on voice control or methods being used for surgical1307.4. User Studiesskill evaluation to extract trajectories and patterns from surgeons’ move-ments could be used for real-time automatic identification of the action andreaction hands.131Chapter 8ConclusionsIn this thesis we have developed image and haptic guidance systems forrobot-assisted laparoscopic surgery. The main focus was to develop systemsthat could be translated into clinic. We have also spent a huge amount ofeffort developing protocols and performing clinical studies, to show evidenceof clinical benefit and also to use the acquired data to further optimize oursystems.Our conclusion for the TRUS-guidance study is that real-time, roboticTRUS guidance during RALRP is feasible and safe, and it provides theconsole surgeon with valuable guidance. The results of this study justifyfollow up studies comparing outcomes of RALRP with and without regis-tered TRUS guidance. In a future clinical study involving high number ofpatients (between 100 to 500), we can show the real clinical benefit of thesystem. In addition, performing a multi-center study with different surgeonswith different expertise would further clarify the usefulness of our system.Our conclusion for the MR-guidance study is that it could be a powerfultool to trade-off positive surgical margins against potency. This is the firstclinically tested MR-guidance system for surgery. We have received a verypositive feedback from surgeons on clinical usability of the system. In future,the system should be tested in a clinical study with minimum 20 patientsundergoing da Vinci Prostatectomy and preoperative MR. Such an studywould show the clinical usability of the system in different patients withdifferent anatomical challenges and prove it’s usefulness. Furthermore, newand improved MR-TRUS registrations could be tested with the developedframework.In chapter 4, we have developed an algorithm to automatically localizesurface fiducials in 3D ultrasound. This algorithm is used to enable au-132Chapter 8. Conclusionstomatic registration of coordinate frames based on the air-tissue boundaryregistration concept. While we used this algorithm for automatic regis-tration of 3D TRUS to the da Vinci surgical system using the da Vinciinstrument tips as the fiducials, this algorithm could also be used for othersimilar applications.In chapter 5, we described a system for real-time remote palpation basedon robotic ultrasound and strain imaging. The system is intuitive and easyto use and has the potential to compensate for the lack of haptic feedback inthe current state of the surgical robotic systems. The system has been testedin user study involving 6 users on a prostate phantom and promising resultshave been obtained. We further tested the system on 5 prostate cancerpatients undergoing da Vinci radical prostatectomy. While the system couldhave some clinical benefit for tumor and calcification detection, it requiressome prior learning and training by the surgeons. Such a training wouldfamiliarize the surgeon with the system and the images so that they can useit during the operation.In chapter 6, we described a system for acquisitions of multi-parametricquantitative ultrasound elastography data from patients in the operatingroom. We further described a framework to evaluate the system’s cancerdetection performance using classification algorithms and comparison withpost-operative whole-mount histopathology. The system is tested on 10 pa-tients and promising cancer detection results have been obtained. In future,more data should be collected and fed into the classification algorithm inorder to increase the accuracy and reliability of the results. Similar to anyother classification algorithm, the more data one provides to the algorithm,the better the classifier would be trained and more reliable would be theresults.In chapter 7, we presented a new control framework for force feedbackteleoperation. We implemented the algorithm on the da Vinci surgical sys-tem using the da Vinci research kit controllers. Development of this algo-rithm involved establishing the dVRK system in the lab and implementingrobust teleoperation, torque control, trajectory tracking and gravity com-pensation controllers. We used the developed system in a user study involv-133Chapter 8. Conclusionsing 9 users testing the system. While the results showed potential benefitsof the proposed approach in standard suturing and palpation tasks, theuser interface of the system could be improved in order for it be used moreintuitively.134Bibliography[1] June, 2015.[2] What are the key statistics about prostate cancer? National CancerInstitute,, June, 2015.[3] T. K. Adebar, O. Mohareri, and S. E. Salcudean. Instrument-basedcalibration and remote control of intraoperative ultrasound for robot-assisted surgery. In Proc. IEEE International Conference on Biomed-ical Robotics and Biomechatronics, pages 24–27, 2012.[4] T. K. Adebar, S. E. Salcudean, S. S. Mahdavi, M. Moradi, C. Nguan,and L. Goldenberg. A robotic system for intra-operative trans-rectalultrasound and ultrasound elastography in radical prostatectomy. In-formation Processing in Computer Assisted Intervensions, 6689:79–89,2011.[5] T. K. Adebar, M. C. Yip, S. E. Salcudean, R. N. Rohling, C. Y. Nguan,and S. L. Goldenberg. Registration of 3d ultrasound through an airtis-sue boundary. Medical Imaging, IEEE Transactions on, 31(11):2133–2142, 2012.[6] S. Ahmad and R. Cao et al. Transrectal quantitative shear wave elas-tography in the detection and characterisation of prostate cancer. Sur-gical Endoscopy, 27:3280–3287, 2013.[7] B. Ahn, Y. Kim, C. K. Oh, and J. Kim. Robotic palpation andmechanical property characterization for abnormal tissue localization.Medical Biological Engineering Computing, page 111, 2012.135Bibliography[8] C. T. Bethea and A. M. Okamura et al. Application of haptic feedbackto robotic surgery. Journal of Laparoendoscopic and Advanced surgicaltechniques Part: A, 14:191–195, 2004.[9] F. J. Bianco, D. J. Grignon, W. A. Sakr, B. Shekarriz, J. Upadhyay,E. Dornelles, and J. E. Pontes. Radical prostatectomy with bladderneck preservation: impact of a positive margin. European Urology,43(5):461–466, 2003.[10] S. Billings, N. Deshmukh, H. Jae Kang, R. Taylor, and E. M. Boctor.System for robot-assisted real-time laparoscopic ultrasound elastogra-phy. In Proc. SPIE, volume 8316, 2012.[11] J. Braun, K. Braun, and I. Sack. Electromagnetic actuator for gen-erating variably oriented shear waves in MR elastography. MagneticResonance in Medicine, 50(1):220–222, 2003.[12] L. Breiman. Random forests. Machine Learning, 45:5–32, 2001.[13] M. Brock, C. Von Bodman, and et al. The impact of real-time elas-tography guiding a systematic prostate biopsy to improve cancer de-tection rate: A prospective study of 353 patients. Journal of Urology,187:2039–2043, 2012.[14] M. Brock, T. Eggert, and et al. Multiparametric ultrasound of theprostate: adding contrast enhanced ultrasound to real-time elastogra-phy to detect histopathologically confirmed cancer. Journal of Urology,189:93–98, 2013.[15] R. Rohling C. Shneider, A. Baghani and S. Salcudean. Remote ultra-sound palpation for robotic interventions using absolute elastography.In Proc. (MICCAI 2012), pages 42–49, 2012.[16] B. E. Chapman and D. L. Parker. An analysis of vessel enhance-ment filters based on the hessian matrix for intracranial mra. In Proc.ISMRM, page 607, 2001.136Bibliography[17] Z. Chen, A. Deguet, R. Taylor, S. DiMaio, G. Fischer, andP. Kazanzides. An open-source hardware and software platform fortelesurgical robotics research. In Proc. (MICCAI’13) Workshop onSystems and Arch. for Computer Assisted Interventions, pages 1–10,2013.[18] A. Cheng, X. Guo, H. Kang, B. Tavakoli, J. U. Kang, R. H. Taylor,and E. M. Boctor. Concurrent photoacoustic markers for direct three-dimensional ultrasound to video registration. page 89435, 2014.[19] C. L. Cheung, C. Wedlake, J. Moore, S. E. Pautler, and T. M. Pe-ters. Fused video and ultrasound images for minimally invasive partialnephrectomy: a phantom study. pages 408–415, 2010.[20] S. Demirci, A. Bigdelou, L. Wang, C. Wachinger, M. Baust, R. Tibre-wal, R. Ghotbi, H. Echstein, and N. Navab. 3d stent recovery fromone x-ray projection. In Proc. MICCAI, pages 178–185, 2011.[21] O. A. Van der Meijden and M. P. Schijven. The value of haptic feed-back in conventional and robot-assisted minimal invasive surgery andvirtual reality training: a current review. Surgical Endoscopy, 23:1180–1190, 2009.[22] S. DiMaio and C. Hasser. The da vinci reseach interface. In Proc.MICCAI Workshop: Syst. Archit. for Comp. Assis.Interven., pages626–634, 2008.[23] K. Dong-Soo, Y. Yong-San, L. Jung-Ju, K. Seong-Young, H. Kwan-Hoe, C. Jong-Ha, P. Young-Bae, and W. Chung-Hee. Arthrobot : anew surgical robot system for total hip arthroplasty. pages 1123–1128,2001.[24] K. J. Draper, C. C. Blake, L. Gowman, D. B. Downey, and A. Fenster.An algorithm for automatic needle localization in ultrasound-guidedbreast biopsies. Medical Physics, 27(8):1971–1980, 2000.137Bibliography[25] H. Eskandari, O. Goksel, and et al. Bandpass sampling of high-frequency tissue motion. IEEE Transactions on UFFC, 58:1332–1343,2011.[26] V. Ficarraa, P. Sooriakumaranc, G. Novarab, O. Schatloffd, A. Brig-antie, H. Van der Poelf, F. Montorsie, V. Pateld, A. Tewari, andA. Mottriea. Systematic review of methods for reporting combinedoutcomes after radical prostatectomy and proposal of a novel system:The survival, continence, and potency (scp) classification. EuropeanUrology, 61(3):541–548, 2012.[27] J. Finkelstein, E. Eckersberger, H. Sadri, S. S. Taneja, H. Lepor, andB. Djavan. Open versus laparoscopic versus robot-assisted laparo-scopic prostatectomy: The european and us experience. Reviews inUrology, 12(1):35–43, 2010.[28] A. F. Frangi. 3D model-based analysis of vascular and cardiac images,2001.[29] T. Freitag, R. M. Jerram, A. M. Walker, and C. G. Warman. Surgicalmanagement of common canine prostatic conditions. Compend ContinEduc Vet, 29(11):656–663, 2007.[30] F. Gaufillet, H. Liegbott, M. Uhercik, and et al. 3d ultrasound real-time monitoring of surgical tools. In Proc. IEEE Ultrasonics Sympo-sium (IUS), pages 2360–2363, 2010.[31] J. Gewirtz, D. Standish, P. Martin, J. A. Kunkel, M. Lilavois, A. Wed-mid, D. I. Lee, and K. J. Kuchenbecker. Tool contact accelerationfeedback for telerobotic surgery. 4:210–220, 2011.[32] T. Gianduzzo, J. R. Colombo, G. P. Haber, J. Hafron, C. Magi-Galluzzi, M. Aron, I. S. Gill, and J. H. Kaouk. Laser roboticallyassisted nerve-sparing radical prostatectomy: a pilot study of techni-cal feasibility in the canine model. BJU International, 102(5):598–602,2008.138Bibliography[33] T. R. J. Gianduzzo, J. R. Colombo, E. El-Gabry, G. P. Haber, andI. S. Gill. Anatomical and electrophysical assessment of the canineperiprostatic neurovascular anatomy: Perspective as a nerve sparingradical prostatectomy model. The Journal of Urology, 179(3):2025–2029, 2008.[34] D. Greenwald, C. G. L. Cao, and E. W. Bushnell. Haptic detection ofartificial tumors by hand and with a tool in a mis environment. IEEETransactions on Haptics, 5(4):131–138, 2012.[35] B. Guillonneau and G. Vallanccien. Laparoscopic radical prostate-ctomy: the montsouris technique. Journal of Urology, 163(6):1643–1649, 2000.[36] I. Hacihaliloglu, R. Abugharbieh, A. J. Hodgson, and R. N. Rohling.Bone surface localization in ultrasound using image phase-based fea-tures. Ultrasound in Medicine Biology, 35(9):1475–1487, 2009.[37] B. A. Hadaschik, T. H. Kuru, C. Tulea, P. Rieker, I. V. Popeneciu,T. Simpfendrfer, J. Huber, P. Zogal, D. Teber, S. Pahernik,M. Roethke, P. Zamecnik, W. Roth, G. Sakas, H. P. Schlemmer, andM. Hohenfellner. A novel stereotactic prostate biopsy system integrat-ing pre-interventional magnetic resonance imaging and live ultrasoundfusion. Journal of Urology, 186(6):2214–2220, 2011.[38] Misop Han, Dan Stoianovici, Chunwoo Kim, Pierre Mozer, Fe-lix Schfer, Shadie Badaan, Bogdan Vigaru, Kenneth Tseng, DoruPetrisor, and Bruce Trock. Tandem-robot assisted laparoscopic radi-cal prostatectomy to improve the neurovascular bundle visualization:a feasibility study. Journal of Urology, 77(2):502–506, 2011.[39] K. Hashtrudi-Zaad and S. E. Salcudean. Analysis of control archi-tectures for teleoperation systems with impedance/admittance masterand slave manipulators. 20:419–445, 2001.[40] R. D. Howe, W. J. Peine, D. A. Kontarinis, and J. S. Son. Remote139Bibliographypalpation technology for surgical applications. IEEE Engineering inMedicine and Biology Magazine, 14:318323, 1995.[41] AJ Hung, ALD Abreu, S. Shoji, AC Goh, AK Berger, MM Desai,M. Aron, IS Gill, and O. Ukimura. Robotic transrectal ultrasonog-raphy during robot-assisted radical prostatectomy. European urology,62(2):341–348, 2012.[42] Z. Jiang, D. Piao, G. R. Holyoak, J. W. Ritchey, K. E. Bartels, G. Slo-bodov, C. F. Bunting, and J. S. Krasinski. Trans-rectal ultrasound-coupled spectral optical tomography of total haemoglobin concentra-tion enhances assessment of the laterally and progression of a trans-missible venereal tumor in canine prostate. Urology, 77(1):237–242,2011.[43] L. M. Johnson, B. Turkbey, W. D. Figg, and P. L. Choyke. Multipara-metric MRI in prostate cancer management. Nature Reviews ClinicalOncology, 11:346353, 2014.[44] S. C. Johnson. Hierarchical clustering schemes. Psychometrika,32(3):241–254, 1967.[45] S. Karimaghaloo, S. Prasad, P. Abolmaesumi, G. Fichtinger, and R. H.Rohling. Towards detecting surgical clips in 3d ultrasound for targetlocalization in radiation therapy: a study on tissue phantoms. In Proc.IEEE EMBS, pages 5282–5285, 2008.[46] Peter Kazanzides, Zihan Chen, Anton Deguet, Gregory S. Fischer,Russell H. Taylor, and Simon DiMaio. An open-source research kit forthe da Vinci® surgical robot. In Proc. IEEE International Conf. onRobotics and Automation (ICRA’14), pages 6434–6439, 2014.[47] C. H. King, M. O. Culjat, M. L. Franco, C. E. Lewis, E. P. Dutson,W. S. Grundfest, and J. W. Bisley. Tactile feedback induces reducedgrasping force in robot-assisted surgery. 2:103–110, 2009.140Bibliography[48] M. Kitagawa, D. Dokko, A. M. Okamura, and D. D. Yuh. Effect ofsensory substitution on suture-manipulation forces for robotic surgi-cal systems. The journal of Thoracic and Cardiovascular Surgery,129:151–158, 2005.[49] M. Kitagawa, A. M. Okamura, B. T. Bethea, V. L. Gott, and W. A.Baumgartner. Analysis of suture manipulation forces for teleoperationwith force feedback. In Proc. (MICCAI’02), pages 155–162, 2002.[50] P. Kozlowski, S. D. Chang, R. Meng, B. Mdler, R. Bell, E. C. Jones,and S. L. Goldenberg. Combined prostate diffusion tensor imaging anddynamic contrast enhanced MRI at 3t–quantitative correlation withbiopsy. Magnetic Resonance Imaging, 28(5):621–628, 2010.[51] A. Kron and G. Schmidt. Stability and performance analysis of kines-thetic control architectures for bimanual telepresence systems. 46:1–26, 2006.[52] A. Kronn and G. Schmidt. A bimanual haptic telepresence system -design issues and experimental results. In Proc. of Int. Workshop onHigh-Fidelity Telepresence and Teleaction, 2003.[53] D. M. Kwartowitz, S. D. Herrell, and R. L. Galloway. Update: towardimage-guided robotic surgery: determining intrinsic accuracy of theda vinci robot. International Journal of Computer Assisted Radiologyand Surgery, 1(5):301–304, 2007.[54] D. J. Lamb and L. Zhang. Challenges in prostate cancer research:animal models for nutritional studies of chemoprevention and diseaseprogression. The Journal of Nutrition, 135(12):3009–3015, 2008.[55] J. Leven, D. Burschka, R. Kumar, G. Zhang, S. Blumenkranz, X. Dai,M. Awad, G. D. Hager, M. Marohn, M. Choti, Ch. Hasser, and R.l H.Taylor. Davinci canvas: A telerobotic surgical system with integrated,robot-assisted, laparoscopic ultrasound capability. In Proc. MedicalImage Computing and Computer-Assisted Intervention MICCAI Lec-ture Notes in Computer Science, volume 3749, pages 811–818, 2005.141Bibliography[56] Q. H. Li, L. Zamorano, A. Pandya, R. Perez, J. Gong, and F. Diaz.The application accuracy of the neuromate robot–a quantitative com-parison with frameless and frame-based surgical localization systems.Computer Aided Surgery, 7(2):90–98, 2002.[57] H. Liu, D. P. Noonan, B. J. Challacombe, P. Dasgupta, L. D. Senevi-ratne, and K. Althoefer. Rolling mechanical imaging for tissue abnor-mality localization during minimally invasive surgery. IEEE Transac-tions on Biomedical Engineering, 57(2):404414, 2010.[58] J. Long, G. Haber, B. H. Lee, J. Guillotreau, R. Autorino, H. Laydner,R. Yakoubi, E. Rizkala, R. J. Stein, and J. H. Kaouk. Real-time robotictransrectal ultrasound navigation during robotic radical prostatec-tomy: initial clinical experience. Journal of Urology, 80(3):608–613,2012.[59] S. S. Mahdavi, N. Chng, I. Spadinger, W. J. Morris, and S. E. Salcud-ean. Semi-automatic segmentation for prostate interventions. MedicalImage Analysis, 15(2):226–237, 2011.[60] M. Mahvash, J. C. Gwilliams, R. Agarwal, B. Vogvolgyi, M. L. Su,D. D. Yuh, and A. M. Okamura. Force-feedback surgical teleopera-tor: Controller design and palpation experiments. In Proc. Symp. OnHaptic Interfaces for Virtual Environments and Teleoperator Systems,pages 465–471, 2008.[61] L. Marks, S. Young, and S. Natarajan. MRI-ultrasound fusion forguidance of targeted prostate biopsy. Current Opinion in Urology,23(1):43, 2013.[62] J. E. McNeal, A. E. Redwine, and et al. Zonal distribution of prostaticadenocarcinoma. correlation with histologic pattern and direction ofspread. Am. J. Surg. Pathol., 12:897–906, 1988.[63] T. Miyagawa, S. Ishikawa, T. Kimura, T. Suetomi, M. Tsutsumi,T. Irie, M. Kondoh, and T. Mitake. Real-time virtual sonography for142Bibliographynavigation during targeted prostate biopsy using magnetic resonanceimaging data. International Journal of Urology, 17(10):855–860, 2010.[64] O. Mohareri, J. Ischia, P. C. Black, C. Schneider, J. Lobo, L. Golden-berg, and S. E. Salcudean. Intraoperative registered transrectal ultra-sound guidance for robot-assisted laparoscopic radical prostatectomy.The Journal of urology, 193(1):302–312, 2015.[65] O. Mohareri, M. Ramezani, T. Adebar, P. Abolmaesumi, and S. Sal-cudean. Automatic localization of the da vinci surgical instrumenttips in 3-d transrectal ultrasound. IEEE Transactions of BiomedicalEngineering, 60(8):2663–2672, 2013.[66] O. Mohareri and S. E. Salcudean. da vinci auxiliary arm as a roboticsurgical assistant for semi-autonomous ultrasound guidance duringrobot-assisted laparoscopic surgery. In The Hamlyn Symposium onMedical Robotics, page 45, 2014.[67] Omid Mohareri, Septimiu Salcudean, and Chris Nguan. Asymmet-ric force feedback control framework for teleoperated robot-assistedsurgery. In Proc. IEEE International Conf. on Robotics and Automa-tion (ICRA’13), pages 5800–5806, Karlsruhe, Germany, May 2013.[68] R. Muthupillai, D. J. Lomas, and et al. Magnetic resonance elas-tography by direct visualization of propagating acoustic strain waves.Science, 269:1854–1857, 1995.[69] K. R. Nightingle, M. L. Palmeri, R. W. Nightingle, and G. E. Trahey.On the feasibility of remote palpation using acoustic radiation source.Journal of Acoustic Soc. Am., 110(1):625–634, 2001.[70] G. Nir. Prostate registration using magnetic resonance elastographyfor cancer localization. UBC PhD dissertation, 2015.[71] G. Nir and S. E. Salcudean. Registration of whole-mount histologyand tomography of the prostate using particle fltering. SPIE MedicalImaging, page 86760, 2013.143Bibliography[72] P. M. Novotny, J. A. Stoll, P. E. Dupont, and R. D. Howe. Real-timevisual servoing of a robot using three-dimensional ultrasound. In Proc.IEEE Int. Conf. on Robotics and Automation, pages 2655–2660, 2007.[73] P. M. Novotny, J. A. Stoll, N. V. Vasilyev, P. J. Del Nido, P. E.Dupont, T. E. Zickler, and R. D. Howe. Gpu based real-time in-strument tracking with three-dimensional ultrasound. Medical ImageAnalysis, 11(5):458–464, 2007.[74] E. Nuno, R. Ortega, N. Barabanov, and L. Basanez. A globaly stablepd controller for bilateral teleoperators. 24:753–758, 2008.[75] A. M. Okamura. Methods for haptic feedback in teleoperated robot-assisted surgery. Industrial robots, 31:499–508, 2004.[76] A. M. Okamura, C. Basdogan, S. Baillie, and W. S. Harwin. Haptics inmedicine and clinical skill acquisition. IEEE Transactions on Haptics,4(3):153–154, 2011.[77] A. M. Okamura, L. N. Verner, T. Yamamoto, and J. C. Gwilliam.Force feedback and sensory substitution for robot-assisted surgery.Surgical Robotics: Systems, Applications and Vision, pages 419–448,2011.[78] K. Okihara, K. Kamoi, M. Kanazawa, T. Yamada, O. Ukimura,A. Kawauchi, and T. Miki. Transrectal ultrasound navigation during-minilaparotomy retropubic radical prostatectomy: Impact on positivemargin rates and prediction of earlier return to urinary continence.International Journal of Urology, 16(10):820–825, 2009.[79] T. Ortmaier, B. Deml, B. Kuebler, G. Passig, D. Reintsema, andU. Seibold. Robot assisted force feedback surgery. In Advances inTelerobotics Springer Tracks on Advanced Robotics STAR 31, pages361–379, 2007.[80] C. Passenberg, A. Peer, and M. Buss. Model-mediated teleoperationfor multi-operator multi-robot systems. In Proc. IEEE International144BibliographyConf. on Intelligent Robots and Systems (IROS), pages 4263–4268,2010.[81] C. Philips. Tracking the rise of robotic surgery for prostate cancer.NCI Cancer Bulletin, 8(16), 2011.[82] P. A. Pinto, P. H. Chung, A. R. Rastinehad, A. A. Baccala,J. Kruecker, C. J. Benjamin, S. Xu, P. Yan, S. Kadoury, C. Chua,J. K. Locklin, B. Turkbey, J. H. Shih, S. P. Gates, C. Buckner,G. Bratslavsky, W. M. Linehan, N. D. Glossop, P. L. Choyke, andB. J. Wood. Magnetic resonance imaging/ultrasound fusion guidedprostate biopsy improves cancer detection following transrectal ultra-sound biopsy and correlates with multiparametric magnetic resonanceimaging. The Journal of Urology, 186(4):1281–1285, 2011.[83] H. G. Van Der Poel, W. De Blok, A. Bex, W. Meinhardt, and S. Horen-blas. Peroperative transrectal ultrasonographyguided bladder neckdissection eases the learning of robot-assisted laparoscopic prostatec-tomy. BJU international, 102(7):849–852, 2008.[84] T. C. Poon and R. N. Rohling. Tracking a 3-d ultrasound probewith constantly visible fiducials. Ultrasound in Medicine & Biology,33(1):152–157, 2007.[85] D. T. Price, R. S. Chari, J. D. Neighbors, S. Eubanks, W. W.Schuessler, and G. M. Preminger. Laparoscopic radical prostatectomyin canine model. Journal of Laparoscopic Surgery, 6(6):405–412, 1996.[86] C. E. Reily, T. Akinbiyi, D. Burschka, D. C. Chang, A. M. Okamura,and D. D. Yuh. Effects of visual force feedback on robot-assisted sur-gical task performance. The journal of Thoracic and CardiovascularSurgery, 135:196–202, 2008.[87] C. Reynier, J. Troccaz, P. Fourneret, A. Dusserre, C. Gay-Jeune,J. L. Descotes, M. Bolla, and J. Y. Giraud. MRI/trus data fusionfor prostate brachytherapy: preliminary results. Medical Physics,31(6):1568–1575, 2004.145Bibliography[88] R. S. Sahebjavaher, A. Baghani, and et al. Transperineal prostate MRelastography: Initial in vivo results. Magnetic Resonance in Medicine,69:411–420, 2013.[89] Ramin S. Sahebjavaher, Ali Baghani, Mohammad Honarvar, RalphSinkus, and Septimiu E. Salcudean. Transperineal prostate MR elas-tography: initial in vivo results. Magnetic resonance in medicine :official journal of the Society of Magnetic Resonance in Medicine /Society of Magnetic Resonance in Medicine, 69(2):411–420, 2013.[90] S. E. Salcudean, R. S. Sahebjavaher, and et al. Biomechanical model-ing of the prostate for procedure guidance and simulation. Soft TissueBiomechanical Modeling for Computer Assisted Surgery, 11:169–198,2012.[91] G. Salomon, J. Kollerman, and et al. Evaluation of prostate cancerdetection with ultrasound real-time elastography: A comparison withstep section pathological analysis after radical prostatectomy. Euro-pean Urology, 54:13541362, 2008.[92] C. Schneider, J. Guerrero, C. Nguan, R. Rohling, and S. E. Salcudean.Intra-operative pick-up ultrasound for robot assisted surgery with ves-sel extraction and registration: A feasibility study. In Proc. Informa-tion Processing in Computer-Assisted Interventions (IPCAI), volume6689, pages 122–132, 2011.[93] S. Sirouspour. Modeling and control of cooperative teleoperation sys-tems. IEEE Transactions on Robotics, 21:1220–1225, 2005.[94] S. Sirouspour and P. Setoodeh. Multi-operator/multi-robot teleop-eration: an adaptive nonlinear control approach. In Proc. IEEE In-ternational Conf. on Intelligent Robots and Systems (IROS), pages1576–1581, 2005.[95] J. A. Smith, R. C. Chan, S. S. Chang, S. D. Herrell, P. E. Clark,R. Baumgartner, and M. S. Cookston. A comparison of the incidence146Bibliographyand location of positive surgical margins in robotic assisted laparo-scopic radical prostatectomy and open retropubic radical prostatec-tomy. Journal of Urology, 178(6):2389–2390, 2007.[96] S.E. Song, N. B. Cho, G. Fischer, N. Hata, C. Tempany, G. Fichtinger,and I. Iordachita. Development of a pneumatic robot for MRI-guidedtransperineal prostate biopsy and brachytherapy: New approaches.pages 2580–2585, 2010.[97] D. Stoianovici, L. L. Whitcomb, J. H. Anderson, R. H. Taylor, andL. R. Kavoussi. A modular surgical robotic system for image guidedpercutaneous procedures. Med Image and Comput Assist Interven-tions, 1496:404–410, 1998.[98] P. J. Stolka, M. Keil, G. Sakas, E. R. McVeigh, R. H. Taylor, andE. M. Boctor. A 3d-elastography-guided system for laparoscopic par-tial nephrectomies. 2010.[99] N. Tan, D. J. A. Margolis, T. D. McClure, A. Thomas, D. S. Finley,R. E. Reiter, J. Huang, and S. S. Raman. Radical prostatectomy: valueof prostate MRI in surgical planning. Abdominal Imaging, 37:664674,2012.[100] M. Tavakoli and R. D. Howe. Haptic effects of surgical teleoperatorflexibility. International Journal of Robotics Research, 28(10):1289–1302, 2009.[101] R. H. Taylor, B. D. Mittelstadt, H. A. Paul, W. Hanson, P. Kazanzides,J. F. Zuhars, B. Williamson, B. L. Musits, E. Glassman, and W. L.Bargar. An image-directed robotic system for precise orthopaedicsurgery. IEEE Transactions on Robotics and Automation, 10(3):261–275, 2011.[102] C. Tempany, S. Straus, N. Hata, and S. Haker. MR-guided prostateinterventions. Magnetic Resonance Imaging, 27(2):356–367, 2008.147Bibliography[103] A. Tobergte, R. Konietschke, and G. Hirzinger. Planning and controlof a teleoperation system for research in minimally invasive roboticsurgery. In IEEE International Conference on Robotics and Automa-tion, pages 4225 – 4232, 2009.[104] A. Trejos, J. Jayender, M. Perri, M. Naish, R. Patel, and R. Malthaner.Robot-assisted tactile sensing for minimally invasive tumor localiza-tion. The International Journal of Robotics Research, 28(9):11181133,2009.[105] E. Turgay, S.E. Salcudean, and et al. Identifying mechanical propertiesof tissue by ultrasound. Ultrasound in Medicine and Biology, 32:221–235, 2008.[106] O. Ukimura, M. M. Desai, S. Palmer, S. Valencerina, M. Gross, A. L.Abreu, M. Aron, and I. S. Gill. 3-dimensional elastic registration sys-tem of prostate biopsy location by real-time 3-dimensional transrectalultrasound guidance with magnetic resonance/transrectal ultrasoundimage fusion. The Journal of Urology, 187(3):1080–1086, 2012.[107] O. Ukimura and I. S. Gill. Real-time transrectal ultrasound guidanceduring nerve sparing laparoscopic radical prostatectomy: pictorial es-say. The Journal of urology, 175(4):1311–1319, 2006.[108] O. Ukimura, J. H. Kaouk, A. Kawauchi, T. Miki, I. S. Gill, M. M. De-sai, A. P. Steinberg, M. Kilciler, C. S. Ng, S. C. Abreu, M. Spaliviero,and A. P. Ramani. Real-time transrectal ultrasonography during la-paroscopic radical prostatectomy. Journal of Urology, 172(1):112–118,2004.[109] O. Ukimura, JH Kaouk, A. Kawauchi, T. Miki, IS Gill, MM Desai,AP Steinberg, M. Kilciler, CS Ng, SC Abreu, M. Spaliviero, andAP Ramani. Real-time transrectal ultrasonography during laparo-scopic radical prostatectomy. Journal Of Urology, 172(1):112–118,2004.148Bibliography[110] O. Ukimura, C. Magi-Galluzzi, and I. S. Gill. Real-time transrectalultrasound guidance during laparoscopic radical prostatectomy: im-pact on surgical margins. The Journal of urology, 175(4):1304–1310,2006.[111] S. Umeyama. Least-squares estimation of transformation parametersbetween two point patterns. IEEE Transactions on Pattern Analysisand Machine Intelligence, 13(4):376–380, 1991.[112] C. R. Wagner, N. Stylopoulos, and R. D. Howe. The role of forcefeedback in surgery: Analysis of blunt dissection. In Proc. Symp. OnHaptic Interfaces for Virtual Environments and Teleoperator Systems,pages 68–74, 2002.[113] J. C. Williams, M. Mahvash, R. Agrawal, B. Vogvolgyi, D. D. Yuh,and A. M. Okamura. Effects of haptic and graphical force feedback forteleoperated palpation. In Proc. IEEE International Conf. on Roboticsand Automation (ICRA’09), pages 677–682, 2009.[114] M. Yip, T. Adebar, R. Rohling, and et. al. 3d ultrasound to stereo-scopic camera registration through an air-tissue boundary. In Proc.Medical Image Computing and Computer Assisted Intervension (MIC-CAI), pages 626–634, 2010.[115] R. Zahiri-Azar and S. E. Salcudean. Motion estimation in ultrasoundimages using time domain cross correlation with prior estimates. IEEETrans. Biomed. Eng., 53:1990–2000, 2000.[116] L. Zhai, J. Madden, and et al. Acoustic radiation force impulse imag-ing of human prostates ex vivo. Ultrasound Med. Biol., 36:576588,2010.[117] M. Zhang, P. Nigwekar, and et al. Quantitative characterization ofviscoelastic properties of human prostate correlated with histology.Ultrasound Med. Biol., 34:1033–1042, 2008.149[118] Y. Zheng, G. Li, M. Chen, Q. Chan, S. Hu, X. Zhao, R. Ehman,E. Lam, and E. Yang. Magnetic resonance elastography with twinpneumatic drivers for wave compensation. In Proc. IEEE EMBS, pages2611–2613, 2007.150Appendix ASemi-AutonomousUltrasound Guidance DuringRobot-Assisted LaparoscopicSurgeryRobotic surgical systems enable surgeons to perform delicate and preciseminimally invasive surgery and also provide a platform for integrating sur-gical image-guidance and automation of particular tasks. Currently suchsystems are primarily controlled by the surgeon in a human-in-the-loopmaster-slave architecture. Introducing autonomy of surgical sub-tasks hasthe potential to assist surgeons by decreasing the workload and also im-proved surgical navigation.This chapter describes a semi-autonomous robotic ultrasound guidancesystem for robot-assisted laparoscopic surgery, implemented on the da Vinci®surgical system. The da Vinci auxiliary patient side manipulator (PSM)is introduced here as a robotic ultrasound surgical assistant (US-PSM). Arobotic pick-up ultrasound transducer is rigidly grasped by the surgeon us-ing the auxiliary PSM, and 3D ultrasound is registered to the tip of theother operating PSMs instrument performing tasks such as tumor resection.Solving a real-time inverse kinematics and ultrasound registration problemenables the auxiliary manipulator to autonomously track the surgical tool ina restricted task-specific workspace and provide real-time ultrasound guid-ance to the surgeon. Furthermore, the proposed strain imaging technique inChapter 5 could be used with this system to also provide subsurface images151A.1. Introductionto the surgeon at the console. The techniques were implemented on theda Vinci® surgical system (classic version) using the da Vinci research kit(dVRK) controllers that enable complete access to all control levels of theda Vinci manipulators via custom mechatronics and open-source software.A.1 IntroductionMedical robots have been used clinically for more than two decades in manyfields including orthopedics, minimally invasive surgery, and image-guidedinterventions. As surgeons and engineers strive to make surgical proceduresmore minimally invasive, challenges arise with regards to workspace limi-tations, tool ergonomics and cooperation between surgeons and assistants.The introduction of robotic laparoscopic surgery has provided surgeons witha minimally invasive option with improved tool dexterity and high qualitystereo-vision. This has increased the number and difficulty of the proceduresthat can be performed. The robotic platform also provides a platform forintegration of surgical guidance with medical imaging and the automationof particular surgical tasks.Surgical guidance has been used successfully during neurosurgery and or-thopedics. The use of image guidance has expanded into abdominal surgery,which is a shifting dynamic and complicated environment. Surgeons cannotrely on pre-operative imaging since significant organ shifts can take placewithin the abdominal cavities. Ultrasound has become the intra-operativemodality of choice for many applications, as it is real-time, non-ionizing andrelatively inexpensive.Though ultrasound has many advantages, it is an operator dependentmodality. The quality of the ultrasound is very dependent on the experi-ence of the ultrasound operator. Ultrasound requires a certain amount ofhand-eye coordination as well in order to understand the spatial relation-ship between the anatomy of the image and the location of the ultrasoundprobe. Intra-operative ultrasound that is used during the robotic surgery ismanipulated by the patient side assistant, instead of the operating surgeon,breaking the normal hand-eye coordination.152A.1. IntroductionDespite the challenges, intra-abdominal ultrasound is currently used reg-ularly during robotic partial nephrectomy. This procedure involves the re-moval of cancerous sections of the kidney, while preserving as many healthnephrons as possible. This more difficult procedure has been shown to havebetter patient outcomes. Ultrasound is used during this procedure to iden-tify the resection margins of the tumour, helping to ensure the completeremoval of the cancerous tissue.Certain other procedures, such as liver resection and some cardio-thoracic,have the potential to benefit from the integration of intra-operative ultra-sound. Ultrasound is routinely used during open liver surgery to providereal-time guidance but has not penetrated as deeply into the laparoscopicmarket.The use of a robotically controlled ultrasound probe would put the con-trol of the ultrasound probe back into the hands of the surgeon, making themindependent of their patient side assistants. In addition, the surgeon wouldnow have the precision of motion that the robotic tools provide allowing formore exact placement of the probe on the organ of interest.In the case of robotic partial nephrectomy, the perioperative outcomesare similar, regardless of whether a laparoscopic or robotic ultrasound probeis used. A robotic probe, however, might provide advantages in terms ofsurgeon autonomy. Perioperative parameters including time spent in theoperating room, blood loss and positive surgical margin rates were compa-rable between patients who underwent robotic partial nephrectomy with alaparoscopic probe (n = 72) or a robotic probe (n = 73).The use of the robotic platform also allows for the potential of automatedmotion in certain situations. Robotic motions can be separated into threemain types, (i) direct control, (ii) shared control and (iii) supervisory control.Direct control describes the typical teleoperated robot, where all motion isdictated by the surgeons motions. In shared control, the robot and thesurgeon use the same resources to complete the task, and in supervisorycontrol, the surgeon would direct the robot to complete a task, but the robotis pre-programmed to make the motions without direct surgeon controlRobots with shared control include those that implement virtual fixtures,153A.1. Introductionusing force feedback to guide the surgeon, who is in control of the tool.Autonomous motion has been explored in surgical robotics. RoboDoc,for instance, used a pre-programmed plan to drill out femur for hip replace-ment [101]. Another drilling robot also uses preoperative imaging to drillout the inner ear for cohlear implants. Other robotics are more reactive totheir environments, using visual servoing based on a real-time ultrasoundimage to control the placement of the robot. Research in surgical robotics ismoving towards the development of a more autonomous system such as thePAKY-RCM for biopsy [97], Arthrobot in orthopedics [23]and Neuromatein neurosurgery [56]. To make these types of autonomous motions moredynamic, The Language of Surgery is being developed to better understandthe motions and actions that make up the complicated task of surgery.We present a system where the third arm of the da Vinci is used toautonomously track the operating arms. We focus on the application of thissystem for partial nephrectomy. This is particularly applicable to partialnephrectomy, where the surgeon must cut out a section of the kidney withoutleaving cancerous tissue behind. Having real-time ultrasound guidance atthe time of resection could provide the surgeon with vital information toimprove surgical margins and avoid internal kidney anatomy. Unfortunatelythis has never been previous possible because the tumour resection requiresthe full attention of both operating arms and an external laparoscopic probedoes not have the dexterity to show the surgeon useful information. Acustom designed ultrasound probe will be held in the third arm of the daVinci. Through a calibration method described below, the ultrasound probewill be autonomously adjusted in order to show the surgeon the tip of histool in relation to critical anatomy.The da Vinci Research Kit (dVRK) controllers were used to implementthis technique on the first generation of the da Vinci surgical system (daVinci classic).154A.2. Materials and MethodsA.2 Materials and MethodsA.2.1 System OverviewFigure A.1 depicts the main components of our research platform. ThedVRK system that has been fully explained in Chapter 7 is used for thisexperiment. The ultrasound transducer (Figure A.1c) used in this work is acustom-made intra-abdominal transducer that is small enough to fit througha standard laparoscopic incision, and can be picked up and maneuvered bythe ProGrasp instrument of the da Vinci [92]. Its unique design allows forit to be easily picked up in a repeatable manner such that the position ofthe ultrasound image, with respect to the da Vinci instrument can be pre-calibrated. As a high frequency probe (The transducer has 128 elements,is 28 mm long and is operated at 7 to 10 MHz ), it allows for high qualityimaging of the kidney and can be placed directly on the kidney surface.The transducer is used in combination with a Sonix TABLET ultrasoundmachine (UltrasonixMedical Corp., Richmond, VA, Canada).Bimanual Teleoperation and Auxiliary PSM ControlA component for independent control of the auxiliary PSM is added tothe bi-manual teleoperation framework established in the work presented inChapter 7. The foot-pedal buttons are employed in this component to inter-change the MTM-PSM teleoperation chain, and to control the auxiliary PSMwith the right of left MTM. Normally, the bi-manual teleoperation wouldwork with (MTMR-PSM1, MTML-PSM2) structure. Using the auxiliarycontrol module, the user can control PSM3 with either MTMR or MTML.We used this control component for autonomous control of the auxiliary armon top of the existing bi-manual teleoperation using the method explainedin the next section.A.2.2 3D Ultrasound and Robot Instrument RegistrationIn our proposed method for control of the auxiliary PSM with ultrasound(US-PSM), the US imaging plane is automatically repositioned using the155A.2. Materials and Methods(a) (b) (c) Figure A.1: The overview of the da Vinci Research Kit controllers andthe da Vinci classic surgical system in UBC RCL Laboratory. Three daVinci PSMs, two for bimanual tele-manipulation and the auxiliary arm forultrasound.PSM wrist such that a 2D ultrasound image continuously contains the tipof a specified da Vinci surgical manipulator as shown in Figure A.2. Sur-geons can then elect to have a real-time ultrasound image that actively trackstheir instruments as they work, allowing them to continuously monitor theirposition relative to sensitive anatomical structures, without pausing to repo-sition the ultrasound probe. Automatically tracking the tip of a da Vinciinstrument with the ultrasound imaging plane requires that the position ofthe instrument tip relative to the US-PSM be known.To perform the registration, we define five coordinate systems: da VinciPSM base coordinate system {O1, C1}, PSM instrument wrist base coordi-nate system {Ow1, Cw1}, the US-PSM base coordinate system {O2, C2}, the156A.2. Materials and MethodsUS-PSM PSM1 da Vinci Tool  and  Ultrasound Probe Intra-operative  Ultrasound PSM Intraoperative pick-up Ultrasound {𝑂1, 𝐶 1} {𝑂2, 𝐶 2} {𝑂𝑤1, 𝐶  𝑤1} {𝑂𝑤2, 𝐶  𝑤2} {𝑂𝑢𝑠, 𝐶  𝑢𝑠} 𝑃 (a)PSM1 Pick-up US transducer US-PSM Instrument tip (P) PSM2 (b)Figure A.2: (a) Registration concept to enable the US-PSM to automaticallyreposition the ultrasound imaging plane to follow PSM tip point P. (b) Theexperimental setup showing the same concept.US-PSM instrument (ProGrasp) wrist base coordinate system {Ow2, Cw2}and the coordinate system of the ultrasound image {Ous, Cus}. {O1, C1}and {O2, C2} are coordinate systems which is located at the base of eachPSM, at the end of the setup joints kinematics chain which is held fixed inthe dVRK experimental setup. Based on the coordinate frames introducedin Figure A.2a, we have:USP = CUSTCw2×Cw2TC2×C2TC1×C1TCw1×Cw1P (A.1)Using our established dVRK software platform, Cartesian position of thetip and wrist angles of the da Vinci operating instruments are known in theircorresponding coordinate frames {O1, C1} and {O2, C2} which are locatedat the base of each PSM as shown in Figure A.2a. Hence, Cw1P , C1TCw1and Cw2TC2 transformations are known in real-time. The custom-designedtransducer creates a static and repeatable transform between the da Vinci157A.3. Experiments and Resultstool (Prograsp) and the ultrasound image. The tool-to-image transform wasfound using the single-wall calibration method implemented in Stradwin.Hence CUSTCw2 is also known. As a result, all that is required for tracking isa registration between the two PSM coordinate systems to calculate C1TC2 .The air-tissue boundary registration method presented in Chapter 2 isused here to compute this transformation intra-operatively. The tip of theda Vinci working instrument is pressed against a tissue surface in a locationthat could be imaged by the US transducer. The da Vinci US-PSM is used tomanipulate the pick-up transducer until the 2D sagittal US image containsthe tip of the instrument. Next, the tool tip is located inside the US image{Ous, Cus} and in {O3, C3} using the fixed usT3. Repeating this procedurefor more than three points inside the work space of the manipulators willenable solving the homogeneous transformation.Once the transformation is found, solving the real-time inverse kinemat-ics on the US-PSM enables the ultrasound imaging plane to automaticallytrack the tip of the other instrument within its working space of the wristwhich is 180 degrees (Yaw and Pitch) and 540 degree rotation (roll). Sur-geons can also switch in and out of tracking mode, by activating the daVinci surgeon consoles clutch pedal, recognized by our software to changethe control mode from automatic to manual surgeon-controlled. Due tothe redundancy in maintaining the PSM instrument tip within the imagingplane using the complete 6-DOF motion of the da Vinci PSM, in our imple-mentation of this method, the US-PSM wrist base Cw2is placed in a fixedlocation by the user, and only the wrist is allowed to move and automaticallytrack the other instruments tip.A.3 Experiments and ResultsA.3.1 da Vinci Surgical Instrument Motion Accuracy inthe DVRK SetupThe relative motion accuracy of the ProGrasp instrument with respect toeach arm’s base coordinate frame is measured using the NDI OptoTrak158A.3. Experiments and ResultsCertus motion capture system. A custom-made 3D printed tool with 8optical markers (shown in Figure A.3) was built so that it can be rigidlypicked up and attached to the ProGrasp tool of the da Vinci. First, the toolwas moved and rotated around in the OptoTrak field of view to construct arigid body on the tool using the NDI 6D Architect software. The orientationof the rigid body coordinate frame was manually adjusted to align withIntuitive convention for wrist orientation: Z-axis of the tip frame is alignedwith the instrument pointing direction and the Y-axis is aligned with thejaw open/close axis.The instrument tip was moved to 20 different points (x,y,z, Roll, Pitch,Yaw) in the robot work-space and the 6D coordinate of each point wascaptured in both OptoTrak frame and DVRK frame. The mean Cartesianmotion error was calculated to be 1.8398± 0.4775 mm and the mean wristmotion error is 2.6176 ± 1.2224 deg . These values could vary from tool totool and arm to arm because of the errors caused by instrument aging andcompliance in the cables. Moving the calibration tool with the da Vinciinstrument will not mask kinematics errors, because accuracy of buildingthe rigid body depends on the OptoTrak system, and is independent of theaccuracy that you move it around.You might want to test some situations in which there are force loadsat the remote center or at the instrument tip. This is where the kinematicswill diverge from actual, due to compliance in the drive-train. This may be(a) (b)Figure A.3: The custom made calibration tool with 8 optical markers at-tached to it. The markers are attached to the tool in a way that minimum3 markers are always visible in the camera field of view.159A.4. Discussions, Future Work and ConclusionsFigure A.4: Semi-autonomous guidance on ex-vivo kidney.relevant for your application.A.4 Discussions, Future Work and ConclusionsThe presented system is particularly applicable to the partial Nephrectomyprocedure, where the surgeon must resect a section of the kidney withoutleaving cancerous tissue behind. Having real-time ultrasound guidance atthe time of resection could provide the surgeon with vital information to im-prove surgical margins and spare internal kidney anatomy. This has neverbeen previously possible because the tumor resection requires the full at-tention of both operating arms and an external laparoscopic probe does nothave the dexterity to show the surgeon useful information.Figure A.5: Ultrasound images of the da Vinci tool pressed on the kidneysurface for registration.160A.4. Discussions, Future Work and ConclusionsFigure A.6: The kidney experiment with da Vinci 3 arms, one holding thekidney, one performing the tissue cutting and poking and one holding theUS probe autonomously tracking the tool.161Appendix BClinical Study ProtocolsB.1 Animal Study at Intuitive Surgical Inc (22October 2012)B.1.1 Setup Preparationˆ 1.1 Initialize the TRUS robot control software (VibroApp), RF imag-ing software and Ultrasonix software for acquiring Ultrasound B-modeand Doppler images.ˆ 1.2 Setup the connections between system components:– a. Connect the TRUS probe to the PC based Sonix TABLET USmachine (Ultrasonix medical corp., Richmond, Canada).– b. Connect the Ethernet cable between Sonix TABLET US ma-chine and back of the da Vinci surgeon console for API datastreaming.– c. Connect motors (Roll, Translate and Vibration) to the controlbox using their serial connections.– d. Connect the control box to the Sonix TABLET US machineusing the 3 in 1 USB connection.ˆ 1.3 After the animal has been placed on the table, sedated and securedin a Trendelenburg position to simulate RALRP procedure, install theTRUS system at the foot of the OR table using the CIVCO stabilizerarm and the bed attachment clamps as it is shown in Fig. 1.ˆ 1.4 Prepare the TRUS probe for insertion into the animals rectum:162B.1. Animal Study at Intuitive Surgical Inc (22 October 2012)apply ultrasound gel and probe cover from sterile packaging to theprobe.ˆ 1.5 Insert the probe into the animals rectum using the gross positioningclamp on the CIVCO arm. The array on the US probe should bepositioned axially so that the Parasagittal array images as much ofthe prostate as possible.B.1.2 Pre-Operative Phase:ˆ 2.1 Transrectal Vibro-Elastography data collection: Capture two tothree sweeps of B-mode and RF data with position information byrotating the Parasagittal imaging plane from -40 to 40 degrees andvibrating the TRUS probe with 2-10 Hz frequencies.ˆ 2.2 Tranperineal Vibro-Elastography data collection:– a. Install the shaker mechanism for Tranperineal vibrations onthe CIVCO stabilizer arm.– b. Capture two to three sweeps of B-mode and RF data withposition information by rotating the Parasagittal imaging planefrom -40 to 40 degrees and vibrating animals perineum using theshaker with frequency in the range of 50-200 Hz.ˆ 2.3 Capture Doppler images of the prostate lateral to evaluate local-ization of the NVB based on Doppler signals in the vasculature.B.1.3 Intra-Operative Phase:ˆ 3.1 The surgeon should start to open up the animal as in the usualRALRP procedure. Insert the da Vinci trocars under vision, dock theda Vinci robot and begin initial dissection.ˆ 3.2 Once the anterior aspect of the prostate has been identified, wecould start registering the TRUS robot to the da Vinci kinematic frameand da Vinci stereo-camera.163B.1. Animal Study at Intuitive Surgical Inc (22 October 2012)ˆ 3.3 Before starting the registration process:– a. Capture two to three sweeps of B-mode data with positioninformation by rotating the Parasagittal imaging plane from -40to 40 degrees to evaluate how the air-tissue boundary (at anteriorpart of the prostate) looks like in a real intra-op scenario. TheUS volumes captured at this stage will also be used to furtherevaluate the automatic tool tip localization algorithm that hasbeen developed for this application [8].– b. Capture B-mode and Doppler images of NVB (visible at theposterior-lateral prostate border) with position information.– c. Perform two sweeps of transrectal Vibro-Elastography andcapture RF data for further processing.– d. Perform two sweeps of transperineal Vibro-Elastography andcapture RF data for further processing.ˆ 3.4 The registration should be done using the air-tissue boundarymethod reported in [9, 10]. Two different approaches will be testedand registration accuracy will be calculated.– Registration between TRUS and da Vinci tool:* a. Adjust the US machine parameters for optimized imaging.* b. Initialize the ISI API kinematic streaming module. Choosethe patient side manipulator number that is being used forregistration. Start the kinematic streaming.* c. Place the da Vinci tool tip against the air-tissue boundary.* d. Use the 3D mouse to rotate the Parasagittal TRUS planeuntil the tip of the tool could be visualized in the Parasagittalimage. Typically, after the user locates the tip of the instru-ment, the image is rotated back and forth in small incrementsuntil maximum tool intensity is seen.* e. If the location of the tool in the ultrasound image is notclear, it is helpful to gently vibrate the da Vinci manipulatormanually, to make its location in the image clearer.164B.1. Animal Study at Intuitive Surgical Inc (22 October 2012)* f. Initialize the transformation finder software and select thetool tip center in the US image selected in 3.4.d.* g. Repeat 3.4.b to 3.4.e four more times, and export thetransformation between the TRUS robot and da Vinci APIframes to the VibroApp software (motor control software) tostart tool tracking.– Registration validation (calculating the tool tracking accuracyafter registration):In this step, the tracking accuracy will be calculated as thedifference between the users defined tool tip location and toollocation resulting from automatic tool tracking after regis-tration. Here is the procedure:h. Click on begin tracking inVibroApp window. The TRUS probe will automatically ro-tate to track the registered da Vinci tool tip. i. Place the daVinci tool tip on the surface of prostate in a random location.j. Record the probe angle. k. Turn off automatic tracking.Take a 5 degrees sweep around the guessed tool location tosee the surrounding US images. l. Turn on manual trackingand try to find the exact location of the tool tip using the3D mouse. m. Record the probe angle. n. Also, record thepoint coordinates (x, y, z) using the API. o. Repeat 3.4.hto 3.4.n for 10 different points on the prostate surface andcalculate mean, max and standard deviation of tracking er-ror. p. Repeat the registration and validation process for 4different users and calculate mean and standard deviation oferror for all users.*ˆ 3.4.2 Registration between TRUS and da Vinci stereo camera usingthe registration tool:– q. Place the registration tool against the air-tissue boundary.– Use the 3D mouse to rotate the Parasagittal TRUS plane untilthe spherical fiducial on the registration tool could be visualized165B.1. Animal Study at Intuitive Surgical Inc (22 October 2012)in the Parasagittal image. A 5 degrees volume will be collectedaround each spherical fiducial.– s. Adjust the imaging focus depth as necessary on the Ultrasonixconsole.– Initialize the transformation finder software and select the tooltip in the collected 5 degrees US volume.– u. Find the corresponding optical marker in the da Vinci cameraframe using the optical tracking software.– v. Repeat 3.4.r to 3.4.u for all of the fiducials on the registrationtool, and record the transformation between the TRUS robotand da Vinci camera frames. Registration validation (calculatingimage overlay accuracy after registration):– w. Place the da Vinci tool tip on the tissue– x. Capture a US volume and record the location of the tool tipin the US volume.– y. Record the snap shot of the tool tip location in the cameraview. Also, record the video of the tool tip placed on the tissue.The tool tip is a common feature in both camera and US frames.– z. Overlay the US volume on the camera view off-line using thecalculated transformation matrix from registration in 3.4.3.– aa. Find the difference in pixels, between the location of thetool tip in the camera view and in the transformed, overlaid USvolume.– bb. Repeat 3.4.w to 3.4.aa for 10 points.– cc. Report the mean and standard deviation of the error.ˆ 3.5 Pick-up ultrasound evaluation:– a. The drop in probe designed to be used during partial nephrec-tomy should also be tested in this environment. Tools should beswitched to use the Prograsp on PSM2. Video recording will beon.166B.1. Animal Study at Intuitive Surgical Inc (22 October 2012)– b. The camera and port should be removed, the transducer placedthrough the incision and then the port and camera replaced. Ad-ditional suture may be required to contain any leaks of abdomenpressure.– c. The surgeon can grasp and drop the transducer multiple times(20). Difficulties and failures will be recorded.– d. The surgeon will use the transducer to locate important struc-tures related to the procedure, such as the prostate base andapex, the NVB and the urethra. Each structure should be identi-fied in both the axial and transverse planes. Images will be savedwhen the surgeon believes he has found the best orientation pos-sible. US volume acquisitions with da Vinci and TRUS will becompared after this step.– e. New freehand techniques of elastography have been developedusing the da Vinci and the pick-up ultrasound transducer.– f. Probe will be left in place throughout the rest of the procedureto be used at the surgeons discretion.– g. Transducer will be removed after the camera at the end of thesurgery.ˆ 3.6 After the registration, the surgeon should continue the steps of theprocedure, while making use of real-time 2D B-mode imaging guid-ance, with both manual repositioning of the imaging planes using the3D mouse and automatic tool tracking. The 3D mouse will be placedon or near the da Vinci surgeon console so the operating surgeon cancontrol the position of the arrays. Real-time B-mode images shouldbe examined before dissection of the prostate base and apex, and sep-aration of the NVB.ˆ 3.7 Once the prostate has been fully mobilized and placed in a spec-imen bag, the TRUS transducer should be fully retracted using theremote manual control, in order to avoid negatively affecting the anas-tomosis.167B.1. Animal Study at Intuitive Surgical Inc (22 October 2012)ˆ 3.8 After the completion of the procedure, remove the probe, stepperand stabilizer arm.B.1.4 System Evaluation Criterion:Robot Spacing Issue:The TRUS robot should be placed at the foot of the OR table, in the spacebetween da Vinci and the table as it is shown in Fig. 1. This configurationhas been tested in the OR in Vancouver General Hospital and the TRUSsystem could be fitted marginally in this spacing. We want to double checkthis issue during an actual RALRP procedure and make sure that the sur-geon could reach the sweet spot for placing the surgical tools and could dothe procedure comfortably without any spacing issue.Procedure Timing Issue:An estimate of the amount time that will be added to the standard RALRPprocedure should be calculated. The extra time added to the procedure isan important factor for evaluation of our image guidance system. Measuringthe amount of time needed to perform all of the above steps in an animalstudy will give us a good estimate for the actual clinical application. In fact,we should time all calibration procedures so we can get an idea of learningtime involved.Imaging Evaluations:The following factors should be checked in the captured images:ˆ How well can we see the air-tissue boundary (at anterior part of theprostate) in the B-mode images once the anterior aspect of the prostatehas been identified?ˆ How well can we see the da Vinci tool tip (or the registration tool)pressed against the air-tissue boundary in the B-mode images? Canwe localize the tool tip in the B-mode images and use its location forregistration?168B.2. Transrectal Probe Disinfection at BC Cancer Agencyˆ How well can we see the NVB in the Doppler images and where is itslocation (at what angle in the US volume)?ˆ Vibro-Elastography images: Cancer detection, prostate boundary de-tection.ˆ Comparison between pick-up ultrasound images and TRUS images.Feasibility evaluation of using pick-up ultrasound for RALRP proce-dure as well.Registration Accuracy:Registration validation will be done using the procedures explained in 3.4and accuracy will be reported to the surgeons.Drop-In Probe Evaluation:This new ultrasound probe will be evaluated by the surgeon with respect toimage quality and manoeuvrability. Any issues with respect to blocking thecamera will be noted. An evaluation sheet will be provided to the surgeonafter the procedure. The freehand elastography results will be compared tothose collected with the transrectal ultrasound transducer.B.2 Transrectal Probe Disinfection at BC CancerAgencyThe probe disinfection station has two columns. The right one (blue) isfilled with a Cidex disinfectant. The left column is water. Make sure youhave gloves during the whole process.ˆ Put the probe in the Cidex column for 15 minutes (hook the cables tothe black plastic on top of the column).ˆ Carefully, take out and move the probe to the water column for 30seconds.169B.2. Transrectal Probe Disinfection at BC Cancer Agencyˆ Rinse for at least 30 seconds and inspect the probe for Cidex residue.ˆ Wipe the probe with a clean towel or tissue.ˆ Cover the probe (e.g., using a pillow cover).ˆ Fill in and sign the disinfection log (in probe case).170Appendix CElastography and the daVinciThe desirability of having haptic or palpation feedback during robot-assistedsurgery has been mentioned by many practitioners who use the da Vincisystem. The tissue mechanical properties felt through palpation, such aselasticity and viscosity, can be acquired by using ultrasound elastographytechniques. Elastography images can then convey the tissue properties visu-ally, and can be helpful in many ways, including by helping determine organand tumour boundaries and the effect of treatment such as thermal ablation.With further processing to simulate tool-tissue interaction, elastic modelscan be used to synthesize haptic feedback at the da Vinci masters withoutcompromising system stability. Previous elastography systems developedfor the da Vinci involved only strain imaging, which contains limited andsubjective information, and is neither repeatable nor quantitative.The quantitative elastography data-set collected for this part of the workwas collected inside the operating room, after the patient is anesthetized onthe OR table and before docking the da Vinci robot. Once the da Vincisurgical system and our TRUS system is docked, there is a very limitedspace to externally reach the patient’s prostate and send mechanical excita-tion. Hence, we propose using an internal mechanical excitation system thatcould be integrated with the da Vinci system and used during the procedureand produce absoluate elasticity maps of the prostate intraoperatively. Thedesign of our internal excitation system will be explained in the next section.171C.1. Tissue Exciter Design and Building for Intraoperative ElastographyC.1 Tissue Exciter Design and Building forIntraoperative ElastographyThe quantitative elastography method requires that the steady state tissuemotion in a volume of tissue be measured. This is because waves can travelin arbitrary directions and measuring the wavelength in a single plane oralong a line can produce large errors. This measurement over a volumeshould be carried out without modifying the wave pattern. Therefore, eventhough it may be simpler to use, it is not advisable to simply attach anexciter to the ultrasound transducer to induce waves in tissue, because theexcitation source would move with the transducer and therefore significantlychange the wave pattern in tissue with transducer motion.Many sources of mechanical excitation have been tested in our groupfor elastography imaging. These include voice coil actuators coupled to thetissue through a rigid mechanical link [11], vibration motors [4], pneu-matic [118] and hydraulic transmission [89] or manual excitation achievedby pushing the ultrasound transducer against the tissue [69]. For an inter-nal excitation system, two actuation options are available, considering thesize and biocompatibility issues: miniature vibration motors and movingcoil actuators. We propose using a miniature brushless DC motor with aneccentric load on its shaft for the actuation mechanism of the exciter.Our exciter design (Figure C.1 and Figure C.2)is composed of a smallcylinder with no sharp edges for easy insertion through the trocar or directlythrough the incision between the trocar and tissue, as done with our pick-up ultrasound transducer. The exciter has a diameter of 11 mm. Accurateexcitation frequencies are achieved by adding a feedback loop or a phaselocked loop. A MEMS-size accelerometer is placed inside the exciter systemfor excitation frequency measurement and to provide feedback to the con-troller. Different components of our designed excitation system are listed inTable C.1.Desirable excitation waves in the frequency range of 50-200 Hz and am-plitude of approximately 1 mm could be achieved using such a system. Thecylinder and the adapter part will be manufactured from a lightweight, stiff,172C.1. Tissue Exciter Design and Building for Intraoperative ElastographyFigure C.1: Shaker enclosure and motor housing design.Figure C.2: (a) Motors and the accelerometer, (b) Motor with eccentricmass on its shaft enclosed in the 3D printed enclosing with its cap open, (c)cap fully closed for testing.173C.2. Clinical Integration PlanTable C.1: Internal shaker components.Component DescriptionMotorBrushless Maxon motor, 6 mm diameter, nominalspeed 11200.0 rpm,Controller digital 1-Q-EC Amplifier 24 V / 1 A, speed controlAccelerometer ADXL 362 3-axis accelerometersterilizable material, as discussed below. The actuator and the connectingcable will also be sterilizable, as is the case with intra-operative ultrasoundtransducer cables. An EM sensor will be integrated with the exciter sothat its location can be tracked with external EM tracking systems on theultrasound machines.An adapter part is designed at one end of the cylindrical exciter for easypick-up and manuverability with the da Vinci ProGrasp forceps. Unlikeour pick-up ultrasound probe, the shaker does not need to be calibrated tothe da Vinci tool tip. Only an easy and fast grasping technique should bedesigned for it. Different grasping methods and adaptor shapes have beendesigned and tested to reach an optimum shape within such size constraints.Three potential designs for the adapter part are illustrated in Figure C.3.Methods for fixing the shaker to the tissue during excitation will be exploredand include suturing it close to the region of interest (e.g. lateral or superiorto the prostate; onto the kidney) or designing a vacuum mechanism thatcould be remotely controlled. We have been able to test several prototypedesigns using rapid prototyping on our Objet 3D printers before having thecasing manufactured as shown in Figure C.4.C.2 Clinical Integration PlanOnce the design is completed, it will be tested for current leakage and othersafety measures by the Biomedical Engineering Department at VancouverGeneral Hospital (VGH). In addition, in order to use the exciter inside apatient in the OR, the sterilization department at VGH requires that eachdevice have validated manufacturer instructions regarding reprocessing, and174C.2. Clinical Integration PlanFigure C.3: (a) Three sides design with 120 degree, (b) Two-sided designwith 180 degree, (c) One sided design.Figure C.4: The internal tissue excited graped by the Prograsp instrumentof the daVinci system.175C.2. Clinical Integration Planbe in accordance with CSA standards (CSA Z314.9). Since we are not amanufacturer as recognized by the hospital, we will need to get the de-vice and sterilization process validated by a third party. We have alreadycompleted this process for the intra-operative pick-up ultrasound probe andare familiar with the procedures. The completed exciter will be sent to Ad-vanced Sterilization Products (ASP), the manufacturer of the STERRAD®.STERRAD® is a sterilization machine that uses hydrogen peroxide plasmato sterilize operating room equipment. This method is often used on moredelicate equipment such as intra-operative ultrasound probes and will beappropriate for our exciter. ASP will create a set of validated instructionssuch that the exciter can be sent through the sterilization department atVGH as any other device used in the OR. We will also test the device toassure that the sterilization process has not affected its efficacy.The use of the STERRAD® system does affect our choice in materials,since all materials used in this device need to be compatible with the clean-ing method as well as be biocompatible. A plastic that is both bio- andSTERRAD® compatible is ULTEM. This plastic is a material originallyproduced by the General Electric Plastic division and is a thermoplasticthat is heat and solvent resistant. It is often used in medical devices, in-cluding the casing of intra-operative probes. Other biocompatible plasticsinclude medical grades of PVC and Polyethylene, PEEK, Polycarbonate,Polysulfone, Polypropylene and Polyurethane. In addition, companies suchas Tristar, specialize in one-on-one material selection and part manufactur-ing. If it is determined that the design of the exciter is beyond what we canmanufacture in the lab, outside consultation is possible.An alternative to using plastics would be to use a stainless steel casing.Surgical grade stainless steel is often used for many re-useable medical de-vices due to its durability and sterilizablity. However, steel may interferewith the electromagnetic tracking system, so plastics are preferred.176


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items