JOURNAL OF LAPAROENDOSCOPIC & ADVANCED SURGICAL TECHNIQUESVolume 14, Number 3, 2004© Mary Ann Liebert, Inc.               ...
192                                                                                                      BETHEA ET AL.   “...
HAPTIC FEEDBACK TO ROBOTIC SURGERY                                                                                    193 ...
194                                                                                                     BETHEA ET AL.FIG. ...
HAPTIC FEEDBACK TO ROBOTIC SURGERY                                                                                        ...
Upcoming SlideShare
Loading in …5

Technical Report Application of Haptic Feedback to Robotic ...


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Technical Report Application of Haptic Feedback to Robotic ...

  1. 1. JOURNAL OF LAPAROENDOSCOPIC & ADVANCED SURGICAL TECHNIQUESVolume 14, Number 3, 2004© Mary Ann Liebert, Inc. Technical Report Application of Haptic Feedback to Robotic Surgery BRIAN T. BETHEA, MD,1 ALLISON M. OKAMURA, PhD,2 MASAYA KITAGAWA, MS,2 TORIN P. FITTON, MD,1 STEPHEN M. CATTANEO, MD,1 VINCENT L. GOTT, MD,1 WILLIAM A. BAUMGARTNER, MD,1 and DAVID D. YUH, MD,1 ABSTRACTRobotic surgical systems have greatly contributed to the advancement of minimally invasive endo-scopic surgery. However, current robotic systems do not provide tactile or haptic feedback to theoperating surgeon. Under certain circumstances, particularly with the manipulation of delicate tis-sues and suture materials, this may prove to be a significant irritation. We hypothesize that hapticfeedback, in the form of sensory substitution, facilitates the performance of surgical knot tying. Thispreliminary study describes evidence that visual sensory substitution permits the surgeon to applymore consistent, precise, and greater tensions to fine suture materials without breakage during ro-bot-assisted knot tying. INTRODUCTION laparoscopic general surgery due, in large part, to the lack of feedback. This deficiency with current roboticT ECHNOLOGICAL ADVANCEMENTS in computer-assisted surgical systems and increasing health care demandshave expanded the field of minimally invasive surgery to systems is a significant handicap in performing the tech- nically more intricate and delicate surgical tasks inher- ent in cardiac surgery. For example, suturing a coronarynew specialties. Robotic surgical systems have several arterial anastomosis with fine polypropylene suture is aadvantages over conventional endoscopic surgery, in- highly dexterous task, in which the surgeon typicallycluding three-dimensional vision, increased range of mo- uses his or her sense of touch to puncture tissue with ation, tremor filtration, and motion scaling, permitting fine needle, pull the suture through, and tie and tightensurgical tasks to be performed in confined spaces.1,2 Fur- knots. In our own observations of experienced and tal-thermore, patients and the health care industry are ag- ented cardiac surgeons training with the da Vinci sur-gressively pursuing procedures with smaller incisions, gical system, fine polypropylene sutures are often bro-less pain, and shorter recovery times.3 ken, and delicate tissues torn, due to the application of Computer-assisted robotic surgical systems have excessive forces conventionally attenuated with hapticmade total endoscopic cardiac surgery a reality. How- feedback. The consequences of such surgical errors orever, current robotic surgical systems are limited by the delays in cardiac surgery (e.g., coronary microvascularlack of tactile or haptic feedback critical to performing or great vessel trauma, prolonged cardiopulmonarycomplex, delicate surgical tasks.4 Clinical successes bypass) present much greater potentials for irreversiblewith robot-assisted minimally invasive cardiac surgery injury, excessive hemorrhage, or even death for thehave lagged behind those achieved with robot-assisted patient. 1Division of Cardiac Surgery, The Johns Hopkins Hospital and 2Department of Mechanical Engineering, Johns Hopkins Uni-versity, Baltimore, Maryland. 191
  2. 2. 192 BETHEA ET AL. “Sensory substitution” is a means by which applied in the operator’s view, to reposition the masters in theirforces are conveyed to the operating surgeon with visual work space, and to focus the endoscopic camera. Orien-or auditory representations. In this preliminary study, we tational alignment is always provided, and positionalassess the feasibility and potential benefits of sensory alignment can be adjusted to permit repositioning of thesubstitution in providing haptic feedback in the context master handles independent of the instrument tips. Mo-of robotic-assisted knot tying. tion scaling and tremor filtering by the digital interface further augment precision. The electronic controller is ca- pable of fully interconnected controls of 48 degrees of METHODS freedom at update rates over 1000 cycles per second. A second subsystem consists of a side cart, made upSubjects of fully sterilizable endoscopic instruments, the instru- ment and camera manipulators, the endoscope, and the The human subjects of this study included surgeons assistant’s user interface. The end-effectors are fully ster-within the Division of Cardiac Surgery at the Johns Hop- ilizable instruments that attach interchangeably to the twokins Hospital; the study intended to provide basic guide- manipulators (automated instrument recognition) andlines on the level and resolution (temporal and magni- provide a total of 6 degrees of freedom (plus tool func-tude) of force-sensing required for knot-tying. tion) intracorporeally. Three manipulators drive the two endoscopic instruments and camera. In turn, these ma-Da Vinci surgical robotic system nipulators are positioned around the body by three pas- sive multiple-link arms mounted to a fixed base. As an The da Vinci computer-enhanced instrumentation sys- emergency fail-safe, the arms can be removed from thetem (Intuitive Surgical Inc., Mountain View, CA) con- patient in seconds.sists of three major components, an input device (sur-geon’s console), a digital interface, and an output device Characterization of applied suture forces(manipulator). The console houses the display system, theinput handles, the digital interface, and the electronic con- In conjunction with the Johns Hopkins University De-troller. The tool handles are serial link manipulators that partment of Mechanical Engineering, a tension measur-serve both as high-resolution input devices reading the ing device (TMD) was constructed, allowing force-sens-position, orientation, and grip commands from the sur- ing on both the left and right hands (Fig. 1). The TMDgeon, and as haptic displays. The image of the surgical measured the tension (in newtons) applied to suture dur-site is transmitted to the surgeon through a high-resolu- ing knot-tying exercises. The “optimal” applied forces fortion stereo display. The system projects the image of the each of several suture materials commonly used insurgical site atop the surgeon’s hands (via mirrored over- cardiac operations (2-0 Ti-Cron and 5-0, 6-0, and 7-0lay optics), while the controller transforms the spatial mo- polypropylene), as determined by the participating sur-tion of the tools into the camera frame of reference. In geons, were measured with the TMD during the perfor-this arrangement, the system restores hand-eye coordi- mance of manually tied knots (Table 1). With these mea-nation and provides a natural scaled-down correspon- surements, we defined an optimal applied tension rangedence in motions. for each suture grade that would avoid breakage yet The user interface permits the surgeon to control cam- assure adequate functional outcome (e.g., knot-tying)era positioning while keeping the slave instrument tips across anticipated tissue compliances. A BFIG. 1. This tension measuring device was developed to measure the forces A: applied to sutures by hand and B: using the daVinci robot.
  3. 3. HAPTIC FEEDBACK TO ROBOTIC SURGERY 193 TABLE 1. OPTIMAL APPLIED FORCES DURING HAND TIES During robotic-assisted knot tying, mean tensions ap- FOR EACH SUTURE TYPE AS MEASURED BY THE plied to fine polypropylene sutures were greater with each TENSION MEASURING DEVICE DURING MANUALLY type of suture when haptic feedback was applied. As the PERFORMED KNOT TIES BY SUBJECT SURGEONS sutures became progressively smaller, increased tensions Manually applied ideal applied to the suture neared statistical significance in the tensions (newtons) 6-0 polypropylene group (P .053) and achieved statis- tical significance in the 7-0 polypropylene group (P 2-0 5-0 6-0 7-0Suture material Ti-Cron Prolene Prolene Prolene 0.001). As with the Ti-Cron group, the standard devia- tions were narrower when knots were tied with the aidMean tension 2.40 1.41 0.71 0.36 of sensory/haptic feedback in each polypropylene group.Standard deviation 0.58 0.14 0.06 0.04 DISCUSSIONSensory substitution trials Traditionally, cardiac surgery is performed through a Five surgeons first performed hand-ties using various median sternotomy, providing optimal access to all car-suture materials commonly used in cardiac operations: 2- diac structures and great vessels. The disfigurement and0 Ti-Cron, 5-0, 6-0, and 7-0 polypropylene (US Surgical pain associated with this extensive incision has been aCorporation, Norwalk, CT). Each surgeon performed longstanding but heretofore acceptable consequence offive ties with each suture and the mean tensions were cardiac surgery. Fueled largely by the advent of less in-recorded. A sensory substitution mechanism, in the form vasive and traumatic laparoscopic and thoracoscopicof a visual color bar scale, was placed in-line with the techniques for noncardiac operations and the rise ofTMD to relay the mean tension applied to the sutures to catheter-based interventions for myocardial revascular-the operating surgeon during the knot-tying task. The vi- ization, minimally invasive approaches to cardiac surgerysual color bar scale was calibrated using the mean suture are being developed.tensions obtained during manually tied knots. The color Computer-augmented, robot-assisted surgery has en-bar progresses from yellow to green to red as tension in- hanced the ability of surgeons to perform minimally in-creases (Fig. 2). vasive procedures by placing a digital interface between The five surgeons then performed robotic-assisted knot the surgeon and the instrument tips. The digital interfacetying as the control technique, followed by performing the enhances the surgeon’s ability to perform minimally in-same tying procedure with the aid of the color bar sensory vasive cardiac surgery in several ways. First, by filteringsubstitution. Each of the surgeons performed five surgical high-frequency signals, surgical tremor is eliminated.knots with each type of suture with and without sensory Second, the digital interface allows for motion scaling infeedback. The surgeons were instructed to either increase which easy-to-perform macroscopic movements at theor decrease the amount of tension applied to the suture to surgeon console are scaled down to a microscopic scalemaintain the color bar in the green zone. The amount of inside the patient, enhancing dexterity. Finally, the com-tension applied to the sutures was determined using the puter interface permits the accurate translation of the sur-tension-measuring device. geon’s hand motions to an endoscopic wrist placed withinStatistical analysis the chest cavity, conferring much higher degrees of free- dom than traditional manually actuated endoscopic in- The paired Student’s t-test was used to compare the struments.suture tensions between robot-assisted knot tying with The da Vinci system has been used successfully to per-and without haptic sensory substitution. form endoscopic coronary artery bypass grafting5,6 and mitral valve repairs.7 These early operations have been performed in carefully selected patients by a select few RESULTS surgeons who were intimately involved with the devel- opment of these systems in the laboratory and the oper- The mean tensions applied to the various suture mate- ating room. Despite these early procedural successes, im-rials with and without haptic/sensory feedback during ro- provements and adjunct technologies are needed beforebotic assisted knot tying were measured. In the Ti-Cron widespread use of the current systems can occur.8 For in-group, the mean tension applied to the suture was lower stance, the lack of tactile feedback has led to prolongedwith the aid of the color bar sensory substitution. Fur- operative times and difficulty in performing some force-ther, the amount of applied tension to the suture was sig- sensitive surgical tasks required in cardiac operations.4nificantly more consistent with haptic feedback as evi- Experienced robotic and endoscopic surgeons use visualdenced by the narrower standard deviation (Table 2). cues (e.g., tissue deformation) to approximate how much
  4. 4. 194 BETHEA ET AL.FIG. 2. Force feedback via sensory substitution. A visual color bar scale was developed to render applied suture tensions tothe operating surgeon (top). The applied tension as a function of time is displayed in the graph at the bottom.force is being applied: however, this type of sensory sub- Ideally, haptic feedback in robotic surgical systemsstitution is applicable only if the surgeon has a good in- would be relayed directly to the surgeon’s control actu-ternal model of tissue consistency to correlate applied ators, rendering a true tactile feedback. This capability,forces and tissue deflection. however, will require the application of force-sensors on TABLE 2. A COMPARISON OF MEAN TENSIONS ACHIEVED DURING THE PERFORMANCE OF ROBOT-ASSISTED SURGICAL KNOT TIES WITH AND WITHOUT SENSORY SUBSTITUTION (VISUAL) FORCE FEEDBACK, WITH STUDENT’S t-TEST COMPARISONS STRATIFIED ACROSS FOUR DIFFERENT SUTURE TYPES Tension measured (newtons; mean standard deviation) Robot assisted Robot assisted ties ties without with visual forceSuture material force feedback feedback P-valueTi-Cron 2-0 5.16 2.46 2.99 0.83 .0015-0 polypropylene 1.54 0.67 1.61 0.27 .5466-0 polypropylene 0.78 0.48 0.99 0.11 .0537-0 polypropylene 0.29 0.14 0.58 0.13 .001
  5. 5. HAPTIC FEEDBACK TO ROBOTIC SURGERY 195the endoscopic instrumentation. Our laboratories are ac- wound complications) warrant the continued safe devel-tively pursuing this goal. As an interim alternative, we opment of this technology. Clearly, the ability to conferhave employed sensory substitution to display haptic in- haptic feedback to present surgical robotic systems wouldformation via a visual display. Previous work in using contribute significantly to the safe performance of car-audio feedback of force levels has been a success in diac surgical procedures with these complex systems.microsurgery.9 However, additional noise, added tothe sounds already associated with cardiac operations ACKNOWLEDGMENT(e.g., cardiopulmonary bypass pump, hemodynamic/pulse oximetery monitors), may prove undesirable. A vi- This research was supported by grants from the Na-sual force-feedback display in the operative field display tional Institutes of Health (R01 EB002004) and themay prove to be more ergonomic and have already found Whitaker Foundation (RG-02-911).application. Image-guided virtual environments in theform of advanced video game systems and surgical sim- REFERENCESulators utilize surface image deformation, sometimes inconjunction with haptic instrumentation, to provide real- 1. Czibik G D’Ancona G, Donias HW, Karamanóukian HL.time simulations of operative procedures to include Robotic cardiac surgery: Present and future applications. Jspine surgery, cholecystectomy, and hysteroscopy, and Cardiothorac Vasc Anesth 2002;16:502–507.colonoscopy.10,11 Future developments in this area in- 2. Schneeberger E, Michler R. An overview of the Intuitivecludes visco-elastic modeling of the surface image de- system: The surgeon’s perspective. Op Tech Thorac Car-formations as well as higher fidelity force computations diovasc Surg 2001;6:170–176.and system validation. In these virtual simulators, the im- 3. Reger T, Jahnke M. Robotic cardiac surgery. AORN Jage itself serves as a sensory substitution modality. The 2003;1:182–186. 4. Shennib H, Bastawisy A, Mack MJ, Moll FH. Computer-impetus of these early experiments was to apply the vir- assisted telemanipulation: an enabling technology for en-tual environmental concept of visual force feedback to doscopic coronary artery bypass. Ann Thorac Surgthe real world surgical task of suture knot tying. 1998;66:1060–1063. Our results demonstrate significantly greater and more 5. Kappert U, Cichon R, Schneider J, et al. Closed-chest coro-consistent tensions applied to suture materials, without nary artery surgery on the beating heart with the use of abreakage, during robotic knot tying enhanced with haptic robotic system. J Thorac Cardiovasc Surg 2000;120:feedback compared to knots tied without feedback. During 809–811.the execution of these experiments, several interesting ob- 6. Mohr FW, Falk V, Diegeler A, et al. Computer-enhancedservations were noted. Surgeons broke the finer polypro- robotic cardiac surgery: Experience in 149 patients. J Tho-pylene sutures on several occasions both with and without rac Cardiovasc Surg 2001;121:842–853.sensory substitution. Interestingly, surgeons were able to 7. Chitwood WR, Nabong LW, Elbeery JE, et al. Robotic mi- tral valve repair: Trapezoidal resection and prosthetic an-partially compensate for the lack of haptic feedback by vi- nuloplasty with the da Vinci surgical system. J Thorac Car-sually observing the deformation of tissue when the knot diovasc Surg 2000;120:1171–1172.was pulled away from the tissue. We also noted that dif- 8. Awad H, Wolf R, Gravlee G. The future of robotic cardiacferent users employed different tying techniques; some sur- surgery. J Cardiothorac Vasc Anesth 2002;16:395–396.geons used equal force with both hands, while others fa- 9. Okamura A, Smaby N, Cutkosky M. An overview of dex-vored increased control with one hand. terous manipulation ICRA 2000;1:255–262. In this preliminary study, we have shown that haptic 10. Downes M, Cavusoglu M, Gantert W, et al. Virtual envi-feedback, in the form of a visual color bar scale, permits ronments for training critical skills in laparoscopic surgery.surgeons to be more consistent, precise, and apply sig- Proceedings of Medicine Meets Virtual Reality Conferencenificantly greater tensions to fine sutures without break- (MMVR)1998, IOS Press, pp. 316–322.age during robotic knot tying. Our visual sensory substi- 11. Rolfsson G, Nordgren A, Bindzau S, et al. Training and as- sessment of laparoscopic skills using a haptic simulator.tution aid represents an initial step towards providing Proceedings of Medicine Meets Virtual Reality Conferencesurgeons with tactile feedback during robotic surgery. (MMVR) 2002, pp. 409–411.This early work with computer-enhanced robotic-assistedcardiac surgery represents the field in its infancy. The as- Address reprint requests to:sociated costs of the robotic systems are indeed high, the David D. Yuhlearning curves extremely steep, and the applications lim- 600 North Wolfe Streetited in the context of cardiac surgery. Nevertheless the Blalock 618substantial potential benefits in terms of the already Baltimore, MD 21287proven clinical advantages of minimally invasive surgery(e.g., shorter hospital stays, less incision pain, fewer E-mail: