The main objective of this paper is to minimize the occluded areas in order to recognize the navigation of the surgeon’s tools for a two-arm autonomous robotic system for laparoscopic procedures. This robotic assistant needs the tracking of the surgeon’s surgical gestures in order to recognize the current maneuver and to execute the automated tasks of the robot. The surgical tools pose estimation is carried out by a Multiple Extended Kalman Filter (MEKF), where the movement models of the surgical tools depend on the maneuver which is being developed. This information is obtained by a maneuvers recognition system which is a part of the multimodal human machine interface (HMI) of the robot. The method proposed for reducing shadows has been applied to three in-vitro maneuvers which appear in the majority of the surgical protocols. The experiments show the behavior of this method for different time intervals of the occlusions.
(Rocky) Jaipur Call Girl - 09521753030 Escorts Service 50% Off with Cash ON D...
IROS 2011 - Surgical Tools Pose Estimation for a Multimodal HMI of a Surgical Robotic Assistant
1. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
System Engineering and Automation Department
http://www.isa.uma.es
University of Malaga (Spain)
INTELLIGENT ROBOTICS AND SYSTEMS (IROS 2011)
SURGICAL TOOLS POSE ESTIMATION FOR A
MULTIMODAL HMI OF A SURGICAL ROBOTIC
ASSISTANT
Belén Estebanez
Enrique Bauzano, Víctor Muñoz-Martínez
2. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
I. General Overview
ROBOTIC ASSISTANTS AND THEIR HUMAN MACHINE
INTERFACES
INDEX
I.Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
DA VINCI (Niemeyer , 2000).
ERM Robot (Muñoz, 2006)
FAce MOUSe (Nishikawa, 2003).
LAPMAN de Medsys (Polet and Donnez, 2004).
3. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
I. General Overview
ROBOTIC ASSISTANTS AND THEIR HUMAN MACHINE
INTERFACES
INDEX
I.Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
DA VINCI (Niemeyer , 2000).
ERM Robot (Muñoz, 2006)
FAce MOUSe (Nishikawa, 2003).
LAPMAN de Medsys (Polet and Donnez, 2004).
KaLAR system (Ko, 2007).
4. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
CISOBOT ROBOTIC ASSISTANT
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Microphone
Markers in surgical tools
E. Bauzano (IROS 2010)
I. General Overview
5. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
I. General Overview
PROBLEMS OF SURGICAL TOOLS TRACKING
INDEX
I.Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Disadvantages
• Existence of devices and surgical personnel in
the operation room
• Difficult location of the 3D sensors in the
operation room
• Specific tools movements during the surgical
task where the markers are hidden to the
sensors
6. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
HMI ARCHITECTURE
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
ROBOTIC ASSISTANT
Movements Control
SURGEON
PATIENT
SURGEON
MODEL
POSE
ESTIMATION
MODEL
TRACKING 3D
VOICE
RECOGNITION
Position and
Orientation
Interpreted Voice
Maneuver
Voice
Command
Maneuver
Patient-Tool
InteractionPatient-Endoscope
Interaction
Surgical Tools
Movements
Position and
Orientation Estimated
7. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
SURGEON MODEL
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Insert Needle Hold Tissue Grab Needle
Maneuver: Specific sequence of basic actions, modeled by HMM.
Assist Knot Tie knot
Basic Action: Interaction between the surgical tools managed by
the surgeon.
Angle Distance Left Tool
Velocity
Right Tool
Velocity
Insert Needle
B. Estebanez (RAAD 2010)
8. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
SURGEON MODEL
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Maneuvers
Library
Intervention
Model
Data
Processing
RECOGNITION
SYSTEM
Position and
Orientation
of the
Surgical Tool
Maneuvers
HMMs
Last maneuver
Developed
Voice
Command
Current Maneuver
with Gestures
Command
CURRENT
MANEUVER
B. Estebanez (RAAD 2010)
9. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
POSE ESTIMATION MODEL
II. Pose Estimation Model
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
Insert Needle Hold Tissue Grab Needle Extract Tool
e2 eNe1
Maneuver is divided into characteristic movements. A Characteristic Movement
(ei) is a well defined gesture inside the maneuver.
CHARACTERISTIC
MOVEMENTS
Selection (ei)
ESTIMATION
FUNCTIONS
Selection (fj)
Position and
Orientation
of the
Surgical Tool
Current
maneuver
Pose Estimation Model is a Multiple Extended Kalman Filter where their
Estimation Functions are selected according to the Current Maneuver and its
last Characteristic Movement.
Position
and
Orientation
Estimated
e3 ei
10. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
III. Implantation and Experiments
IMPLANTATION: CUTTING MANEUVER
2 4 6 8 10 12
80
100
120
Y(mm)
2 4 6 8 10 12
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12
0
20
40
60
Y(mm)
2 4 6 8 10 12
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12
80
100
120
Y(mm)
2 4 6 8 10 12
-1350
-1300
-1250
time (sec)
Z(mm)
2 4 6 8 10 12
0
20
40
60
Y(mm)
2 4 6 8 10 12
-1400
-1350
-1300
-1250
time (sec)
Z(mm)
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions
e1 e2 e3 e4 e3 e3 e5 e1 e2 e6 e1 e6 e1 e5
Surgeon’s Left Hand Surgeon’s Right Hand
14. SystemEngineeringand
AutomationDepartment
María Belén Estebanez Campos
belen@uma.es
CONCLUSIONS
To add a learning system so the MEKF may change its set of characteristic
movements depending on the surgeon
Improve the functions that model the characteristic movements or automate
their identification (splines, fourier series…)
FUTURE WORKS
A methodology for estimating the location of the surgeon’s tool has been
developed when there are occluded areas.
This work is valid when the occluded areas are produced in the short term
during the characteristic movements of the surgical tools or in presence of
obstacles.
The correct estimation can be made only when the maneuver is predicted
accordingly.
IV. Conclusions and Future Works
INDEX
I. Overview
II.Pose
Estimation
Model
III.Experiments
IV.Conclusions