The implementation of guidance in robotic microsurgery is challenging since object detection and activity recognition are required. On the other hand, virtual reality simulation providing full information of the virtual object allows easy implementation of guidance algorithms. In this work, we implemented haptic assistance inside a virtual environment for posture correction during execution of peg transfer task, and the result showed a decrease in the task completion time.

1. Haptic assistance for
robotic surgical simulation
○Saul Alexis Heredia Pereza，Kanako Haradaa,b，Mamoru Mitsuishia
aThe University of Tokyo
bJapan Science and Technology Agency
Mitsuishi Harada Lab.

2. 2Introduction
• Providing assistance according
to the context of the surgical
task require:
• Since it is challenging to
implement in a real scenario, we
use VR simulation to test our
haptic guidance framework.
Mitsuishi Harada Lab.
Master-slave surgical robotic system MM-3
[1] Mitsuishi, M., Morita, A., Sugita, N., Sora, S., Mochizuki, R., Tanimoto, K., ... & Harada, K. (2013). Master-slave robotic platform and its feasibility study for micro-neurosurgery. The
International Journal of Medical Robotics and Computer Assisted Surgery, 9(2), 180-189.
Master side
Using haptic devices
to influence the user
movement towards a
target posture.
Slave side
Constraining the robot
kinematics using high
level control
algorithms.
Guidance in robotic microsurgery
Apply the optimal
corrective action
• Object recognition
with posture
information
• Activity recognition
and prediction

3. 3Materials and methods
• We focus on the peg transfer task which is
typically used for surgical skills
assessment.
• Existing VR simulators that implement the
peg transfer task were conceived for
laparoscopic surgery.
• We developed a VR simulator based on a
standard peg transfer board and triangles
kit commercialized by FLS [1], downscaled
to a factor of 1:4, to use along with
⌀3.5mm shaft microsurgery tools.
Mitsuishi Harada Lab.
[1] Fundamentals of Laparoscopic Surgery. https://www.flsprogram.org/(accessed 2018/7/16)

4. 4Forceps gesture detection
Grabbed
0
1
time
Grasp
if struggling and tips are touching a block
then grabbed=1
else grabbed=0
Opening Closing
Closed
Opened
Struggling
?
Is block
Obstacle
Grabbed
0
1
time
Release
Using the information from the virtual environment we detect the user gestures
such as grasping or releasing using finite state machine and decision trees.
……
Mitsuishi Harada Lab.

5. 5Activity recognition
G=grasp
R=release
G RLeft tool
Right tool G R
G R
time
G
…
R
Grasping block Transferring block Placing block on peg
Mitsuishi Harada Lab.
After a forceps gesture is detected, assuming the user always follows the
same sequence, we can determine the current activity:

6. 6Automatically annotated video testMitsuishi Harada Lab.
Peg transfer task: pick up a block with the non-dominant hand, transfer the block to the
other hand, and place the block on the opposite side peg.

7. 7
F
T
• For each activity we define: the instrument, the object, and the
target
Object transform
relative to
instrument’s end
effector
Haptic guidance virtual fixture framework
Instrument
Object
target
transform
End
effector
transform
E T
1
X TO
    ,a R    toAxisAngle
End effector
target transform
1
0 1
R p
XE  
   
   
 
Error
F kp Dp
T k a
 
 
 

Force feedback
Torque feedback
O
Haptic device
Phantom
Premium 1.5
6DOF (3D
systems, USA)
Mitsuishi Harada Lab.
• Given the current activity, and the objects’
posture, a force and torque is exerted

8. 8Haptic guidance in peg transfer task
Activity
Block grasping Block transferring Block depositing onto peg
Instrument Left tool Right tool Right tool
Object Tool tip Tool tip Block held by right tool
Target Closest block to left tool Block held by left tool Closest peg to held block
Mitsuishi Harada Lab.

9. 9Results
• We tested the proposed method
under two experimental
conditions: without force
feedback (NFF) and using our
haptic guidance virtual fixture
implementation (GVF).
• The recorded data from the
simulation was postprocessed to
calculate the total task time, and
the average time for each step.
Mitsuishi Harada Lab.
The results from four different operators
32% shorter
decrease
of 44%
decrease
of 16%
decrease
of 14%

10. 10Discussion
• In the absence of force feedback, the operator required more effort
to execute the task.
• Amongst the three analyzed activities, the block grasping was more
benefited from the haptic guidance.
• The results suggest hat haptic guidance will play an important role
in robotic microsurgery to reduce the execution time of tasks that
require bimanual skills and hand-eye coordination.
Mitsuishi Harada Lab.
Acknowledgments
This work was funded by ImPACT Program of
Council for Science, Technology and Innovation,
Cabinet Office, Government of Japan.