To support complex tasks in neurosurgery master-slave robotic systems with compact robotic forceps have been developed. Before these systems are inserted in the surgical room, it is necessary to ensure their robust and safe operation. Virtual Reality (VR) simulation may complement this development process by providing a software platform in where the robot control algorithms can be tested without damaging the expensive prototype. In recent years the increase in the computing power along with newer algorithms optimizations enables real time simulation with both visual and physical realism, allowing acquisition of kinematic data for surgical skills analysis, and synthetic data generation for training machine learning algorithms; ultimately interactive simulators provide a safe environment in where neurosurgeons can repeatedly practice without impacting the patient’s health. Despite state-of-the-art simulators, there is little work in simulation of robot-assisted microsurgery. In this talk we present the development of VR simulations for robotic neurosurgery focusing in the micro suturing task with haptic feedback, detailing the methods and current applications. This work was funded by ImPACT Program of Council for Science, Technology and Innovation, Cabinet Office, Government of Japan.

1. VR simulations for neurosurgery
〇Saul Heredia1), Kanako Harada1,2), Hirofumi Nakatomi3),
Nobuhito Saito3), Akio Morita4), Mamoru Mitsuishi1)
1)School of Engineering, The University of Tokyo
2)Japan Science and Technology Agency
3)Department of Neurosurgery, The University of Tokyo Hospital
4)Department of Neurosurgery, Nippon Medical School
the presenting author has no conflict of interest to disclose
3月1日（金）16:00〜18:00 A会場（タワーホール）
ImPACTセミナー バイオニックヒューマノイドが拓く新産業革命
IS-8

2. 2Introduction
 Simulation is the imitation of a real-world process or system over time.
 Useful when the real system cannot be engaged, may not be
accessible, may be dangerous, or it is being designed but not yet built.
Simulation
Physical
Computational
High precision
Engineering design
& analysis
Interactivity
virtual reality,
videogames
Not real-time
Real time

3. 3
1980s
Military flight
simulators.
Rise of gaming industry,
high-resolution graphics
1990s-2000s
2016
Commercial VR
Headsets (Oculus)
Gemini, Apollo
simulators (NASA)
1960s-1970s
Virtual Reality (VR) Simulation
 “Technologies that allow
people to interact with
3D computerized
databases in real time
using their natural
senses and skills” [1].
 Applications: medicine,
education, engineering,
architecture, design, etc.
 Evolved along with the
development of digital
computers and
videogames.
[1] McCloy, R., & Stone, R. (2001). Science, medicine, and the
future: Virtual reality in surgery. BMJ: British Medical Journal,
323(7318), 912.
Telepresence
and robotics
Computational
mechanics
Computer
graphics
VR
Simulation
Hardware
interface
System
modeling
3D rendering

4. 4Simulation in neurosurgery
 Physical simulation. Cadaver,
animal tissues, silicone rubber, 3D
printed models, etc.
 Advantage: Realistic
 Disadvantages: Costly, qualitative
evaluation.
 VR-based simulation.
Computational models and
algorithms.
 Pros: Low cost, repeatability,
quantitative evaluation.
 Cons: speed-accuracy trade-off,
approximated methods are used.
[1] Breimer, G. E., et al. (2015). Design and evaluation of a new synthetic brain simulator for endoscopic third ventriculostomy, Journal of Neurosurgery: Pediatrics PED, 15(1), 82-88.
[2] Delorme, S., et al. (2012). NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Operative Neurosurgery, 71(suppl_1), ons32-ons42.
[1]
[2]

5. 5VR simulation and robot-assisted surgery
Benefits
• For engineers: prototype testing before production, trying automatic control
algorithms.
• For neurosurgeons: practicing, preoperative planning, skills assessment.
Our objective: development of VR simulators for robotic neurosurgery.
Existing simulators mainly
focus in the da Vinci surgical
system, devised for
laparoscopic surgery but not
for microsurgery.
[1] Julian, D., et al. (2018). A comparative analysis and guide to virtual reality robotic surgical simulators. The International Journal of Medical Robotics and Computer Assisted Surgery, 14(1), e1874.
[1]

6. 6Transsphenoidal tumor resection
[1] Nukariya, H., et al. (2016) Development of a miniature neurosurgical robotic system with multi-DOF forceps targeted for tasks in deep spaces. ROBOMECH2016, 1A1-03a1.
Tumor deformable model simulated
using position based dynamics [2]
Triangle mesh
Tetrahedralization
Particles connected by springs
7 DOF Robot for micro neurosurgery [1]
Articulated forceps
Simulation of transsphenoidal pituitary
tumor resection

7. 7Evaluating dynamic motion scale using VR simulation
1/10
1
1/3
Region-based dynamic motion scaling
Tumor
Movement is scaled down as
approaching critical tissues
[1] Heredia‐Pérez, S. A., et al. (2019).
Virtual reality simulation of robotic
transsphenoidal brain tumor resection:
Evaluating dynamic motion scaling in a
master‐slave system. The International
Journal of Medical Robotics and
Computer Assisted Surgery, e1953.

8. 8Microvascular anastomosis
Suture 0.02
mm
< 0.7 mm
[1] Mitsuishi, et al. (2013). Master-slave robotic platform and its feasibility study for micro-neurosurgery. The International
Journal of Medical Robotics and Computer Assisted Surgery, 9(2), 180-189.
Master-slave robotic system
MM3 Robot
End-to-end anastomosis of micro vessel is task that
require high precision, a Master-Slave robotic
platform was developed to assist this procedure [1].
Neurosurgeons require hours of training using
artificial vessel to proficiency. VR simulation will
provide a safe platform for repeatedly practice
without damage the robot.
Needle diameter
0.07 mm
Video

9. 9Simulation method
Joints
Linear blend
skinning for
visualization
Rigid bodies
,m I
,k DMass,
moment of
inertia
Stiffness,
damping
, ,E 
,k D,m I
Continuum elasticity
Rigid body dynamics

Spline extrusion for visualization
…
Rigid bodies with capsule collision geometry
…
,k D
Rotational spring
,m I
Rigid body
,m I
Rigid body
For real-time simulation, the objects are approximated using rigid body dynamics.
Thread Needle
Vessel
Simulated as a rigid body connected
to an end of the thread.
Finite element
simulation
Rigid-body
based
deformation
[1] S. Heredia, et al. 2018. Development of microvascular anastomoses simulator for robotic surgery. The 18th Japan VR Medical Society Academic Conference.

10. 10Developed system
VR simulator
3D monitor
Realistic rendering
Haptic device
force feedback
Software was developed
in C++ using the NVIDIA
PhysX engine to solve
the rigid body dynamics
Features:
• Real time (60 FPS)
• Extract simulation information
(trajectories, collisions, etc.)
• Visual realism
Video

11. 11Research workflow
Data
Simulator
Needle grasping
Needle adjustment
Positioning (Right)
Insertion (Right)
Extraction
Looping
Thread grasping
Knot tying
Positioning (Left)
Insertion (Left)
Automated surgical
activity annotation
Frames
Surgical skills
from expert
user
Current
activity
Training data
for activity
recognition
from video
Robot
Haptic
assistance
Operator

12. 12
 We work in the development of VR simulators for neurosurgery,
focusing in real-time interaction and visual realism.
 VR simulators provide a safe environment for testing robot
control algorithms, as well for surgical training.
 Data can be extracted from simulator for surgical skill analysis,
or used as training data for machine learning.
Acknowledgments
This work was funded by ImPACT Program of Council for Science, Technology
and Innovation, Cabinet Office, Government of Japan.
Closure