This document describes the development of a virtual reality simulator for robotic brain tumor resection. Key components included realistic tissue deformation and cutting, force feedback, and dynamic motion scaling. Preliminary results found that using force feedback and dynamic motion scaling reduced collisions with healthy brain tissue and improved accuracy. Future work will add full robot kinematics, virtual fixtures, and other enhancements to simulate the procedure more realistically. The goal is to create an effective training tool to help surgeons practice complex neurosurgeries.

1. Development of a Virtual Reality
Simulator for Robotic Brain
Tumor Resection
Saul Heredia

2. Introduction
• Objective: Develop a Virtual Reality Neurosurgery Simulator.
• Specifications:
• Real time, Collision Detection, Force feedback, Visual realism, 3D Stereo, Soft
Tissue Interaction (deformation, cutting), Dynamic Motion Scaling
• Justification: Training, Preoperative Planning, Skills Assessment.
http://www.intechopen.com/books/endoscopy-innovative-uses-
and-emerging-technologies/endoscopic-skull-base-surgery-in-the-
pediatric-patient

3. • Challenges:
• Articulated tool interaction: most
simulators focus on straight tools,
limiting the interaction to the
tooltip.
• Mesh deformation and cutting:
State of the art simulators are
mostly based on Finite Element
Method, changes in the topology
are computationally heavy.
• Case of study: Nasal approach
pituitary tumor resection.
(Xuemei Liu, 2015)

4. Simulator Components
• Physical Components • Logical Components
Haptic
Device
Rigid Body
Dynamics
Soft Body
Deformation
3D Stereo
Rendering
User
Motion
Scaling
Actual
Position
Target
Position
Exerted
Force
Collision
Avoidance
Simulation
Inverse Motion
Scaling
PID
Controller
PC
Haptic
Interfaces
Foot
Pedal
PC
Stereo
Projector
UDP
Master
Simulation
(Slave)
User
Video
Converter

5. Methods
• Rigid body collisions: Using the
physics engine NVIDIA PHYSX,
shapes are approximated using
convex decomposition.
• Deformable models: Using the
engine NVIDIA FLEX, a Position-
Based Dynamics solver.
(Miles Macklin, 2014)(Khaled Mamou, 2009)

6. Force Feedback
ˆx
Actual position
x
Ideal position
Obstacle
Actual
Target
F=?

7. Dynamic Motion Scaling
as
bs
ad
bd
Forceps
 1 a bs s s   
0
a
a b
d
d d



 
 
No interpolation
Linear interpolation
aM
bM
Low
Normal
High
0.1
1/3
1
No interpolation Linear interpolation

8. Visualization
Normal mapping
Environment mapping Ambient occlusion Depth of Field
Lens distortion
Lens dirt
Inter-axial distance
Convergence
Stereo projection
Left eye Right eye
3D depth perception
OpenGL – GLSL Shaders

9. Experiment
• Task: Remove the membrane,
and extracting the tumor
• Conditions
• Participants
• 3 Neurosurgeons
• 2 Medicine students
• 4 Engineering students
• Participant were asked to fill a survey.
MembraneTumor
Normal tissue
Testing
condition
Force
Feedback
Motion Scaling
1 No Constant
2 Yes Constant
3 Yes Dynamic without interpolation
4 Yes Dynamic with linear interpolation

10. Measurements
• Tooltip trajectories,
velocities
• Collision points
• Exerted forces
• Clutch and forceps usage
• Percentage of Removed
Tissue
(Azarnoush, 2015)

12. Preliminary Results
0
10
20
30
40
50
60
70
80
1 2 3 4
%
Condition
Percentage of removed tumor
0
0.5
1
1.5
2
2.5
3
1 2 3 4
%
Condition
Percentage of removed brain tissue
0
0.01
0.02
0.03
0.04
0.05
0.06
1 2 3 4
Condition
Percentage of removed brain tissue /
percentage of removed tumor
0
2
4
6
8
10
12
14
1 2 3 4
Condition
Foot pedal usage

13. Results from surveys
Question Strongly
disagree
Disagree Nor agree or
disagree
Agree Strongly
agree
To extract the tumor completely was difficult to perform. 1 5 3
The tool movement was consistent with my intention. 2 3 3 1
The forceps grasping was accurate. 3 1 3 2
The force feedback due to touching soft tissues was
noticeable.
1 3 5
Motion scaling was helpful during interaction with the soft
tissues.
3 4 2
The behavior of the soft tissues was natural. 2 5 2
The force feedback was disturbing while removing the
tissues.
3 1 4 1

14. Conclusions
• With the use of Force Feedback (FF)
the number of collisions with the bone
was reduced in 50%.
• The use of Dynamic Motion Scaling
along with FF reduced the number of
collisions with the healthy brain tissue
in 60% an 80% with interpolation,
• Although the time for completing the
task is increased, the accuracy is
improved, making the process safer.
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
1 2 3 4
Condition
Collisions
Tool vs Bone Tool vs Tool Normal Tissue

15. Future Work
• Take into consideration the shape of the full robot.
• Add virtual fixtures to simulation.
• Simulating the bone drilling procedure.
• Simulating bleeding and vacuum tool interaction with the fluids.
• Force Feedback with 7 DOF output (3 Linear force, 3 Gimbals torque,
and forceps aperture)
• Add support to virtual camera controlling using foot pedals.

16. Video

17. References
• Azarnoush H., et la., (2015). “Neurosurgical virtual reality simulation
metrics to assess psychomotor skills during brain tumor resection”. CARS,
10(5), 603-618.
• Mamou, K., & Ghorbel, F. (2009, November). A simple and efficient
approach for 3D mesh approximate convex decomposition. In 2009 16th
IEEE (ICIP) (pp. 3501-3504).
• Miles Macklin, et al. “Unified Particle Physics for Real-Time Applications”,
SIGGRAPH 2014.
• Xuemei Liu, et al., “Deformation of Soft Tissue and Force Feedback Using
the Smoothed Particle Hydrodynamics,” Comp. Math. Meth. Med., vol.
2015, Article ID 598415, 10 pages, 2015.