Embracing The Future Of Serious Gaming and Immersive Technologies In Medical Education Recent technological advances have given rise to a variety of consumer-level immersive technologies including virtual and augmented reality headsets such as the HTC VIVE that provide the opportunity to develop highly interactive and immersive serious games and virtual simulations. In this presentation a discussion of immersive technologies and their application in serious games and virtual simulations for medical education and training will be provided.

1. Embracing The Future Of Immersive
Technologies In Medical Education
Bill Kapralos (PhD)
University of Ontario Institute of Technology, Oshawa, Ontario, Canada.

2. Overview:
Before We Start
Immersive Technologies
Immersive Technologies: Examples
Moving Forward
Before We Start
Fear of Technology (1):
Not a New Phenomenon

3. (Socrates, 470-399 BC) → “This discovery of yours will create
forgetfulness in the learners’ souls, because they will not use
their memories; they will trust to the external written
characters and not remember of themselves.”
Conrad Gessner (1516 –
1565) → information overload
Must regulate the trade so
public won’t suffer with
“confusing and harmful
abundance of books.”
Fear of Technology (2):
Not a New Phenomenon (cont.)

4. More recently (very recently!!)
CNN → “Email hurts IQ more than pot”
Telegraph article → “Twitter and Facebook could harm
moral values” and “Facebook and MySpace generation
'cannot form relationships’”
Daily Mail → “How
using Facebook could
raise your risk of
cancer.”
Something to Consider
(1):

5. “Anyone who tries to make a distinction
between education and entertainment doesn't
know the first thing about either”
Marshall McLuhan (1911 - 1980)

6. Immersive Technologies
Simulation (1):
Thousands of Years
Current Emphasis (e.g., U.S. Military)
Immersive technologies (virtual reality, augmented reality, and

8. What is Immersive Technology (1):
Blurring the Line
Technology that blurs the line between the physical world and
digital or simulated world, thereby creating a sense of
Augmented reality (AR) and virtual reality (VR)
Coupled with serious games and virtual sims
very engaging learning applications

9. What is Immersive Technology (2):
Single person could perceive
prerecorded experience via visual,
sound, smell, vibration, and wind
Why All the Hype Recently?
Increase in computational power and decrease in size of
electronic components has led to decreasing cost and rising
availability of consumer-level immersive tech
Helped advance recent adoption of virtual simulation hand
and arm tracking technologies using controllers such Leap
Motion hand sensor, Microsoft Kinect, and Thalmic Labs

10. What is Immersive Technology (3):
Myo are allowing for development of novel interaction
methods and techniques.
Why All the Hype Recently? (cont.)
Such devices provide more natural and immersive interactions
→ help overcome the limitations associated with traditional
keyboard and mouse-based humancomputer interactions
where tasks are performed very differently to a real-life
situation

11. What is Immersive Technology (4):
Providing designers/developers of virtual sims great
freedom to develop highly immersive applications
VR industry → $1.37 billion (USD) in 2015 to $33.90
billion (USD) by 2022 (Markets and Markets 2016)
Goldman Sachs (2016) → AR and VR to be $95 billion
industry by 2025 (Hall and Takabashi 2017).

12. What is Immersive Technology (5):
What is Driving the Advances?
changing
Has been suggested that VR/AR will “drive a revolution in
role under the following health applications
Education and training
Surgical planning
Telemonitoring
Patient experience
Treatment and therapies
AR-enhanced surgery and patient care.
https://1189 medtechboston.medstro.com/blog/2016/05/24/16045/

13. Why Bother (1):
Why Use Immersive Technologies (AR/VR)?
Cost effective alternative → sim labs are expensive!
Offers trainees opportunity to safely and cost-efficiently train
until they reach a specific competency level
Convenient → mobile VR…
Team training → users can be located anywhere
Laparoscopic surgeons err less by playing video games

14. (NY study) → hand-eye coordination (more later)
Potential Applications:
Cognitive → decision making … Technical
skill development ?
Current Buzz (1):
Virtual and Augmented Reality
It is all about virtual and augmented reality (VR/AR)
Computing speed has increased tremendously and

15. Immersive Technologies:
Examples
VR and Cardiology Education (1):
VR Headset to See Inside the Infant Heart

16. Stanford University, School of Medicine
"The heart is a complicated three-dimensional organ, and it's really hard to describe what's
going on inside of it — especially when something is going wrong . . . Virtual reality
eliminates a lot of that complexity by letting people go inside the heart and see what's
happening themselves — it's worth way more than a thousand words.“ Dr.
David M. Axelrod, clinical Assistant Professor of Pediatric Cardiology, Stanford Uni.
Room Scale VR Medical Education (1):
Anesthesia Training

17. Serious game for epidural procedure training
Virtual Patients (VPs) (1):
Medical Education Environment of Future
Low risk educational tool

18. “Virtual patients allow students to learn without putting real patients at risk…
No actual patients are harmed in the process of learning from virtual patients.”
Norm Berman, MD, Professor of pediatrics at the Geisel School of Medicine at
Dartmouth.
Framework to facilitate creation of VPs and dialogue scenarios
to teach about verbal/nonverbal comm.
Clinicians can create complex dialogue scenarios and link
it directly to a VP → lip-sync, facial expressions
More on VPs in next presentation…
Haptics (1):
But is it All About Viewing?

19. Big component of immersive technology is interaction
Traditional input/interaction devices are not ideal
-machine touch interactions
High quality haptic devices capable of simulating the sense
of touch with high fidelity are cost prohibitive
Consumer-level haptic devices have recently become
Geomagic Sculpt Novint Falcon
Haptics (2):

20. Perceptual-Based Rendering
Can sound “trick” the perceptual system into believing haptic
fidelity provided by low-end haptic devices is better than it
Conducted preliminary studies that examined this
Electroencephalogram (EEG) monitoring with aim of
correlating brain signals to haptic-sound interaction

21. Designing Immersive
Experiences

22. Overview (1):

23. Not a Trivial Task!

24. Overview (2):
The Development Team
Immersive sims/serious game development assumes an
interdisciplinary team
Medical professionals → content experts
Educational experts → instructional designers
Artists
“Techies” → computer scientists, game developers
Can lead to problems
Content experts may not be aware of technical limits

25. Overview (3):
Game developers may not understand importance of subtle
details
Combining Instruction and Game Design
Requires a synergy between instructional design and game
design
Seemingly opposed approaches that have radically different
histories
Educational aspect must be primary consideration
The “book of content” already exists before the designer starts
→ the game must be about the “book”
Two “prime directives” to follow simultaneously

26. Overview (4):
1. To make the game/application engaging
2. To follow the “book” → to be accurate
Combining Instruction and Game Design (cont.)
Bumpy past → according to one game designer, “the
instructional designers brought into the creation of a learning
game because they supposedly know how to get people to
learn, typically suck the fun out of any game they get their
hands on”
Many examples of “bad serious games”
Edutainment era of the late 1970s and 1980s

27. Overview (5):
Typically, a bad serious game is due to a lack of proper
design and often times, lacking appropriate instructional
design → US Army study
The Development Team (cont.)
Very important to the design and development process for the
game developers to experience first-hand “a day in the life” of
the practitioner
Agile Instructional Design
Basic idea is that it is emergent, adaptive, iterative, and model-
based → in the sense of creating prototypes rather than in the
sense of following prescriptive models

28. Overview (6):
Agile (rapid) design → iterative and incremental
development where functional, even if incomplete version
is created as soon as possible

29. Moving Forward
Show Me the Proof (1):
Greater Evidence of Effectiveness
Greater studies are required that methodically verify the
effectiveness of virtual simulations and serious gaming
Some proof does exist but it is scattered and sporadic fMRI
studies have validated effectiveness of VR and games in
treatment of phobias and in distracting patients in process of
burn treatment, chemotherapy
Seymour et al. compared performance of surgery
residents who trained with VR t
better performance for residents who trained with

30. VR during laparoscopic cholecystectomy procedure
Dig Rush → “therapeutic video game to treat
Amblyopia” by prescription

31. Something to Consider (1):
How Can Virtual Sims be Combined with Other Ed.
Techniques to Enhance Learning?
Open area of research → many questions remain to be
answered here!
Current Generation of Learners
Challenging the current (outdated) teaching methods
Prefer “learning by doing” and require interaction
Spend considerable amount of time playing games
motivated by surpassing their peers and being at top of their
peer group throughout most of their student life”

32. Something to Consider (2):
It’s Not Only About the Technology
Need to think about how we use the technology
Technology isn’t the answer to everything → it has its
limitations and there are many things we can’t do
Proper design required → majority of such applications are
lacking instructional design and design in general
Must carefully consider team and resources to create
effective product that makes good use of technology
Most students lack computing power to run latest
VR applications → computationally expensive
Many unanswered questions, so ongoing research is needed
to inform what we do → many opportunities

33. Something to Consider (3):
How Much Realism/Fidelity is Needed ? (cont.)
Given that perfect multi-sensory fidelity is impossible to
achieve, with current technology
How much realism is needed to maximize learning?
Unclear if high realism is needed for either enjoyment or to
maximize learning & when some cues are too realistic

34. Something to Consider (4):
→ “Uncanny valley”
effect

35. The Future (1):
Technology Will Continue to Improve
Imagine the opportunities in five years, let alone 10, 15
Microsoft HoloLens 2 will include AI coprocessor for
implementing deep neural networks (DNNs) →
algorithms for pattern recognition and machine learning
(possibly key to solving some of the biggest challenges
facing AR today)
Electronic skin (e-skin) → soft, bendable, and wearable
tech that allows user to manipulate objects that exist only
in virtual world

36. The Future (2):
Technology Will Continue to Improve (cont.)
Interfaces and interaction methods are becoming more natural
→ optimized for humans rather than computers
Advancements in gesture tracking, motion tracking, eye
tracking, and other technologies → laying groundwork for
natural interaction methods that will be essential for the
success of augmented reality
Gesture and motion tracking
Eye tracking
Voice
Facial tracking
Brain control

37. The Future (3):
Technology Will Continue to Improve (cont.)
IBM Watson → Q & A computer system capable of
answering questions posed in natural language
IBM & Unity partner to bring the power of AI to developers
with IBM Watson Unity SDK (Feb 2018)
Allow developers to easily integrate Watson cloud
services into Unity applications → configure projects to
understand speech, talk with users, and understand the
intent of a user in natural language
Easy for developers to take advantage of modern AI
techniques through set of cloud-based services

38. The Future (4):
Coupling Simulation with Makerspaces

40. Think of the opportunities!
Boston Children’s Hospital
Simulator Program’s Engineering
Workshop (Makerspace)
3D printed surgical drill attached to
consumer-level haptic device with simple
electronic components to simulate surgical
drilling
“Food for Thought” (1):

41. Chocolate Covered Brocolli & Brussel Sprouts
the outside it looks nice
Not just about using the latest and greatest technology →
proper design and testing is a must!

42. And Finally… Greater
Work Remains so This is
Definitely not the End…
One Final Quote (1)

43. “We don’t stop playing because we grow old;
we grow old because we stop playing”
George Bernard Shaw (1856 – 1950))