This session outlines the importance of data capture and using sound research methodology to determine the best use of your game-based learning intervention. We explore pertinent learning and behavioral theories, including outcome levels, elements of fidelity, and appraise different strategies that have both succeeded and failed in the use of games in healthcare education. More importantly, we explore the practical aspects of embedding data collection to prove the game’s impact to healthcare education and health.

1. Measuring healthcare
outcomes using serious
games, gamification, and
virtual reality
Todd P Chang, MD
MAcM
serious play conference
2018.07.11
(Manassas, VA)

2. Disclosures / Conflicts of Interest
• American Heart Association
• National Board of Medical Examiners
• Oculus from FaceBook
• Independent Contractor with SBC Med Sim, LLC
2

3. WHY MAKE HEALTHCARE
GAMES AT ALL?
3

4. 4
We do it for the patients

5. Objectives
Map out
measurable
Outcomes for both
Mediating &
Moderating games
1
Define the 4
Outcome levels for
measuring
Effectiveness of a
Game
2
Discuss three case
studies measuring
effectiveness of
Games on patient
outcomes
3

6. HOW CAN
GAMES
INFLUENCE
PATIENT
OUTCOMES?

7. 7

8. Landers et al. 2014 Sim & Gaming
Mediating
• Knowledge
• Skill
• Attitude
Moderating
8

9. MEDIATING
Going through the game improves KSA
9

10. 10

11. 1000 CUT
JOURNEY
Virtually experience life as a black
child

12. Landers et al. 2014 Sim & Gaming
Mediating
• Knowledge
• Skill
• Attitude
Moderating
12

13. MODERATING
The game or
gamification
elements
improve
engagement of
an otherwise
effective tool

14. 14

15. Landers et al. 2014 Sim & Gaming
Mediating
Going through the game
improves KSA
Moderating
The game improves
engagement of an
effective tool
15

16. 16
Outcome 01
Use
Frequency
Outcome 02
Change in
Mood
Outcome 00
Mood prior to
VR

17. 17
Outcome 01
Use
Scores
Outcome 02
Knowledge
Gain
Outcome 00
Prior
Outcomes
Outcome 03
Knowledge
Application
(Behavior)
Outcome 04
Health Metric
(Result)

18. 18
Outcome 01
Use
(Behavior)
Outcome 02
Attitude
Outcome 03
Health Metric
(Result)

19. What if the game user is
the healthcare provider,
not the patient?
19

20. 20

21. OUTCOME LEVELS
Objective 2: Kirkpatrick’s 4 Levels of Evaluation

22. Evaluation and Feedback
Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation

23. Evaluation and Feedback
Reaction
Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation

25. Evaluation and Feedback
Reaction
Learning
Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation

26. Evaluation and Feedback
Reaction
Learning
Behavior
Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation

27. Evaluation and Feedback
Reaction
Learning
Behavior
Results
Sullivan GM. J Med Educ. 2011;3(2):121-4.
Kirkpatrick’s Levels of Evaluation

28. • Barsuk et al. Crit Care Med 2009;37(10):2697-701.
• Barsuk et al. Acad Med 2010;85(10 Suppl):S9-12.
• Barsuk et al. MJ Qual Saf 2014;23(9):749-56.

29. Outcome A
Use
Scores
Outcome B
Knowledge
Gain
Outcome C
Prior
Knowledge

30. Outcome A
Use
Scores
Outcome B
Knowledge
Gain
Outcome C
Prior
Knowledge
Outcome 01
Provider Action
(Behavior)
Outcome 02
Patient State
(Result)
Outcome 03
Prior Patient
State (Result)
Outcome 04
Health Metric
(Result)

31. CAN MY GAME PREDICT
HOW MUCH IMPROVEMENT
WILL OCCUR?
A question of
Validity

34. Discriminant Validity Testing
Expected Low
Performers
Expected High
Performers
Uncovers any bias due to:
• Differences in Video Game Experience
• “Game Cheating” behavior
• Lack of Functional Fidelity in a Game
• A different target audience

35. Gerard et al. Validity
Evidence for a Serious Game
to Assess Performance on
Critical Pediatric Emergency
Medicine Scenarios. Simul
Healthc 2018;13(3):168-80.

36. 3 CASE
STUDIE
S

37. HOW TO
MEASURE
‘MULTI-PATIENT
CARE’?
A discriminant validity study

38. Background
• With higher patient volumes but fewer providers,
multi-patient care is now common
• Poor multi-patient care leads to increased errors
• Multi-patient care is not routinely taught nor
universally assessed

39. Purpose
To validate a serious games version of a pediatric ED
that measures multi-patient care based on experience:
1. Undergraduate students (jrs, srs)
2. Medical students (MS-II)
3. Resident physicians (PGY-1 through 3)
4. Attending / Fellow physicians (PGY-4+)

40. Acknowledgments to the Stemmler Fund &
BreakAway Games, Ltd

43. 43

45. Outcomes & Analyses
Predictors
• Experience
• Multi-tasking ability
• Video Game frequency
Game Outcomes
• % Sentinel Orders
• Time to 1st Sentinel Order
• Time to 2nd Sentinel Order
• Time to Discharge
• % Patients Seen
• # correct Diagnoses
• # Differential Diagnoses
45

46. Results
Undergrads MSIIs Residents Attgs
n 11 22 15 23
46

47. Results
47
0
25
50
75
100
# Patients Seen
Undergrad MSII Residents Attgs

48. Results
48
0
250
500
750
Time to SO1 Time to SO2 Time to d/c
Time-to-Critical Action (game seconds)
Undergrad MSII Resident Attg

49. Results
• MANCOVA analyses revealed a statistically
significant difference in game performance after
controlling for video game experience & multi-
tasking ability.
• The two outcomes that the game was able to
distinguish expertise in were:
– # sentinel orders
– % correct diagnoses

50. 50
The one-way MANCOVA showed that there was statistically significant
difference (p=0.03) between the skill groups on the combined dependent
variables after controlling for game play frequency, F(21, 153) = 0.700, p =
.828, Wilks' Λ = .768, partial η2 = .084, and for MTAT placement score, F(21,
153) = 1.365, p = .144, Wilks' Λ = .610, partial η2 = .152.
Follow up univariate one-way ANCOVAs were performed. A Bonferroni
adjustment was made such that statistical significance was accepted at p <
.008. There were statistically significant differences in adjusted means for
total percentage of sentinel orders completed (F(3, 59) = 31.702, p < .0005,
partial η2 = .617), and number of correct diagnoses made per patient
seen(F(3, 59) = 21.441, p < .0005, partial η2 = .522).

51. Results
51
0
0.2
0.4
% Correct Diagnoses
Undergrad MSII Resident Attg
0
20
40
60
# Sentinel Orders
Undergrad MSII Residents Attgs

52. Results
• Linear Regression for these two outcomes showed
that Expertise accounts for:
– 64.3% of the variation in # sentinel orders
– 50.4% of the variation in % correct diagnoses

53. Next Steps
• VitalSigns will emphasize the two outcomes that
the game was able to distinguish expertise in
– # sentinel orders
– % correct diagnoses
• The UI needs to be re-examined to determine why
time-to-critical action did not differ
– New input methods (e.g. mouse vs. finger vs. keyboard)

54. RESUSCITATIONS
& PROVIDER
STRESS
PHYSIOLOGY
A fidelity & validity study

55. Background
• Low-frequency, high-stakes medical events
– Need trained, competent health providers
– Are too infrequent and too high-stakes to allow
trainees to ‘practice’
• Immersive Virtual Reality is an emerging, unproven
simulation technology for low-frequency, high-stakes
events

56. Purpose
To describe ED provider stress physiology during
actual & Virtual Reality resuscitations by measuring
Heart Rates
To determine performance differences between
residents & attendings within the VR

57. Acknowledgments to Oculus from FaceBook, aiSolve, and
BioFlightVR

60. 60

61. Outcomes & Analyses
Predictors
• Experience
• Video Game frequency
• Caffeine Intake
Game Outcomes
• Heart Rate changes
• Time-to-critical Actions
61

62. Results - Heart Rate changes (/min)
Critical
ED
Events
VR
Events
70 80 90 100
Delta = +13.9*
Delta =
+6.5*
*p < 0.05

63. Results - Performance
Resident Attg
Variable n Mean
Std
Dev
n Mean
Std
Dev
p-value
Sc.4  Cricothyrotomy (sec) 4 90.50 39.32 6 31.83 16.99 0.019*

64. Results - Performance
Resident Attg
Variable n Mean
Std
Dev
n Mean
Std
Dev
p-value
Sc.4  Cricothyrotomy (sec) 4 90.50 39.32 6 31.83 16.99 0.019*
# lorazepam ordered 15 4.67 1.84 19 2.63 1.57 0.003*
# levetiracetam ordered 15 0.93 0.88 19 0.89 0.94 0.864
# fosphenytoin ordered 15 1.73 0.80 19 1.00 0.67 0.014*

65. Next Steps
• VR scenarios will emphasize the 2 outcomes that
the VR was able to distinguish expertise in
– # anti-epileptic drugs
– Time-to-advanced airway
• The UI needs to be re-examined to determine why
other time-to-critical action did not differ
– Easier inputs and assistance
• Target audience needs to be more novice

66. GAMIFICATION
IN CPR
TRAINING
A randomized-control interventional
study

67. Purpose
To determine if a PBL gamification system (points-
badges-leaderboards) improves self-initiated simulated
CPR practice frequency and skill

68. Background
• Healthcare providers do not routinely provide high-
quality chest compressions
• CPR skills begin to decay at 3 months
• Strategies to increase CPR practice frequency are
have yet to be developed
Cheng A et al. JAMA Pediatrics 2015. Sutton R et al. Pediatrics 2011.

69. Acknowledgments to the American Heart
Association (Western State Grant-in-Aid)

70. QCPR Leaderboard
70

71. Methods - Simulator
71

72. 72
Chang TP et al. Leveraging Quick Response Code Technology to facilitate Simulation-based Leaderboard Competition. Sim
Healthc 2018.

73. Outcomes & Analyses
Predictors
• Access to a Selfie-based
Leaderboard
Game Outcomes
• Frequency of self-motivated
CPR practice
• CPR performance
73

74. 74

76. 76
Control Intervention All access Total
Number of
participants
319 600 95 1014
% 31.4 59.2 9.4 100.0
Control Intervention All access Total
Number of
plays
685 1165 143 1993
Plays per
participant
2.14 1.94 1.51 1.97

77. Adjusted (for Crossover effect) results
77
Control Intervention p-value
Number of
participants
319 600
Mean plays per
person (95%CI)
1.62
(1.36 to 1.88)
1.71
(1.46 to 1.96)
0.6
Mean score
(95%CI)
89.3
(87.6 to 91.0)
90.7
(89.4 to 92.0)
0.19
• No Primary Effect
• No Crossover Effect
• No change in CPR score

78. • Halo-seeking effect & Satiety
• Insubstantial or Divisive
competition
• PBL as insufficient gamification
mechanics
• Short-term vs long-term
expectations

79. Circling Back
Map out
measurable
Outcomes for both
Mediating &
Moderating games
1
Define the 4
Outcome levels for
measuring
Effectiveness of a
Game
2
Discuss three case
studies measuring
effectiveness of
Games on patient
outcomes
3

80. Contact information
• Todd P Chang, MD MAcM
Children’s Hospital Los Angeles / Keck School of Medicine at the
University of Southern California
80

81. 81