This document discusses how learning analytics can help take a more scientific approach to evaluating serious games. It presents three examples: 1) Using the xAPI standard to track all in-game interactions, 2) Applying game learning analytics to collect, analyze, and visualize learner data from a first aid game pilot study, and 3) Conducting a case study using analytics from a game for teaching subway navigation to people with Down syndrome. The document argues that learning analytics can help create better games and formally validate their impact on learning.

1. Using Learning Analytics to support
a more scientific approach to
Serious Games: Three Examples
Baltasar Fernandez-Manjon
balta@fdi.ucm.es , @BaltaFM
e-UCM Research Group , www.e-ucm.es
Jornadas eMadrid, 2017, 04/07/2017
Realising an Applied Gaming Eco-System

2. Can Learning analytics help?
- Do games actually works?
- Usually, no full formal evaluation has been carried out
- Limited number of users
- Formal evaluation could be as expensive as creating
the game (or even more expensive)
- Difficult to deploy games in the classroom
- Teachers have very little info about what is happening
when a game is being used

3. xAPI Serious Games application profile
New standard interactions model
developed and implemented in
Experience API (xAPI) by UCM with ADL
(Ángel Serrano et al, 2017).
The model allows tracking of all
in-game interactions as xAPI traces
(e.g. level started or completed,
interactions with NPC or game items,
options selected, score increased)
https://www.adlnet.gov/serious-games-cop

4. Analytics and Game Learning Analytics
Game Learning Analytics
(GLA) for Serious Games:
- collect, analyze and
visualize data from learners’
interactions
Can GLA be systematized?
Realising an Applied Gaming Eco-System

5. Systematization of Analytics Dashboards
As long as traces follow xAPI format, these analysis
do not require further configuration!
Also possible to configure game-dependent analysis and
visualizations for specific games and game characteristics.

6. First Aid - CPR validated game
• Collaboration with Centro de Tecnologias Educativas de Aragon, Spain
• Identify a cardiac arrest and teach how to do a cardiopulmonary
resuscitation to middle and high school students
• Validated game, in 2011, 4 schools with 340 students
Marchiori EJ, Ferrer G, Fernández-Manjón B, Povar Marco J, Suberviola González JF, Giménez Valverde A. Video-
game instruction in basic life support maneuvers. Emergencias. 2012;24:433-7.
Available at http://first-aid-game.e-ucm.es

7. BEACONING GLA Pilot: Experiment description
• 227 students
• 1, 2, 3 and 4 year of ESO and 1 year of
BACHILLERATO
• From 12 to 17 years old
• Only 4 morning sessions
• Game rebuilt with uAdventure
• Included analytics using RAGE tracker
based on xAPI specification

8. Experimental design
3 Steps:
• Pre-Test
• Gameplay
xAPI LA
• Post-Test

9. Learning compared to original experiment
• Original experiment
with the game
• Original experiment,
control group
• Current experiment
(from 8 to 9.8 out of 15)
Lower learning but still
significative!
Replicability of Results

10. Predicting post test score
1.With pre test information + game traces
Greater importance of:
- score in pre test
- game habits
- interactions with game elements

11. Predicting post test score
2.Only with game traces
Greater importance of:
- interactions with game elements
- scores in game levels
Long-term goal: predict score solely with in-game actions and,
therefore, avoid the pre test.

12. Case study: Downtown
• Serious Game designed and develop to
teach young people with Down
Syndrome to move around the city using
the subway
• Evaluated with 45 people with cognitive
dissabilities
• Full Analysis of xAPI GLA info under way
• Audience: People between 15 and 40 y/o with
Down syndrom

13. Case Study: Downtown
• From user requirements to a game
design and its observables
• Know more about how and what is
learn by people with Down Syndrome
13

14. Next steps: Cyberbullying

15. Desing based on research studies
Seminario eMadrid sobre Serious games 2017-02-24 15

16. Social Networks Risks
Seminario eMadrid sobre Serious games 2017-02-24 16
Implications of the social networks

17. Experimental design
3 Steps:
• Pre-Test
• Gameplay
xAPI LA
• Post-Test

18. The experiment: initial validation
With students from 3 institutes (Madrid, Zaragoza, Teruel)
223 pre-post and gameplays (121 males, 102 females)

19. First results
Age average
Pre-test value
average
Post-test
value average14,2
6,385,72
Once validated Ministry of Education is
interested in creating a NOOC for teachers
training where the game is used

20. Integrating xAPI LA in games authoring
Previous game engine eAdventure (in Java)
• Helps to create educational
point & click adventure games
Platform updated to uAdventure (in Unity)
Full integration of game learning analytics
into uAdventure authoring tool
No extra effort required to integrate default analytics into uAdventure
games!

21. Game Learning analytics can help us to:
create better games and to (formally)
validate games
• Moving from pre-post to Learning
Analytics based evaluation
• To use games as assessments
Conclusions

22. 22
Thank You!
Gracias!
¿Questions?
• Mail: balta@fdi.ucm.es
• Twitter: @BaltaFM
• GScholar: https://scholar.google.es/citations?user=eNJxjcwAAAAJ&hl=en&oi=ao
• ResearchGate: www.researchgate.net/profile/Baltasar_Fernandez-Manjon
• Slideshare: http://www.slideshare.net/BaltasarFernandezManjon