This document summarizes research on using serious games to assess learning through analytics. It discusses how games can be validated using pre-post tests to ensure effectiveness and provide training data for machine learning models. Interaction data from validated games can then train models to predict learning from gameplay without exams. The researchers developed tools like a game analytics tracker, validation tool, and analysis tool to facilitate collecting interaction data, validating games, and analyzing results. Their authoring tool integrates these analytics capabilities. Future work will integrate machine learning models into the validation tool to directly provide assessment scores based on interaction data. The goal is to close the assessment loop for serious games.

1. Bootstrapping serious games to assess
learning through analytics
Freire-Morán, Manuel; Calvo-Morata, Antonio; Martinez-Ortiz, Ivan;
Fernández-Manjón, Baltasar
balta@fdi.ucm.es
Dept. of Software Engineering and Artificial Intelligence
Complutense University of Madrid
EDUCON 2023 | May 1-4 2023 | Salmiya, Kuwait

2. Serious games for assessment
Serious Games are effective at
teaching, creating awareness, changing behaviours, …
What obstacles remain to make them more useful for assessment?
Development
high costs
must design for both
fun & learning
Testing & Analytics
waterfall testing
much worse than
agile
require infrastructure
Deployment
much harder to
produce than other
LMS activities
must communicate
SG, analytics & LMS
Assessment
intrusive exams… or
validated games,
analytics, & Machine
Learning

3. ● Validating a game:
○ ensures effectiveness at stated goal
○ typically via pre-post experiments
○ provides training data for assessment
○ can also collect player interactions, so that
● ML trained on validation data can
predict learning from interactions
● Predicted learning without exam =
stealth assessment
○ non-intrusive
○ does not break player immersion
Serious games to assess learning
Improving evidence-based assessment of players using serious games Telematics and Informatics
Volume 60, July 2021, 101583 https://doi.org/10.1016/j.tele.2021.101583

4. 1. validation via
pre-post tests
2. interaction data + target
Once model is trained:
from interactions to
predictions, can be used
for assessment
3. train ML model to predict
target from interactions
proving evidence-based assessment of players using serious games Telematics and Informatics
Volume 60, July 2021, 101583 https://doi.org/10.1016/j.tele.2021.101583

5. Deploying assessment-friendly SGs
● Analytics require player ids
○ interactions from a given player should
be stored & analyzed together
○ not necessary to know real player
identity, as long as id is consistent
● Simplest possible analytics-enabled
deployment = a link from LMS
○ link must support launch +
communication to & from LMS
○ a mediator tool solves many
deployment issues, can be integrated
with server-side analytics
Using e-Learning Standards to Improve Serious Game Deployment and Evaluation EDUCON
2022, 10.1109/EDUCON52537.2022.9766573

6. Using e-Learning Standards to Improve Serious Game Deployment and
Evaluation EDUCON 2022, 10.1109/EDUCON52537.2022.9766573
What the teacher sees
1. generate LTI link
from authoring tool
2. paste into LTI widget
in LMS course
3. see how students
are playing / have
played
What really happens
see illustration right
What students see
1. click link
2. play game

7. Democratizing Game Learning Analytics for Serious Games GALA 2021
https://doi.org/10.1007/978-3-030-92182-8_16

8. ● Analytics as part of design
○ incremental improvement is better than
waterfall models, but improvement is
hard to track without analytics!
○ requires developers/designers to choose
in-game interactions to report, and
tracker component within each game
to report those interactions
○ requires tools to analyze results
● Analytics infrastructure
○ server-side component to validate,
store & retrieve interactions
○ analytics dashboards provide the
insights into collected interactions
○ infrastructure also for deployment,
validation, and finally assessment
Testing and improving an assessment-friendly SG
e-UCM Game Learning Analytics
components, based on xAPI

9. e-UCM xAPI tracker
● C#, JS and Unity.
● High-level interface for devs.
● Compatible with xAPI for SGs.
SIMVA validation tool
● Orchestrates LA activities and LTI.
● Manages users, groups (A/B), pre-post,
analysis and trace storage.
● Unity plugin for easy setup with OAuth2.0.
TxMon analysis tool
● Analyzes xAPI traces.
● Default visualizations.

10. Developing assessment-friendly games
● Game development is expensive
○ Technical expertise to code
○ Art assets may require talented artists
○ Game design (for the fun) + domain
experience (for the education)
● Assessment requires data
○ from interactions
(or it will not be stealth)
○ collected via an analytics
infrastructure
uAdventure, our authoring environment built
on Unity, integrates analytics

11. uAdventure authoring tool
● Narrative and geolocated
gameplay styles
● Easy to use by non-experts
● GLA out-of-the-box
● Exports native and WebGL
● Includes e-UCM Tracker

12. Overview: obstacles and tools

13. Next steps
● Serious Games can be a powerful method for student assessment in
complex issues (i.e. knowledge, awareness) and it is open to include AI
techniques
● Still complex but open software and standards could help in the process
● Assessment not yet fully integrated into Simva
● Building new module to
○ train a simple ML models from validation data
○ use trained models to provide predictions based on gameplay interactions
○ upload results as activity scores via LTI, making them available in host LMS
● Would close SG lifecycle, from authoring to evaluation
● All the technical solutions should take into account the ethical aspects
and comply with the privacy regulations (e.g. EU GDPR)

14. Thanks!
Contact: balta@fdi.ucm.es
This work has been partially funded by the Regional Government of Madrid (eMadrid S2018/TCS-4307, co-funded by the
European Structural Funds FSE and FEDER), by the Ministry of Education (PID2020-119620RB-I00) and by the
Telefónica-Complutense Chair on Digital Education and Serious Games.
https://github.com/e-ucm/uAdventure
https://github.com/e-ucm/simva
Our publications: https://www.e-ucm.es/publications/all