Simva: Simplifying the scientific validation of serious games

1. Simva: Simplifying the scientific
validation of serious games
Baltasar Fernández-Manjón
Ivan J. Pérez-Colado, Antonio Calvo-Morata,
Cristina Alonso-Fernández, Manuel Freire,
Iván Martínez-Ortiz
@baltaFM
IEEE ICALT 2019
Brazil, 15-18 July 2019

2. Validation of Serious Games
Serious games effective tool applied in multiple fields.
However, still need for:
● Evidences of the effectiveness of the games
● Validation of the serious game design
Methods:
● Pre-Post questionnaires
● Survey
● Interaction data (Game and Learning Analytics)
Very complex process: time, cost, human errors...

3. Pre-Post Questionnaires Issues: Collection
Paper-based questionnaires:
● Waste of time in preparation, printing before each experiment and distributing
them to participants
● Transcribe responses later for analysis (arduous task, error prone)
Computer-based questionnaires:
● Internet access is mandatory for online questionnaires
● If questionnaires are offline, need to collect them (tedious task)

4. Pre-Post Questionnaires Issues: Storage
Paper-based questionnaires:
● Huge accumulations of paper
● How, when and where destroy the data
Computer-based questionnaires:
● Need backup not to lose data

5. Pre-Post Questionnaires Issues: Relate data
How to relate all the data of the same user?
Different information collected from users:
● Pre-post questionnaires
● Interaction data
How to identify the users?
● Name, National Document Non-anonymous data
● Random User Identifier Anonymous data

6. Simva
Simva is a tool for scientific validation of serious games.
Goal: to simplify the previously-identified issues:
● Connection with surveys system and learning analytics system
● Management of participants and surveys
● Data storage: questionnaires responses and traces of interactions
● Control while experiments are in play: questionnaires finished, data sent

7. Simva: Surveys and Learning Analytics
Simva is connected with:
● LimeSurvey
○ Create and manage surveys
○ Link users and surveys (access control)
○ Export and download responses
● e-UCM Analytics
(Game Learning Analytics System)
○ Connection with game tracker
○ Collect and download user traces

8. Simva: Participants management
Users with the same identifier in LimeSurvey and e-UCM Analytics
Participants management: creation of
classes & users
Pseudonymization
Anonymous 4-letter tokens
automatically provided: same identifier
for pre-test, post-test and game traces
PDF with identifiers to distribute them in
the experiment: allow anonymization
while teachers can relate user and token

9. Simva: Data Storage
Store questionnaires responses and interaction data
Researcher or session managers
can easily know the users
(given by token) that completed
all steps
Download all data in the
same format (CSV) to
analyse it in batch

10. Simva: Control during the experiments
The game can check Simva
to see if user has completed
some survey and the game
● surveys status: started,
finished, not found
Game will only allow users
to continue if they have
completed the previous step
(e.g. only access game after
completing pre-test)

11. Simva: Additional metadata
Allow additional metadata to be stored in
Simva to complement experiments
For instance, to conduct A/B experiments
● In the same experiment,
distribute two different
questionnaires for each
half class

12. Conclusions
● Validation of serious games is a complex, error-prone process
● Simva tool aims to simplify all the possible issues
○ Before the experiments:
■ Managing users & surveys
■ Providing anonymous identifiers to users
○ During the experiments:
■ Collecting and storing all data collected
■ Relating different data from users
■ Allowing additional metadata
○ After the experiments:
■ Simplifying downloading of all data collected

13. THANKS!!
Baltasar Fernández-Manjón @baltaFM
balta@fdi.ucm.es