SlideShare a Scribd company logo
Bootstrapping serious games to assess
learning through analytics
Freire-Morán, Manuel; Calvo-Morata, Antonio; Martinez-Ortiz, Ivan;
Fernández-Manjón, Baltasar
balta@fdi.ucm.es
Dept. of Software Engineering and Artificial Intelligence
Complutense University of Madrid
EDUCON 2023 | May 1-4 2023 | Salmiya, Kuwait
Serious games for assessment
Serious Games are effective at
teaching, creating awareness, changing behaviours, …
What obstacles remain to make them more useful for assessment?
Development
high costs
must design for both
fun & learning
Testing & Analytics
waterfall testing
much worse than
agile
require infrastructure
Deployment
much harder to
produce than other
LMS activities
must communicate
SG, analytics & LMS
Assessment
intrusive exams… or
validated games,
analytics, & Machine
Learning
● Validating a game:
○ ensures effectiveness at stated goal
○ typically via pre-post experiments
○ provides training data for assessment
○ can also collect player interactions, so that
● ML trained on validation data can
predict learning from interactions
● Predicted learning without exam =
stealth assessment
○ non-intrusive
○ does not break player immersion
Serious games to assess learning
Improving evidence-based assessment of players using serious games Telematics and Informatics
Volume 60, July 2021, 101583 https://doi.org/10.1016/j.tele.2021.101583
1. validation via
pre-post tests
2. interaction data + target
Once model is trained:
from interactions to
predictions, can be used
for assessment
3. train ML model to predict
target from interactions
proving evidence-based assessment of players using serious games Telematics and Informatics
Volume 60, July 2021, 101583 https://doi.org/10.1016/j.tele.2021.101583
Deploying assessment-friendly SGs
● Analytics require player ids
○ interactions from a given player should
be stored & analyzed together
○ not necessary to know real player
identity, as long as id is consistent
● Simplest possible analytics-enabled
deployment = a link from LMS
○ link must support launch +
communication to & from LMS
○ a mediator tool solves many
deployment issues, can be integrated
with server-side analytics
Using e-Learning Standards to Improve Serious Game Deployment and Evaluation EDUCON
2022, 10.1109/EDUCON52537.2022.9766573
Using e-Learning Standards to Improve Serious Game Deployment and
Evaluation EDUCON 2022, 10.1109/EDUCON52537.2022.9766573
What the teacher sees
1. generate LTI link
from authoring tool
2. paste into LTI widget
in LMS course
3. see how students
are playing / have
played
What really happens
see illustration right
What students see
1. click link
2. play game
Democratizing Game Learning Analytics for Serious Games GALA 2021
https://doi.org/10.1007/978-3-030-92182-8_16
● Analytics as part of design
○ incremental improvement is better than
waterfall models, but improvement is
hard to track without analytics!
○ requires developers/designers to choose
in-game interactions to report, and
tracker component within each game
to report those interactions
○ requires tools to analyze results
● Analytics infrastructure
○ server-side component to validate,
store & retrieve interactions
○ analytics dashboards provide the
insights into collected interactions
○ infrastructure also for deployment,
validation, and finally assessment
Testing and improving an assessment-friendly SG
e-UCM Game Learning Analytics
components, based on xAPI
e-UCM xAPI tracker
● C#, JS and Unity.
● High-level interface for devs.
● Compatible with xAPI for SGs.
SIMVA validation tool
● Orchestrates LA activities and LTI.
● Manages users, groups (A/B), pre-post,
analysis and trace storage.
● Unity plugin for easy setup with OAuth2.0.
TxMon analysis tool
● Analyzes xAPI traces.
● Default visualizations.
Developing assessment-friendly games
● Game development is expensive
○ Technical expertise to code
○ Art assets may require talented artists
○ Game design (for the fun) + domain
experience (for the education)
● Assessment requires data
○ from interactions
(or it will not be stealth)
○ collected via an analytics
infrastructure
uAdventure, our authoring environment built
on Unity, integrates analytics
uAdventure authoring tool
● Narrative and geolocated
gameplay styles
● Easy to use by non-experts
● GLA out-of-the-box
● Exports native and WebGL
● Includes e-UCM Tracker
Overview: obstacles and tools
Next steps
● Serious Games can be a powerful method for student assessment in
complex issues (i.e. knowledge, awareness) and it is open to include AI
techniques
● Still complex but open software and standards could help in the process
● Assessment not yet fully integrated into Simva
● Building new module to
○ train a simple ML models from validation data
○ use trained models to provide predictions based on gameplay interactions
○ upload results as activity scores via LTI, making them available in host LMS
● Would close SG lifecycle, from authoring to evaluation
● All the technical solutions should take into account the ethical aspects
and comply with the privacy regulations (e.g. EU GDPR)
Thanks!
Contact: balta@fdi.ucm.es
This work has been partially funded by the Regional Government of Madrid (eMadrid S2018/TCS-4307, co-funded by the
European Structural Funds FSE and FEDER), by the Ministry of Education (PID2020-119620RB-I00) and by the
Telefónica-Complutense Chair on Digital Education and Serious Games.
https://github.com/e-ucm/uAdventure
https://github.com/e-ucm/simva
Our publications: https://www.e-ucm.es/publications/all

More Related Content

Similar to Bootstrapping serious games to assess learning through analytics - Baltasar Fernández-Manjón

Gaming Learning Analytics Contributing to the serious games ecosystem
Gaming Learning Analytics Contributing to the serious games ecosystemGaming Learning Analytics Contributing to the serious games ecosystem
Gaming Learning Analytics Contributing to the serious games ecosystem
Baltasar Fernández-Manjón
 
NetGPT 1.0 PRD Presentation.pdf
NetGPT 1.0 PRD Presentation.pdfNetGPT 1.0 PRD Presentation.pdf
NetGPT 1.0 PRD Presentation.pdf
JOHNADEMILUYI3
 
Learning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesLearning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive Disabilities
Baltasar Fernández-Manjón
 
Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...
Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...
Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...
Baltasar Fernández-Manjón
 
NetGPT 1.0 PRD Presentation2.pdf
NetGPT 1.0 PRD Presentation2.pdfNetGPT 1.0 PRD Presentation2.pdf
NetGPT 1.0 PRD Presentation2.pdf
JOHNADEMILUYI3
 
uAdventure simplifying narrative serious games development - icalt 2019 (1)
uAdventure  simplifying narrative serious games development - icalt 2019 (1)uAdventure  simplifying narrative serious games development - icalt 2019 (1)
uAdventure simplifying narrative serious games development - icalt 2019 (1)
Baltasar Fernández-Manjón
 
Applying learning analytics in serious games
Applying learning analytics in serious games Applying learning analytics in serious games
Applying learning analytics in serious games
Baltasar Fernández-Manjón
 
IRJET- Clustering Students through Data Mining and Gamified Learning
IRJET- Clustering Students through Data Mining and Gamified LearningIRJET- Clustering Students through Data Mining and Gamified Learning
IRJET- Clustering Students through Data Mining and Gamified Learning
IRJET Journal
 
Gala Conference 2018 Presentation
Gala Conference 2018 PresentationGala Conference 2018 Presentation
Gala Conference 2018 Presentation
Cristina Alonso
 
11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...
11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...
11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...
eMadrid network
 
WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...
WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...
WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...
eMadrid network
 
Icce21 systematizing game learning analytics for improving serious games life...
Icce21 systematizing game learning analytics for improving serious games life...Icce21 systematizing game learning analytics for improving serious games life...
Icce21 systematizing game learning analytics for improving serious games life...
Baltasar Fernández-Manjón
 
Simplifying Serious Games Authoring and Validation with uAdventure and SIMVA
Simplifying Serious Games Authoring and Validation with uAdventure and SIMVASimplifying Serious Games Authoring and Validation with uAdventure and SIMVA
Simplifying Serious Games Authoring and Validation with uAdventure and SIMVA
Baltasar Fernández-Manjón
 
DSDT Meetup April 2021
DSDT Meetup April 2021DSDT Meetup April 2021
DSDT Meetup April 2021
DSDT_MTL
 
Integrating learning analytics into a game authoring tool
Integrating learning analytics into a game authoring toolIntegrating learning analytics into a game authoring tool
Integrating learning analytics into a game authoring tool
Baltasar Fernández-Manjón
 
Learning analytics for improving educational games jcsg2017
Learning analytics for improving educational games jcsg2017Learning analytics for improving educational games jcsg2017
Learning analytics for improving educational games jcsg2017
Baltasar Fernández-Manjón
 
Game development software engineering: digital educational game promoting al...
Game development software engineering: digital educational  game promoting al...Game development software engineering: digital educational  game promoting al...
Game development software engineering: digital educational game promoting al...
IJECEIAES
 
Tcea 2014 Video Game Design for New TEKS
Tcea 2014   Video Game Design for New TEKSTcea 2014   Video Game Design for New TEKS
Tcea 2014 Video Game Design for New TEKS
Mike Ploor
 
Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...
Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...
Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...
Ana Rus Cano Moreno
 
Rsqrd AI: From R&D to ROI of AI
Rsqrd AI: From R&D to ROI of AIRsqrd AI: From R&D to ROI of AI
Rsqrd AI: From R&D to ROI of AI
Sanjana Chowdhury
 

Similar to Bootstrapping serious games to assess learning through analytics - Baltasar Fernández-Manjón (20)

Gaming Learning Analytics Contributing to the serious games ecosystem
Gaming Learning Analytics Contributing to the serious games ecosystemGaming Learning Analytics Contributing to the serious games ecosystem
Gaming Learning Analytics Contributing to the serious games ecosystem
 
NetGPT 1.0 PRD Presentation.pdf
NetGPT 1.0 PRD Presentation.pdfNetGPT 1.0 PRD Presentation.pdf
NetGPT 1.0 PRD Presentation.pdf
 
Learning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive DisabilitiesLearning Analytics Serious Games Cognitive Disabilities
Learning Analytics Serious Games Cognitive Disabilities
 
Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...
Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...
Using Game Learning Analytics to Improve the Design, Evaluation and Deploymen...
 
NetGPT 1.0 PRD Presentation2.pdf
NetGPT 1.0 PRD Presentation2.pdfNetGPT 1.0 PRD Presentation2.pdf
NetGPT 1.0 PRD Presentation2.pdf
 
uAdventure simplifying narrative serious games development - icalt 2019 (1)
uAdventure  simplifying narrative serious games development - icalt 2019 (1)uAdventure  simplifying narrative serious games development - icalt 2019 (1)
uAdventure simplifying narrative serious games development - icalt 2019 (1)
 
Applying learning analytics in serious games
Applying learning analytics in serious games Applying learning analytics in serious games
Applying learning analytics in serious games
 
IRJET- Clustering Students through Data Mining and Gamified Learning
IRJET- Clustering Students through Data Mining and Gamified LearningIRJET- Clustering Students through Data Mining and Gamified Learning
IRJET- Clustering Students through Data Mining and Gamified Learning
 
Gala Conference 2018 Presentation
Gala Conference 2018 PresentationGala Conference 2018 Presentation
Gala Conference 2018 Presentation
 
11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...
11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...
11_04_2019 EDUCON eMadrid special session on "Game learning analytics for edu...
 
WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...
WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...
WEEF/GEDC eMadrid_Systematizing Game Learning Analytics for Improving Serious...
 
Icce21 systematizing game learning analytics for improving serious games life...
Icce21 systematizing game learning analytics for improving serious games life...Icce21 systematizing game learning analytics for improving serious games life...
Icce21 systematizing game learning analytics for improving serious games life...
 
Simplifying Serious Games Authoring and Validation with uAdventure and SIMVA
Simplifying Serious Games Authoring and Validation with uAdventure and SIMVASimplifying Serious Games Authoring and Validation with uAdventure and SIMVA
Simplifying Serious Games Authoring and Validation with uAdventure and SIMVA
 
DSDT Meetup April 2021
DSDT Meetup April 2021DSDT Meetup April 2021
DSDT Meetup April 2021
 
Integrating learning analytics into a game authoring tool
Integrating learning analytics into a game authoring toolIntegrating learning analytics into a game authoring tool
Integrating learning analytics into a game authoring tool
 
Learning analytics for improving educational games jcsg2017
Learning analytics for improving educational games jcsg2017Learning analytics for improving educational games jcsg2017
Learning analytics for improving educational games jcsg2017
 
Game development software engineering: digital educational game promoting al...
Game development software engineering: digital educational  game promoting al...Game development software engineering: digital educational  game promoting al...
Game development software engineering: digital educational game promoting al...
 
Tcea 2014 Video Game Design for New TEKS
Tcea 2014   Video Game Design for New TEKSTcea 2014   Video Game Design for New TEKS
Tcea 2014 Video Game Design for New TEKS
 
Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...
Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...
Downtown, A Subway Adventure: Using Learning Analytics to Improve the Develop...
 
Rsqrd AI: From R&D to ROI of AI
Rsqrd AI: From R&D to ROI of AIRsqrd AI: From R&D to ROI of AI
Rsqrd AI: From R&D to ROI of AI
 

More from eMadrid network

Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo TovarRecognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
eMadrid network
 
A study about the impact of rewards on student's engagement with the flipped ...
A study about the impact of rewards on student's engagement with the flipped ...A study about the impact of rewards on student's engagement with the flipped ...
A study about the impact of rewards on student's engagement with the flipped ...
eMadrid network
 
Assessment and recognition in technical massive open on-line courses with and...
Assessment and recognition in technical massive open on-line courses with and...Assessment and recognition in technical massive open on-line courses with and...
Assessment and recognition in technical massive open on-line courses with and...
eMadrid network
 
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
eMadrid network
 
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth CobosMeta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
eMadrid network
 
Best paper Award - Miguel Castro
Best paper Award - Miguel CastroBest paper Award - Miguel Castro
Best paper Award - Miguel Castro
eMadrid network
 
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdfSeminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
eMadrid network
 
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid network
 
Open_principles_and_co-creation_for_digital_competences_for_students.pdf
Open_principles_and_co-creation_for_digital_competences_for_students.pdfOpen_principles_and_co-creation_for_digital_competences_for_students.pdf
Open_principles_and_co-creation_for_digital_competences_for_students.pdf
eMadrid network
 
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
eMadrid network
 
eMadrid_KatjaAssaf_DigiCred.pdf
eMadrid_KatjaAssaf_DigiCred.pdfeMadrid_KatjaAssaf_DigiCred.pdf
eMadrid_KatjaAssaf_DigiCred.pdf
eMadrid network
 
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdfPresentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
eMadrid network
 
EDC-eMadrid_20230113 Ildikó Mázár.pdf
EDC-eMadrid_20230113 Ildikó Mázár.pdfEDC-eMadrid_20230113 Ildikó Mázár.pdf
EDC-eMadrid_20230113 Ildikó Mázár.pdf
eMadrid network
 
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
eMadrid network
 
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
eMadrid network
 
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
eMadrid network
 
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
eMadrid network
 
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
eMadrid network
 
2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...
2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...
2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...
eMadrid network
 
2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...
2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...
2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...
eMadrid network
 

More from eMadrid network (20)

Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo TovarRecognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
Recognizing Lifelong Learning Competences: A Report of Two Cases - Edmundo Tovar
 
A study about the impact of rewards on student's engagement with the flipped ...
A study about the impact of rewards on student's engagement with the flipped ...A study about the impact of rewards on student's engagement with the flipped ...
A study about the impact of rewards on student's engagement with the flipped ...
 
Assessment and recognition in technical massive open on-line courses with and...
Assessment and recognition in technical massive open on-line courses with and...Assessment and recognition in technical massive open on-line courses with and...
Assessment and recognition in technical massive open on-line courses with and...
 
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
Recognition of learning: Status, experiences and challenges - Carlos Delgado ...
 
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth CobosMeta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
 
Best paper Award - Miguel Castro
Best paper Award - Miguel CastroBest paper Award - Miguel Castro
Best paper Award - Miguel Castro
 
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdfSeminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
Seminario eMadrid_Curso MOOC_Antonio de Nebrija_Apología del saber.pptx.pdf
 
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
eMadrid-Opportunities and Design Challenges in the Gaming4Coding Project_Pete...
 
Open_principles_and_co-creation_for_digital_competences_for_students.pdf
Open_principles_and_co-creation_for_digital_competences_for_students.pdfOpen_principles_and_co-creation_for_digital_competences_for_students.pdf
Open_principles_and_co-creation_for_digital_competences_for_students.pdf
 
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
Competencias_digitales_del_profesorado_universitario_para_la_educación_abiert...
 
eMadrid_KatjaAssaf_DigiCred.pdf
eMadrid_KatjaAssaf_DigiCred.pdfeMadrid_KatjaAssaf_DigiCred.pdf
eMadrid_KatjaAssaf_DigiCred.pdf
 
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdfPresentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
Presentazione E-Madrid - 12-01-2023 Ruth Kerr.pdf
 
EDC-eMadrid_20230113 Ildikó Mázár.pdf
EDC-eMadrid_20230113 Ildikó Mázár.pdfEDC-eMadrid_20230113 Ildikó Mázár.pdf
EDC-eMadrid_20230113 Ildikó Mázár.pdf
 
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
2022_12_16 «“La informática en la educación escolar en Europa”, informe Euryd...
 
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
 
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
2022_12_16 «Efecto del uso de lenguajes basados en bloques en el aprendizaje ...
 
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
2022_11_11 «AI and ML methods for Multimodal Learning Analytics»
 
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
2022_11_11 «The promise and challenges of Multimodal Learning Analytics»
 
2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...
2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...
2022_11_11 «Biometrics and Behavior Understanding Technologies for e-Learning...
 
2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...
2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...
2022_10_07 «Desarrollando el Pensamiento Computacional a través de los juegos...
 

Recently uploaded

A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
Jean Carlos Nunes Paixão
 
DRUGS AND ITS classification slide share
DRUGS AND ITS classification slide shareDRUGS AND ITS classification slide share
DRUGS AND ITS classification slide share
taiba qazi
 
Digital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments UnitDigital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments Unit
chanes7
 
Main Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docxMain Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docx
adhitya5119
 
Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...
Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...
Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...
National Information Standards Organization (NISO)
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
Jean Carlos Nunes Paixão
 
Smart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICTSmart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICT
simonomuemu
 
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
The simplified electron and muon model, Oscillating Spacetime: The Foundation...The simplified electron and muon model, Oscillating Spacetime: The Foundation...
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
RitikBhardwaj56
 
clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
Priyankaranawat4
 
The basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptxThe basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptx
heathfieldcps1
 
CACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdfCACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdf
camakaiclarkmusic
 
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Dr. Vinod Kumar Kanvaria
 
Assessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptxAssessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptx
Kavitha Krishnan
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
Nicholas Montgomery
 
S1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptxS1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptx
tarandeep35
 
Pride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School DistrictPride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School District
David Douglas School District
 
A Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptxA Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptx
thanhdowork
 
The basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptxThe basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptx
heathfieldcps1
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
TechSoup
 
The Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collectionThe Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collection
Israel Genealogy Research Association
 

Recently uploaded (20)

A Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdfA Independência da América Espanhola LAPBOOK.pdf
A Independência da América Espanhola LAPBOOK.pdf
 
DRUGS AND ITS classification slide share
DRUGS AND ITS classification slide shareDRUGS AND ITS classification slide share
DRUGS AND ITS classification slide share
 
Digital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments UnitDigital Artifact 1 - 10VCD Environments Unit
Digital Artifact 1 - 10VCD Environments Unit
 
Main Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docxMain Java[All of the Base Concepts}.docx
Main Java[All of the Base Concepts}.docx
 
Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...
Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...
Pollock and Snow "DEIA in the Scholarly Landscape, Session One: Setting Expec...
 
Lapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdfLapbook sobre os Regimes Totalitários.pdf
Lapbook sobre os Regimes Totalitários.pdf
 
Smart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICTSmart-Money for SMC traders good time and ICT
Smart-Money for SMC traders good time and ICT
 
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
The simplified electron and muon model, Oscillating Spacetime: The Foundation...The simplified electron and muon model, Oscillating Spacetime: The Foundation...
The simplified electron and muon model, Oscillating Spacetime: The Foundation...
 
clinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdfclinical examination of hip joint (1).pdf
clinical examination of hip joint (1).pdf
 
The basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptxThe basics of sentences session 6pptx.pptx
The basics of sentences session 6pptx.pptx
 
CACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdfCACJapan - GROUP Presentation 1- Wk 4.pdf
CACJapan - GROUP Presentation 1- Wk 4.pdf
 
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
Exploiting Artificial Intelligence for Empowering Researchers and Faculty, In...
 
Assessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptxAssessment and Planning in Educational technology.pptx
Assessment and Planning in Educational technology.pptx
 
Film vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movieFilm vocab for eal 3 students: Australia the movie
Film vocab for eal 3 students: Australia the movie
 
S1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptxS1-Introduction-Biopesticides in ICM.pptx
S1-Introduction-Biopesticides in ICM.pptx
 
Pride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School DistrictPride Month Slides 2024 David Douglas School District
Pride Month Slides 2024 David Douglas School District
 
A Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptxA Survey of Techniques for Maximizing LLM Performance.pptx
A Survey of Techniques for Maximizing LLM Performance.pptx
 
The basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptxThe basics of sentences session 5pptx.pptx
The basics of sentences session 5pptx.pptx
 
Introduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp NetworkIntroduction to AI for Nonprofits with Tapp Network
Introduction to AI for Nonprofits with Tapp Network
 
The Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collectionThe Diamonds of 2023-2024 in the IGRA collection
The Diamonds of 2023-2024 in the IGRA collection
 

Bootstrapping serious games to assess learning through analytics - Baltasar Fernández-Manjón

  • 1. Bootstrapping serious games to assess learning through analytics Freire-Morán, Manuel; Calvo-Morata, Antonio; Martinez-Ortiz, Ivan; Fernández-Manjón, Baltasar balta@fdi.ucm.es Dept. of Software Engineering and Artificial Intelligence Complutense University of Madrid EDUCON 2023 | May 1-4 2023 | Salmiya, Kuwait
  • 2. Serious games for assessment Serious Games are effective at teaching, creating awareness, changing behaviours, … What obstacles remain to make them more useful for assessment? Development high costs must design for both fun & learning Testing & Analytics waterfall testing much worse than agile require infrastructure Deployment much harder to produce than other LMS activities must communicate SG, analytics & LMS Assessment intrusive exams… or validated games, analytics, & Machine Learning
  • 3. ● Validating a game: ○ ensures effectiveness at stated goal ○ typically via pre-post experiments ○ provides training data for assessment ○ can also collect player interactions, so that ● ML trained on validation data can predict learning from interactions ● Predicted learning without exam = stealth assessment ○ non-intrusive ○ does not break player immersion Serious games to assess learning Improving evidence-based assessment of players using serious games Telematics and Informatics Volume 60, July 2021, 101583 https://doi.org/10.1016/j.tele.2021.101583
  • 4. 1. validation via pre-post tests 2. interaction data + target Once model is trained: from interactions to predictions, can be used for assessment 3. train ML model to predict target from interactions proving evidence-based assessment of players using serious games Telematics and Informatics Volume 60, July 2021, 101583 https://doi.org/10.1016/j.tele.2021.101583
  • 5. Deploying assessment-friendly SGs ● Analytics require player ids ○ interactions from a given player should be stored & analyzed together ○ not necessary to know real player identity, as long as id is consistent ● Simplest possible analytics-enabled deployment = a link from LMS ○ link must support launch + communication to & from LMS ○ a mediator tool solves many deployment issues, can be integrated with server-side analytics Using e-Learning Standards to Improve Serious Game Deployment and Evaluation EDUCON 2022, 10.1109/EDUCON52537.2022.9766573
  • 6. Using e-Learning Standards to Improve Serious Game Deployment and Evaluation EDUCON 2022, 10.1109/EDUCON52537.2022.9766573 What the teacher sees 1. generate LTI link from authoring tool 2. paste into LTI widget in LMS course 3. see how students are playing / have played What really happens see illustration right What students see 1. click link 2. play game
  • 7. Democratizing Game Learning Analytics for Serious Games GALA 2021 https://doi.org/10.1007/978-3-030-92182-8_16
  • 8. ● Analytics as part of design ○ incremental improvement is better than waterfall models, but improvement is hard to track without analytics! ○ requires developers/designers to choose in-game interactions to report, and tracker component within each game to report those interactions ○ requires tools to analyze results ● Analytics infrastructure ○ server-side component to validate, store & retrieve interactions ○ analytics dashboards provide the insights into collected interactions ○ infrastructure also for deployment, validation, and finally assessment Testing and improving an assessment-friendly SG e-UCM Game Learning Analytics components, based on xAPI
  • 9. e-UCM xAPI tracker ● C#, JS and Unity. ● High-level interface for devs. ● Compatible with xAPI for SGs. SIMVA validation tool ● Orchestrates LA activities and LTI. ● Manages users, groups (A/B), pre-post, analysis and trace storage. ● Unity plugin for easy setup with OAuth2.0. TxMon analysis tool ● Analyzes xAPI traces. ● Default visualizations.
  • 10. Developing assessment-friendly games ● Game development is expensive ○ Technical expertise to code ○ Art assets may require talented artists ○ Game design (for the fun) + domain experience (for the education) ● Assessment requires data ○ from interactions (or it will not be stealth) ○ collected via an analytics infrastructure uAdventure, our authoring environment built on Unity, integrates analytics
  • 11. uAdventure authoring tool ● Narrative and geolocated gameplay styles ● Easy to use by non-experts ● GLA out-of-the-box ● Exports native and WebGL ● Includes e-UCM Tracker
  • 13. Next steps ● Serious Games can be a powerful method for student assessment in complex issues (i.e. knowledge, awareness) and it is open to include AI techniques ● Still complex but open software and standards could help in the process ● Assessment not yet fully integrated into Simva ● Building new module to ○ train a simple ML models from validation data ○ use trained models to provide predictions based on gameplay interactions ○ upload results as activity scores via LTI, making them available in host LMS ● Would close SG lifecycle, from authoring to evaluation ● All the technical solutions should take into account the ethical aspects and comply with the privacy regulations (e.g. EU GDPR)
  • 14. Thanks! Contact: balta@fdi.ucm.es This work has been partially funded by the Regional Government of Madrid (eMadrid S2018/TCS-4307, co-funded by the European Structural Funds FSE and FEDER), by the Ministry of Education (PID2020-119620RB-I00) and by the Telefónica-Complutense Chair on Digital Education and Serious Games. https://github.com/e-ucm/uAdventure https://github.com/e-ucm/simva Our publications: https://www.e-ucm.es/publications/all