Applying Learning Analytics 
in Serious Games 
Baltasar Fernandez-Manjon 
e-UCM research group, www.e-ucm.es 
balta@fdi.uc...
About me and context 
• CS Professor at Complutense University 
• Director of e-UCM 
• e-UCM research group about 
Learnin...
Educational games in the classroom: game 
• The Radix Endeavor 
• https://www.radixendeavor.org 
• MMOG with Science, 
Tec...
Educational games in the classroom: settings 
• Master students in the e-learning 
class 
• 13 students, 8 nationalities 
...
Educational games in the classroom: outcome 
• Students were really engaged for the whole hour 
• They enjoyed the experie...
Learning Analytics 
• Improving education based on 
data analysis 
• Data driven 
• Evidence-based education 
• Educationa...
Learning analytics steps 
• 1. Collecting large amounts of data from a number of channels 
• Interaction with the system 
...
The 4 Levels of Learning Analytics 
• Descriptive 
• What has happened? 
• Diagnostic 
• Why did it happen? 
• Predictive ...
Game Analytics: a parallel world 
• Application of analytics to game 
development and research 
• Telemetry 
• Data obtain...
Uses of Learning Analytics in educational games 
• Game testing – game analytics 
• It is the game realiable? 
• How many ...
The Feedback loop 
• Not only important the feedback but also when is provided 
• Learning analytics can help in improving...
Use of the educational games: Black box model 
• Games as independent pieces of 
content 
• No information about what is 
...
Designing for Learning Analytics in Games 
• Educational games need to be designed for Learning Analytics 
• Games make us...
Use of educational/serious games at the school 
Takeuchi, L. M., & Vaala, S. (2014). Level up learning: A national survey ...
Use of games at the school
Our work in Serious/Educational 
Games
eAdventure game platform 
Open code authoring environment for the production of point-and-click 
adventure games & immersi...
http://first-aid-game.e-ucm.es With MGH-Harvard University 
http://www.chermug.eu 
With ONT, educ@ONT
eAdventure + Learning Analytics 
Communication 
API 
Game Engine 
{ 
type: 'input', 
timeStamp: some_timestamp, 
device: '...
GLEANER 
• GLEANER: Game Learning Analytics for education research 
• Open code framework to capture game traces 
20 
Refe...
GLEANER Architecture
GLEANER Architecture - Technologies
GLEANER Analysis 
• Reporter has access to the database, and presents its data through reports 
• graphics, heat-maps, rel...
ADL eXperience API (xAPI) 
‣ Result of Project Tin Can 
‣ Tracks experiences, informal learning, real-world 
experiences (...
25 
Activity Streams 
• http://activitystrea.ms 
• Collaboration between Google, Facebook, Microsoft and others 
• Allows ...
Reporting 
Systems 
Assessme 
nt Services 
Semantic 
Analysis 
Statistical 
Services 
xAPI Learning Record Store (LRS) 
26...
GLEANER traces and xAPI
Example: A game to learn XML 
http://gleaner.e-ucm.es/lostinspace/play/index.html
About the game... 
• A basic platform game to acquire familiarity with XML documents 
• Learn the syntax 
• Understand nes...
Understanding the game… 
Main character 
Goal 
Write XML snippets here 
to move the main 
character to the goal 
Power-ups...
Educational goals 
• Basic understanding of XML Syntax 
• Markup syntax 
• Basic documents 
• Attributes 
• Complex docume...
LA perspective 
• What does GLEANER trace? 
• Higher level events 
• Generic traces & game specific traces 
• Only those r...
Main dashboard (GLEANER Report) 
33
Progress report 
34 
User 
Session 
Phases 
completed 
Current 
score 
Learning achievements 
Achievements are 
updated an...
Detailed traces 
35 
Filters XML snipets 
The fragments are updated in real time as submitted by 
the student. The instruc...
LA simplify SG deployment in the classroom 
• Experiment: Gamified experience in the classroom 
Goals 
• First to complete...
Experimental design and Learning Analytics 
• Learning Analytics should be in-line with the experimental design 
• Annonym...
La Dama Boba : Educational game for promoting 
students interest in classical theater 
Based on The Foolish Lady by Lope d...
Experiment game vs class 
• Teacher with an improved 
standard content 
• 757 students 
• From 8 middle and high schools 
...
Formal evaluation pre-post 
• Off-line learning analytics 
• Formal evaluation of games is 
very complex and expensive 
• ...
learning analytics outcomes: scores, time, ...
How the user play the game? 
42
Open issues in learning analytics in games 
• How to distinguish automatically game exploration from conceptual 
errors? 
...
Other important issues in LA 
• Ethical and legal aspects 
• Security model 
• Ownership of information 
• Informing the u...
Conclusions 
• LA in Serious Games has a great potential from the application and research 
perspective 
• Ease the deplym...
Thank you! 
46 
balta@fdi.ucm.es 
@BaltaFM
Our current projects 
RAGE 
H2020 ICT2014-21ª 
Realising an Applied Gaming Eco-system
Upcoming SlideShare
Loading in …5
×

Applying learning analytics in serious games

2,699 views

Published on

Learning Analytics and how to use in educational or serious games for improving the use of the games
game traces
evidence based education
Talk at the Ecole Normal Superior, Lyon, France

Published in: Education
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,699
On SlideShare
0
From Embeds
0
Number of Embeds
359
Actions
Shares
0
Downloads
91
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide
  • 1. Collecting large amounts of data from a number of channels – including, but not limited to, online learning environments, social, mobile – and perhaps in the future, games. Couple this data with various learning theories and we can begin to form a more holistic picture of a student’s learning progress than just theories.
    2. Translating that data into actionable insights. It may be impossible to track how much a student really absorbed from one lesson but the system CAN track his/her behaviour and use that as a signal. Here are a few examples of behavioural signals:
    - Language of frustration in any media. - Low time on site, relative to the class. - Long lag between logins. - Tracking areas of studies in which the student is weak in over years. - Detecting the TYPE of mistakes that was made – careless or a fundamental lack of understanding? - Theoretically, learning analytics would even be able to track whether or not a student is guessing in a multiple choice test.
    3. Personalization and adaptation. Once the system gets the signal, it can then personalize each student’s learning environment. For example, if a student spends significantly less time attempting to solve a problem compared to other students, the system can display prompts and clues to keep him/her going – in real time. This is crucial because when a student gets feedback is just as crucial to learning as what feedback the students get. This wasn’t possible in the past, where students have to wait at least a few days for their assignments to be marked.
    4. Predicting the best course in the future. As students use the system for a prolonged period of time, educators will be able to track what works and what doesn’t – and adjust accordingly. In fact, it will soon be possible for each student to essentially be working with a custom-built and personalized curriculum that’s unique to them.
  • Descriptive: What has happened? Look at facts, figures, and any other data you have that give you a detailed picture. Did a student fail a math quiz? What concepts were mastered and what ones weren’t?
    Diagnostic: Why did it happen? Examining the descriptive elements allows you to critically assess why an outcome happened? The student did ok on geometry questions but bombed the algebra-based material? Was less class time spent on the algebra stuff? Were different types (or amounts) of homework given? Look for explanations
    Predictive: What will happen? This is where you look forward: What would the outcome be based on different elements. Think of it as a choose your own adventure – will the student learn the algebra based material better if X, Y, or Z happened?
    Prescriptive: What should I do How can a specific outcome be achieved through the use of specific elements? Take what you’ve learned through 1, 2, and 3 and apply it in an effort to achieve the learning outcome you’re looking for!
  • Educational game, to get teenagers interested in Spanish classical theatre.
  • Applying learning analytics in serious games

    1. 1. Applying Learning Analytics in Serious Games Baltasar Fernandez-Manjon e-UCM research group, www.e-ucm.es balta@fdi.ucm.es @BaltaFM École Normale Supérieure de Lyon, Novembre 2014 http://www.slideshare.net/BaltasarFernandezManjon/
    2. 2. About me and context • CS Professor at Complutense University • Director of e-UCM • e-UCM research group about Learning technologies • 15 researchers • Serious games • Development of technology • eAdventure, GLEANER • Application to the medical domain • European projects • H2020 – RAGE (call 21) • SEGAN • www.e-ucm.es RAGE -H2020 2
    3. 3. Educational games in the classroom: game • The Radix Endeavor • https://www.radixendeavor.org • MMOG with Science, Technology, Engeneering and Math (STEM) topics • Many different quests • Adapted to middle and high school curriculum • Facilities for teachers • Class configuration • Quest assignation • Now being tested in actual settings
    4. 4. Educational games in the classroom: settings • Master students in the e-learning class • 13 students, 8 nationalities • No previous info about the game • If stuck ask for help or watch the help videos • One hour free play • Who perform “better” at 30 minutes and at the end • Publish screen captures in the class Moodle course forum
    5. 5. Educational games in the classroom: outcome • Students were really engaged for the whole hour • They enjoyed the experience • Consider the game a commercial quality game • No one asked for help • All of them published the screenshots in the Moodle class forum • Educational outcome • Only 4 of the students complete more than 2 quests • Most of the students just spend the time exploring the environment but not doing anything “useful” or learning anything • Actual data can be contradictory with simple observation
    6. 6. Learning Analytics • Improving education based on data analysis • Data driven • Evidence-based education • Educational theories/approaches can be contrasted with actual data • Related with … • Educational data mining • Business intelligence • Visual analytics www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf 6
    7. 7. Learning analytics steps • 1. Collecting large amounts of data from a number of channels • Interaction with the system • 2. Translating that data into information or actionable insights. • It may be impossible to track how much a student really absorbed from one lesson but the system CAN track his/her behaviour and use that as a signal • 3. Use of the information for different purposes • Personalization and adaptation. Once the system gets the signal, it can then personalize each student’s learning environment. • Assessment. Use that information for formative or summative assessment • Predicting the best course in the future. As students use the system for a prolonged period of time, educators will be able to track what works and what doesn’t – and adjust accordingly. http://www.edudemic.com/grades-2-0-how-learning-analytics-are-changing-the-teachers-role/
    8. 8. The 4 Levels of Learning Analytics • Descriptive • What has happened? • Diagnostic • Why did it happen? • Predictive • What will happen? • Prescriptive • What should I do? http://www.edudemic.com/4-levels-learning-analytics-graphic/
    9. 9. Game Analytics: a parallel world • Application of analytics to game development and research • Telemetry • Data obtained over distance • Mobile games, MMOG • Game metrics • Interpretable measures of data related to games • Player behavior • User metrics • Generics metrics • Genre specific metrics • Game specific metrics 9
    10. 10. Uses of Learning Analytics in educational games • Game testing – game analytics • It is the game realiable? • How many students finish the game? • Average time to complete the game? • Game evaluation • From pre-post test to learning analytics evaluation • Game deployment in the class • Real-time information for supporting the teacher • “Stealth” student evaluation • Knowing what is happening when the game is deployed in the class
    11. 11. The Feedback loop • Not only important the feedback but also when is provided • Learning analytics can help in improving feedback http://www.wired.com/2011/06/ff_feedbackloop/ http://safety.fhwa.dot.gov/speedmgt/ref_mats/fhwasa12004/
    12. 12. Use of the educational games: Black box model • Games as independent pieces of content • No information about what is happening during the in-game play • Or very simple • Completed or not completed • Time used 12 del Blanco et al (2013). Using e-Learning standards in educational video games. Computer Standards & Interfaces 36 (1) pp. 178–187
    13. 13. Designing for Learning Analytics in Games • Educational games need to be designed for Learning Analytics • Games make use of in game mechanisms for the assessment of player performance and progress • Interrelate observable in game behaviors to a competency based model related with learning outcomes rather than game performance • What are the relevant educational situations that want to be identified? • Do we have the way to communicate the game state? • Varibles, flags, identifiers Implications of Learning Analytics for Serious Game Design. Jannicke Baalsrud Hauge, Riccardo Berta, Giusy Fiucci, Baltasar Fernandez Manjon, Carmen Padron-Napoles, Wim Westera and Rob Nadolski. IEEE ICALT 2014
    14. 14. Use of educational/serious games at the school Takeuchi, L. M., & Vaala, S. (2014). Level up learning: A national survey on teaching with digital games. New York: The Joan Ganz Cooney Center at Sesame Workshop
    15. 15. Use of games at the school
    16. 16. Our work in Serious/Educational Games
    17. 17. eAdventure game platform Open code authoring environment for the production of point-and-click adventure games & immersive learning simulations Easy to include Learning Analytics in eAdventure games http://sourceforge.net/projects/e-adventure/
    18. 18. http://first-aid-game.e-ucm.es With MGH-Harvard University http://www.chermug.eu With ONT, educ@ONT
    19. 19. eAdventure + Learning Analytics Communication API Game Engine { type: 'input', timeStamp: some_timestamp, device: 'some_device', action: 'some_action', target: 'target_id', data: { key1: value, ...} } { type: 'logic', timeStamp: some_timestamp, event: 'some_event', target: 'some_id', data: { key1: value, ...} } Learning Analytics Database
    20. 20. GLEANER • GLEANER: Game Learning Analytics for education research • Open code framework to capture game traces 20 Reference model in the EU NoE GALA, http://e-ucm.github.io/gleaner/ Ángel Serrano-Laguna, Javier Torrente, Pablo Moreno-Ger, Baltasar Fernández-Manjón (2014): Application of Learning Analytics in Educational Videogames. Entertainment Computing, Elsevier (in press, early access available).
    21. 21. GLEANER Architecture
    22. 22. GLEANER Architecture - Technologies
    23. 23. GLEANER Analysis • Reporter has access to the database, and presents its data through reports • graphics, heat-maps, relational tables… • Evaluator has access to the database and checks the educational defined goals in the assessment model
    24. 24. ADL eXperience API (xAPI) ‣ Result of Project Tin Can ‣ Tracks experiences, informal learning, real-world experiences (not just completions) ‣ Allows data storage AND retrieval (ex. 3rd party reporting and analytics tools) ‣ Enables tracking mobile, games, ITS, and virtual worlds experiences ‣ Developed by open source community 24 From Damon Regan (ADL) presentation at SINTICE2013
    25. 25. 25 Activity Streams • http://activitystrea.ms • Collaboration between Google, Facebook, Microsoft and others • Allows reporting of experiences, not just completions • Format: <Actor> <Verb> <Object> (I did this): Simple Statement: I (actor) watched (verb) a video on protecting employee data (object) Complex: in the context of [information assurance certification training] with result [timestamp:2013-0618T18:30:32.360Z ]. From Damon Regan (ADL) presentation at SINTICE2013
    26. 26. Reporting Systems Assessme nt Services Semantic Analysis Statistical Services xAPI Learning Record Store (LRS) 26 From Damon Regan (ADL) presentation at SINTICE2013
    27. 27. GLEANER traces and xAPI
    28. 28. Example: A game to learn XML http://gleaner.e-ucm.es/lostinspace/play/index.html
    29. 29. About the game... • A basic platform game to acquire familiarity with XML documents • Learn the syntax • Understand nesting and attributes • Gain agility writing and reading XML documents • Designed as a complementary activity in a Web Programming Course • For undergraduate computer science students • Not developed with eAdventure
    30. 30. Understanding the game… Main character Goal Write XML snippets here to move the main character to the goal Power-ups (new syntax elements)
    31. 31. Educational goals • Basic understanding of XML Syntax • Markup syntax • Basic documents • Attributes • Complex documents • Basic programing skills • Sequencing • Loops
    32. 32. LA perspective • What does GLEANER trace? • Higher level events • Generic traces & game specific traces • Only those relevant for our learning objectives • The aggregator will filter and transmit: • Level completion events • Each XML fragment submitted by the player
    33. 33. Main dashboard (GLEANER Report) 33
    34. 34. Progress report 34 User Session Phases completed Current score Learning achievements Achievements are updated and highlighted in real time
    35. 35. Detailed traces 35 Filters XML snipets The fragments are updated in real time as submitted by the student. The instructor can filter to see only mistakes, or all the fragments submitted by a specific students.
    36. 36. LA simplify SG deployment in the classroom • Experiment: Gamified experience in the classroom Goals • First to complete the game • Maximum points in the game but • Initially students were not informed that LA was in place After some playing time LA data was showed to the students ‣ Instructor knows what is happening in the class ‣ Behavior change when students know that LA was in place • More oriented to success and less exploration or “playing” Ángel Serrano-Laguna, Baltasar Fernández-Manjón (2014): Applying learning analytics to simplify serious games deployment in the classroom. Proceedings of the 2014 IEEE Global Engineering Education Conference (EDUCON) Pages 872-877.
    37. 37. Experimental design and Learning Analytics • Learning Analytics should be in-line with the experimental design • Annonymization • Use of the data • Specific regulations in several domains • Medical: ethics committees
    38. 38. La Dama Boba : Educational game for promoting students interest in classical theater Based on The Foolish Lady by Lope de Vega The game is available at http://damaboba.e-ucm.es/ (in Spanish) Also in in Android Market
    39. 39. Experiment game vs class • Teacher with an improved standard content • 757 students • From 8 middle and high schools in Madrid region • Control group and experimental group
    40. 40. Formal evaluation pre-post • Off-line learning analytics • Formal evaluation of games is very complex and expensive • Pre-test • Post-test • Very few games have been formally probed to be effective • Similar results with Learning Analytics than with pre-post test? 40
    41. 41. learning analytics outcomes: scores, time, ...
    42. 42. How the user play the game? 42
    43. 43. Open issues in learning analytics in games • How to distinguish automatically game exploration from conceptual errors? • Traces across different types of games • Long term learning across games
    44. 44. Other important issues in LA • Ethical and legal aspects • Security model • Ownership of information • Informing the user • Anonymization of information • Aspects specially relevant if you are working with kids or in the medical domain! 44
    45. 45. Conclusions • LA in Serious Games has a great potential from the application and research perspective • Ease the deplyment of game in the classroom • Simplify game evaluation with users • LA in Serious Games should benefits from Games Analytics experience and work • Still complex to implement LA in SG • Increases the (already high) cost of the games • Frameworks and new standards specifications could greatly simplify LA implementation and adoption 45
    46. 46. Thank you! 46 balta@fdi.ucm.es @BaltaFM
    47. 47. Our current projects RAGE H2020 ICT2014-21ª Realising an Applied Gaming Eco-system

    ×