Learning Analytics Applied to Serious Games
European Conference in Game-Based Learning (ECGBL
2013), Porto, Portugal
Balta...
About me and context
‣ CS Professor at Complutense U.
• Director of e-UCM
‣ e-UCM research group about
Learning technologi...
Learning analytics and SG?
The NMC Horizon Report:
2013 Higher Education
Edition
‣ new and emerging
technologies on
teachi...
Serious Games use?
‣ Serious Games have probed to be educationally
effective in several domains
• Medicine, military, busi...
Black box model
‣ Games as independent
pieces of content
‣ No information about
what is happening
during the in-game
play
...
Learning Analytics
‣ Improving education
based on data
analysis
• Data driven
‣ Evidence-based
education
‣ Related with …
...
Different uses
‣ Learning analytics
• Data analysis that helps students improve learning
outcomes.
‣ Academic/program anal...
When data is analyzed?
‣ Off-line
• Analyzing data after use
• Discovering patterns of use
• Allows to improve the experie...
Extensive Data
‣ Large number of participants
‣ Relatively limited number of variables
‣ Usually very little demographic i...
Intensive Data
‣ Relatively low number of participants
‣ Large number of observations for each variable
‣ Large number of ...
But there is a parallel world …
11
http://us.battle.net/wow/en/media/screenshots/races?keywords=&view#/goblins04
Game Analytics
‣ Application of
analytics to game
development and
research
‣ Telemetry
• Data obtained over
distance
• Mob...
Game metrics
‣ User metrics
• Customer
• Player
‣ Performance
metrics
• Technical
infrastructure
‣ Process metrics
• Devel...
Game metrics
14
Springer, Game Analytics Maximizing the Value of Player Data, pag 22
Game requirements for LA
‣ Most of games are black boxes.
• No access to what is going on during game play.
‣ We need acce...
SESSION
VARIABLES
PLAYER
GAME
VARIABLES
GAME
VARIABLES
TRACES
GOALS
From game data to educational information
PLAYER
VARIA...
Sessions
GAME PLAYER SESSIONS+ =
A player plays a game. Produces a game session.
One game session is a set of traces of ON...
VARIABLES
GENERATO
R
SESSION
Analyzing game sessions
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E...
Variable generator: User sessions
VAR
DEFINITION
VAR+ =
TRACE
TRACE
varName:‘beatgame‘,
operation: {
type:'trace_present‘
...
Goal Generators: User sessions
VAR
VAR
GOAL
DEFINITION
GOAL
RESULT=
usersession: {
beatgame: true,
score: 4747
}
id: ‘fina...
PLAYER * GAME
Analyzing game results
SESSION
SESSION
VAR VAR
VAR VAR
GOAL
RESULT
GOAL
RESULT
SESSION
SESSION
VAR VAR
VAR V...
eAdventure game platform
Open code authoring environment for the production of point-and-click
adventure games & immersive...
http://www.chermug.eu
With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es
With ONT, educ@ONT
eAdventure + Learning
Analytics
Game Engine
Communication
API
{
type: 'input',
timeStamp: some_timestamp,
device: 'some_de...
GLEANER
‣ GLEANER: Game Learning Analytics for education
research
• Open code framework to capture game traces
25
Referenc...
GLEANER Collector
‣ Implemented: Nodejs + Mongodb
‣ A web server ready to receive traces
‣ Cleans and sorts the data
GLEANER Analysis
‣ Reporter has access to the database, and presents
its data through reports
• graphics, heat-maps, relat...
Example: A game to learn XML
http://gleaner.e-ucm.es/lostinspace/play/index.html
About the game...
‣ A basic platform game to acquire familiarity with
XML documents
• Learn the syntax
• Understand nestin...
Understanding the game…
Goal
Main
character
Power-ups
(new syntax
elements)
Write XML
snippets here to
move the main
chara...
Educational goals
‣ Basic understanding of XML Syntax
• Markup syntax
• Basic documents
• Attributes
• Complex documents
‣...
LA perspective
‣ What does GLEANER trace?
• Higher level events
• Generic traces & game specific traces
• Only those relev...
Main dashboard (GLEANER Report)
33
Progress report
34
User
Session
Phases
completed
Current
score
Learning
achievements
Achievements are
updated and
highligh...
Detailed traces
35
Filters XML
snipets
The fragments are updated in real time
as submitted by the student. The
instructor ...
La Dama Boba : The Game
Based on The Foolish Lady by Lope de Vega
The game is available at http://damaboba.e-ucm.es/ (in S...
Experiment game vs class
‣ 757 students
‣ From 8 middle and
high schools in Madrid
region
‣ Control group and
experimental...
Formal evaluation pre-post
‣ Similar results with
Learning Analytics
than with pre-post
test?
39
learning analytics outcomes
41
Other issues in LA
‣ Ethical and legal aspects
‣ Security model
‣ Ownership of information
• Informing the user
‣ Anonymiz...
But there are new especifications
and develpments that could
sistematize the work
43
ADL eXperience API (xAPI)
‣ Result of Project Tin Can
‣ Tracks experiences, informal learning, real-
world experiences (no...
45
Activity Streams
‣ http://activitystrea.ms
‣ Collaboration between Google, Facebook, Microsoft
and others
‣ Allows repo...
Reporting
Systems
Assessme
nt Services
Semantic
Analysis
Statistical
Services
xAPI Learning Record Store (LRS)
46
From Dam...
‣ Raw data can feed several systems
• An LRS
• A Learning Analytics System
eAdventure + LA with xAPI
Raw data
LRS
Learning...
IMS Global Learning Analytics Interoperability
Framework
http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf
Conclusions
‣ LA in Serious Games has a great potential from the
application and research perspective
• Simplify more comp...
Thank you!
51
balta@fdi.ucm.es
@BaltaFM
Our current projects
Increasing patient safety
using games
Modelling/teaching medical procedures at
National Transplant Or...
Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal
Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal
Upcoming SlideShare
Loading in …5
×

Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal

2,344
-1

Published on

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,344
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
33
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide
  • Educational game, to get teenagers interested in Spanish classical theatre.
  • Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal

    1. 1. Learning Analytics Applied to Serious Games European Conference in Game-Based Learning (ECGBL 2013), Porto, Portugal Baltasar Fernandez-Manjon, balta@fdi.ucm.es , @BaltaFM e-UCM research group, www.e-ucm.es
    2. 2. About me and context ‣ CS Professor at Complutense U. • Director of e-UCM ‣ e-UCM research group about Learning technologies • 15 researchers • Serious games - Application to the medical domain • European projects - GALA - SEGAN - CHERMUG • www.e-ucm.es 2
    3. 3. Learning analytics and SG? The NMC Horizon Report: 2013 Higher Education Edition ‣ new and emerging technologies on teaching, learning, and research Time-to-Adoption Horizon: Two to Three Years ‣ Games and Gamification ‣ Learning Analytics 3 http://www.nmc.org/publications/2013-horizon-report-higher-ed
    4. 4. Serious Games use? ‣ Serious Games have probed to be educationally effective in several domains • Medicine, military, business, corporate training ‣ But still is a low adoption of Serious Games • Cost? ROI? ‣ Serious Games considered usually as a complementary content • Mainly used for motivational purposes • No actual impact on the final mark ‣ Difficult to include Serious Games in the learning curriculum • Assessment of acquired learning? 4
    5. 5. Black box model ‣ Games as independent pieces of content ‣ No information about what is happening during the in-game play ‣ Or very simple • Completed or not completed • Time used 5 del Blanco et al (2013). Using e-Learning standards in educational video games. Computer Standards & Interfaces 36 (1) pp. 178–187
    6. 6. Learning Analytics ‣ Improving education based on data analysis • Data driven ‣ Evidence-based education ‣ Related with … • Educational data mining • Business intelligence • Visual analytics 6 www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
    7. 7. Different uses ‣ Learning analytics • Data analysis that helps students improve learning outcomes. ‣ Academic/program analytics • Data analysis that provides information of what is happening in a specific program and how to plug holes or otherwise adjust. ‣ Institutional analytics • Data analysis that helps make decisions about how to improve at the institutional level. 7 Learning Impact Blog, Big data: Cool; Small data: Cooler http://www.imsglobal.org/blog/?p=258
    8. 8. When data is analyzed? ‣ Off-line • Analyzing data after use • Discovering patterns of use • Allows to improve the experience for future ‣ Real time • Analyzing data while the system is in use to improve/adapt the current learning experience • Allows to also use it in actual presential classes ‣ Mixed approach 8
    9. 9. Extensive Data ‣ Large number of participants ‣ Relatively limited number of variables ‣ Usually very little demographic information ‣ Relatively few observations for each user ‣ Wide but shallow data set 9 Adapted from: Learning Analytics and Educational Data Mining Workshop New York University – CREATE Lab April 4–5, 2013
    10. 10. Intensive Data ‣ Relatively low number of participants ‣ Large number of observations for each variable ‣ Large number of variables for each participant, such as • User actions, In-Game Events • Survey responses; Extensive demographic information • Video Observations • Biometric Data (HR, RESP, GSR, EEG, EKG) • Eye-tracking ‣ Narrow but deep data set ‣ Correlations among different data? 10 Adapted from : Learning Analytics and Educational Data Mining Workshop New York University – CREATE Lab April 4–5, 2013
    11. 11. But there is a parallel world … 11 http://us.battle.net/wow/en/media/screenshots/races?keywords=&view#/goblins04
    12. 12. Game Analytics ‣ Application of analytics to game development and research ‣ Telemetry • Data obtained over distance • Mobile games, MMOG ‣ Game metrics • Interpretable measures of data related to games 12
    13. 13. Game metrics ‣ User metrics • Customer • Player ‣ Performance metrics • Technical infrastructure ‣ Process metrics • Development of the game ‣ User metrics • Generics metrics • Genre specific metrics • Game specific metrics 13
    14. 14. Game metrics 14 Springer, Game Analytics Maximizing the Value of Player Data, pag 22
    15. 15. Game requirements for LA ‣ Most of games are black boxes. • No access to what is going on during game play. ‣ We need access to game “guts” • Game state, game variables ‣ Or the game must communicate with the outside world • Using some logging framework • Not applicable to COTS games (yet) ‣ Mozila Open Badges? http://openbadges.org/ 15
    16. 16. SESSION VARIABLES PLAYER GAME VARIABLES GAME VARIABLES TRACES GOALS From game data to educational information PLAYER VARIABLES User Information Educational System Information SERIOUS GAME
    17. 17. Sessions GAME PLAYER SESSIONS+ = A player plays a game. Produces a game session. One game session is a set of traces of ONE specific player in ONE specific game. SESSIONS = TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E player started phase 1 player scored 200 in phase 1 player clicked in Help button
    18. 18. VARIABLES GENERATO R SESSION Analyzing game sessions TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E TRAC E VAR VAR VAR VAR VAR VAR VAR VAR VAR VAR GOAL RESULT GOAL RESULT GOAL RESULT GOAL RESULT GOAL RESULT GOALS GENERATO R VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N GOAL DEFINITIO N GOAL DEFINITIO N GOAL DEFINITIO N GOAL DEFINITIO N
    19. 19. Variable generator: User sessions VAR DEFINITION VAR+ = TRACE TRACE varName:‘beatgame‘, operation: { type:'trace_present‘ ,traceType: ‘logic', trace: { event:‘end_game’ } } event: ‘end_game', timeStamp: 1380181610150 usersession.beatgame = true;+ =
    20. 20. Goal Generators: User sessions VAR VAR GOAL DEFINITION GOAL RESULT= usersession: { beatgame: true, score: 4747 } id: ‘finalResult’ result: ‘beatgame && score > 3000’ } usersession: { beatgame: true, score: 4747, goals: { finalResult: true } } = + +
    21. 21. PLAYER * GAME Analyzing game results SESSION SESSION VAR VAR VAR VAR GOAL RESULT GOAL RESULT SESSION SESSION VAR VAR VAR VAR GOAL RESULT GOAL RESULT VARIABLES GENERATO R GOALS GENERATO R VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N VAR DEFINITIO N GOAL DEFINITIO N GOAL DEFINITIO N GOAL DEFINITIO N GOAL DEFINITIO N
    22. 22. eAdventure game platform Open code authoring environment for the production of point-and-click adventure games & immersive learning simulations Easy to include Learning Analytics in eAdventure games
    23. 23. http://www.chermug.eu With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es With ONT, educ@ONT
    24. 24. eAdventure + Learning Analytics Game Engine Communication API { type: 'input', timeStamp: some_timestamp, device: 'some_device', action: 'some_action', target: 'target_id', data: { key1: value, ...} } { type: 'logic', timeStamp: some_timestamp, event: 'some_event', target: 'some_id', data: { key1: value, ...} } Learning Analytics Database
    25. 25. GLEANER ‣ GLEANER: Game Learning Analytics for education research • Open code framework to capture game traces 25 Reference model in the EU NoE GALA
    26. 26. GLEANER Collector ‣ Implemented: Nodejs + Mongodb ‣ A web server ready to receive traces ‣ Cleans and sorts the data
    27. 27. GLEANER Analysis ‣ Reporter has access to the database, and presents its data through reports • graphics, heat-maps, relational tables… ‣ Evaluator has access to the database and checks the educational defined goals in the assessment model
    28. 28. Example: A game to learn XML http://gleaner.e-ucm.es/lostinspace/play/index.html
    29. 29. About the game... ‣ A basic platform game to acquire familiarity with XML documents • Learn the syntax • Understand nesting and attributes • Gain agility writing and reading XML documents ‣ Designed as a complementary activity in a Web Programming Course • For undergraduate computer science students ‣ Not developed with eAdventure
    30. 30. Understanding the game… Goal Main character Power-ups (new syntax elements) Write XML snippets here to move the main character to the goal
    31. 31. Educational goals ‣ Basic understanding of XML Syntax • Markup syntax • Basic documents • Attributes • Complex documents ‣ Basic programing skills • Sequencing • Loops
    32. 32. LA perspective ‣ What does GLEANER trace? • Higher level events • Generic traces & game specific traces • Only those relevant for our learning objectives ‣ The aggregator will filter and transmit: • Level completion events • Each XML fragment submitted by the player
    33. 33. Main dashboard (GLEANER Report) 33
    34. 34. Progress report 34 User Session Phases completed Current score Learning achievements Achievements are updated and highlighted in real time
    35. 35. Detailed traces 35 Filters XML snipets The fragments are updated in real time as submitted by the student. The instructor can filter to see only mistakes, or all the fragments submitted by a specific students.
    36. 36. La Dama Boba : The Game Based on The Foolish Lady by Lope de Vega The game is available at http://damaboba.e-ucm.es/ (in Spanish)
    37. 37. Experiment game vs class ‣ 757 students ‣ From 8 middle and high schools in Madrid region ‣ Control group and experimental group
    38. 38. Formal evaluation pre-post ‣ Similar results with Learning Analytics than with pre-post test? 39
    39. 39. learning analytics outcomes
    40. 40. 41
    41. 41. Other issues in LA ‣ Ethical and legal aspects ‣ Security model ‣ Ownership of information • Informing the user ‣ Anonymization of information ‣ Aspects specially relevant if you are working with kids! 42
    42. 42. But there are new especifications and develpments that could sistematize the work 43
    43. 43. ADL eXperience API (xAPI) ‣ Result of Project Tin Can ‣ Tracks experiences, informal learning, real- world experiences (not just completions) ‣ Allows data storage AND retrieval (ex. 3rd party reporting and analytics tools) ‣ Enables tracking mobile, games, ITS, and virtual worlds experiences ‣ Developed by open source community 44 From Damon Regan (ADL) presentation at SINTICE2013
    44. 44. 45 Activity Streams ‣ http://activitystrea.ms ‣ Collaboration between Google, Facebook, Microsoft and others ‣ Allows reporting of experiences, not just completions ‣ Format: <Actor> <Verb> <Object> (I did this): Simple Statement: I (actor) watched (verb) a video on protecting employee data (object) Complex: in the context of [information assurance certification training] with result [timestamp:2013-0618T18:30:32.360Z ]. From Damon Regan (ADL) presentation at SINTICE2013
    45. 45. Reporting Systems Assessme nt Services Semantic Analysis Statistical Services xAPI Learning Record Store (LRS) 46 From Damon Regan (ADL) presentation at SINTICE2013
    46. 46. ‣ Raw data can feed several systems • An LRS • A Learning Analytics System eAdventure + LA with xAPI Raw data LRS Learning Analytics System Statements Analyzer Statements Analyzer EXPERIENCE API EXPERIENCE API
    47. 47. IMS Global Learning Analytics Interoperability Framework http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf
    48. 48. Conclusions ‣ LA in Serious Games has a great potential from the application and research perspective • Simplify more complex and complete experiments ‣ LA in Serious Games should benefits from Games Analytics experience and work ‣ Still complex to implement LA in SG • Increases the (already high) cost of the games ‣ Frameworks and new standards specifications could greatly simplify LA implementation and adoption 50
    49. 49. Thank you! 51 balta@fdi.ucm.es @BaltaFM
    50. 50. Our current projects Increasing patient safety using games Modelling/teaching medical procedures at National Transplant Organization
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×