Learning Assessment
 with Serious Games

Collaboratory for Interactive Learning Research (C.I.L.R.)

          Dept. of Cu...
Performance-based Assessment

          Monitoring of students' progress in relationship to
          identified learner o...
Characteristics of Effective
    Performance Assessment
          Students should be active participants, not passive
    ...
Cognitive Learning Theory

          Knowledge is constructed. Learning is a process of creating
          personal meanin...
Definitions of Assessments

              Formal and informal ways of eliciting evidence of intended student
           A...
Casual Games

          Video game or online game targeted at a mass audience
          Any type of gameplay
          Fit...
Casual games

        Based on Ludology            Examples:
        school of game design         Sudoku
        princip...
Serious Games

          Games for Education, Instruction & Training
          Serious Play (Rieber, 1996), Serious Games ...
Virtual Worlds/Complex Games

          If story-based, then it is based on the
          Naratology school of game design...
Current Focus for Game Design

          Current focus (of the industry) is in game
          design
          What comes ...
Assessment of Learning
     Technology + Learner  Learning?




          Computers

                                 Stu...
Problems of Assessment
     Can we (teachers) know what they (learners) really
     learned?
     UNINTENDED CONTENTS?    ...
Game Studies / Game Research

          European-based                   Scientific research
     GAME STUDIES           ...
Getting into the Black Box

     Qualitative Analysis

                                                          Video-tap...
Measuring Change

     Quantitative Analysis

                                     Educators’ method:
                    ...
New Research Methods


          Large data sets
     DATA MINING                            RESEARCH FRAMEWORK

         ...
In Situ Data Collection

Information Trails
Tracking a Moving Object [Vehicle]

          Vehicle tracking is a way of monitoring the locations,
          movements, ...
Tracking Devices
     INFORMATION RECEIVED FROM    DATA RECEIVED = EVIDENCE OF
     TRACKING DEVICE / SOFTWARE   VIOLATION...
Characteristics of Tracker
          The tracker is a visual moderation and logging software product,
          which enab...
Agent Tracking

          Tracking is the process of locating a moving object (or
          several ones) in time.
     
...
Tracking a Moving Object [GPS]

          A tracking unit is a device that determine the precise
          location of an ...
What Can Be Learned
         Tracking a live moving agent (sea turtle) in its habitat via a
         tracking device attac...
Components of Tracking

     (A) PROCESS                                     (B) TIMELINE




Christian Sebastian Loh, Ph....
Data Processing

         Tracking the movement of Sea Turtle: Process +
          Timeline




          Data Flow: Data...
Foraging » Trails


                                                    foraging
             Animals                     ...
Serious Games & Assessment

            (Serious)
         Video    Games
                                 ?              ...
Serious Games & Assessment

            (Serious)
         Video    Games
                                        ?       ...
Information Nodes

          Conceptually: Series of planned event hooks that are
          embedded within an information...
Information Trails

          Conceptually: A series of agent-detectable markings left by
          a moving agent within ...
Tracking an Avatar
        Avatar: a representation of self in a game environment
        (An extension of self that behav...
Serious Games & Assessment

            (Serious)
         Video    Games                                                 ...
Serious Games & Assessment
      Serious Games
       (with embedded
      Information Nodes)
                            ...
Game Development

       Game Developers’ Toolkit                             Game Mods
                                  ...
Embedding Information Trails

       Game Developers’ Toolkit                             Game Mods
                      ...
In Situ Data Collection
 Assessment of Digital Game-based
 Researchers:

 Learning using Information Trails
              ...
Data Visualization




Christian Sebastian Loh, Ph.D.
                                 Christian Sebastian LOH [csloh@siu....
Data Visualization




Christian Sebastian Loh, Ph.D.   Southern Illinois University Carbondale, USA
Data Visualization




Christian Sebastian Loh, Ph.D.   Southern Illinois University Carbondale, USA
Upcoming SlideShare
Loading in …5
×

Learning Assessment With Serious Games

1,069 views

Published on

Assessment of Learning with Serious Games

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,069
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
19
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Learning Assessment With Serious Games

  1. 1. Learning Assessment with Serious Games Collaboratory for Interactive Learning Research (C.I.L.R.) Dept. of Curriculum & Instruction, COEHS Southern Illinois University Carbondale Carbondale, Illinois, U.S.A.
  2. 2. Performance-based Assessment Monitoring of students' progress in relationship to identified learner outcomes. The Office of Technology Assessment of the U.S. Congress  defines performance assessment as “any form of testing that requires a student to create an answer or a product  that demonstrates his or her knowledge or skills.” This differs from traditional testing methods which require a student to select a single correct answer or to fill in the blank.  Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  3. 3. Characteristics of Effective Performance Assessment Students should be active participants, not passive “selectors of the single right answer." Intended outcomes should be clearly identified and should  guide the design of a performance task. Students should be expected to demonstrate mastery of  those intended outcomes when responding to all facets of the task.  Students must demonstrate their ability to apply their knowledge and skills to reality-based situations and scenarios.  A clear, logical set of performance-based activities that students are expected to follow should be evident. A clearly presented set of criteria should be available to  help judge the degree of proficiency in a student response.  Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  4. 4. Cognitive Learning Theory Knowledge is constructed. Learning is a process of creating personal meaning from new information and prior knowledge. Learning has social components. Group work is valuable.  People perform better when they know the goal, see models, and know how their performance compares to the standard.  There is great variety in learning styles, attention spans,  memory, developmental paces, and intelligences. It's important to know when to use know when to use  knowledge, how to adapt it, how to manage one's own learning. Motivation, effort, and self-esteem affect learning and  performance. Learning isn't necessarily a linear progression of discrete skills.   Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  5. 5. Definitions of Assessments Formal and informal ways of eliciting evidence of intended student  Assessment learning  Any assessment for which the first priority in its design and practice  Assessment for Learning (AfL) is to serve the purpose of promoting student learning  When the evidence of learning elicited by the assessment is actually  Formative (use of) Assessment used to adapt teaching to meet the learning needs  Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  6. 6. Casual Games Video game or online game targeted at a mass audience Any type of gameplay Fit any genre.  Simple rules  Lack of commitment required -- very little time  commitment  Need no special skills to play  Comparatively low production and distribution costs for the producer  Played on a personal computer online in web browsers   Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  7. 7. Casual games Based on Ludology Examples: school of game design  Sudoku principles    Tetris, Minesweeper  Quick, fun, engaging  Card games  Not story-based  Casino games Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  8. 8. Serious Games Games for Education, Instruction & Training Serious Play (Rieber, 1996), Serious Games (Abt, 1970) Instructors » teachers (public school) » business trainers  Learners » students » employees    Salesman of the Month A Force More Powerful Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  9. 9. Virtual Worlds/Complex Games If story-based, then it is based on the Naratology school of game design principles The story provides the context for  interaction The story “grounds” the experience  The storyline engages the mind/personality Player becomes the “hero” in the plot  Objects in the environment are interact-  able and usually for a reason (plot-related)   Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  10. 10. Current Focus for Game Design Current focus (of the industry) is in game design What comes after design & development?  Implementation and then evaluation Design and evaluation (assessment) is  intertwined  Assessment methodology usually stem from  design Game designers need to think about  assessment as they design games (NOW)  Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  11. 11. Assessment of Learning Technology + Learner  Learning? Computers Students/Trainees (Serious) Video Games Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  12. 12. Problems of Assessment Can we (teachers) know what they (learners) really learned? UNINTENDED CONTENTS? INTENDED CONTENTS? “How do we get into the Black Box ?” Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  13. 13. Game Studies / Game Research European-based  Scientific research GAME STUDIES GAME RESEARCH (Netherland)  Problem/scenario-based MIT  learning (Language/linguistics)  Pioneered by military-based Recommendation why we training  should study games »  Data-driven conclusions Recommendation: because it is a rising social  Why we should study games phenomenon, and is To effect change in  interesting to society    Learning  Phenomenological  Education Ethnographical  Performance  Feministic, etc.   Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  14. 14. Getting into the Black Box Qualitative Analysis Video-taping the whole event Short (1-2 sessions) Medium (a few weeks)  Long (a few months… even  years)  We need to double/triple the  time for data analysis Limited generalization  Case Studies   Ethnography   Phenomenology Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  15. 15. Measuring Change Quantitative Analysis Educators’ method:  Attitude surveys   Measuring effects of new technology against traditional teaching method  Pre-test» Intervention»Post- test  Randomized testing Better Methods?  Require “harder” statistics!  Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  16. 16. New Research Methods Large data sets DATA MINING RESEARCH FRAMEWORK Conceptual Frameworks »  Sampling at regular intervals Test Models  Design for learning/   Tens of thousands/millions data points Data mining design for assessment Learning framework   Analyzing for patterns in  data  Design-based learning? Assessment framework   Clustering of data (amateur vs. experts)  Criterion-Reference (for   Associative data mining games?)  Multimedia mining  Information Trails Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  17. 17. In Situ Data Collection Information Trails
  18. 18. Tracking a Moving Object [Vehicle] Vehicle tracking is a way of monitoring the locations, movements, status and behavior of a vehicle or fleet of vehicles.  This is achieved through a combination of an electronic tracking device fitted in the vehicle, a method of returning the data to the user and PC or web-based  software. The data is turned into information by management What tools in conjunction with a visual display on computerized mapping software.  reportingcan be learned? Driver 1 on schedule, lunch break: 2 eggs, 24 chips, 7 mushrooms and fried bread Driver 3 helped friend move house Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  19. 19. Tracking Devices INFORMATION RECEIVED FROM DATA RECEIVED = EVIDENCE OF TRACKING DEVICE / SOFTWARE VIOLATION » NOTIFICATION Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  20. 20. Characteristics of Tracker The tracker is a visual moderation and logging software product, which enables meeting participants to keep track of their meeting goals and the necessary steps to achieving them, as well as allowing  contributions to be anchored within the meeting log immediately. The tracker graphically depicts the course of the current meeting and locates it within the overall context. Tangential departures from the meeting objectives can immediately be seen.  The course of the meeting is structured efficiently and becomes traceable. At the end of a meeting, the minutes are already completed and can immediately be approved by all participants.  For the most important meeting situations (such as project kick-off meeting, information meeting, decision meeting, crisis meeting, problem analysis meeting, etc.), the tracker offers proven agenda  templates which will ensure an efficient meeting process. Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  21. 21. Agent Tracking Tracking is the process of locating a moving object (or several ones) in time.  Tracking a Tropical Depression Tracking the Position of an Object via GPS Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  22. 22. Tracking a Moving Object [GPS] A tracking unit is a device that determine the precise location of an animal, person, or other asset to which it is attached and to record the position of the asset at  The recorded location data can be stored within the regular intervals. tracking unit, or it may be transmitted to a central location data base, or internet-connected computer.  This allows the asset’s location to be displayed against a map backdrop either in real-time or when analyzing the track later, using customized software.  Ref: http://en.wikipedia.org/wiki/GPS_tracking Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  23. 23. What Can Be Learned Tracking a live moving agent (sea turtle) in its habitat via a tracking device attached to the agent  Christian Sebastian LOH [csloh@siu.edu] University • Ref: http://www.bonairenature.com/turtles/turtletracking.html Christian Sebastian Loh, Ph.D. Southern Illinois Carbondale, USA
  24. 24. Components of Tracking (A) PROCESS (B) TIMELINE Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  25. 25. Data Processing  Tracking the movement of Sea Turtle: Process + Timeline Data Flow: Data Logger » Pusher » Puller » Store Data Mining: Data Collection » Reduction » Visualization   Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  26. 26. Foraging » Trails foraging Animals Food Source Path Finding (Trails) start • Pirolli, P., & Card, S. K. (1999). Information foraging. Psychological Review, 106(4), 643-675. Information Foraging • Chi, E. H., Pirolli, P., Chen, K., & Pitkow, J. (2001, March 31-April 4). Using information scent to model user information needs and actions on the Web. Paper presented at the SIGCHI '01, Seattle, WA. Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  27. 27. Serious Games & Assessment (Serious) Video Games ? (Verification Assessment of Learning) Learning Goal Process of Learning start Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  28. 28. Serious Games & Assessment (Serious) Video Games ? (Verification Assessment of Learning) Information Trails Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  29. 29. Information Nodes Conceptually: Series of planned event hooks that are embedded within an information ecology. Operationally: Series of planned event hooks waiting to be  trigger by players’ avatars as they traverse a virtual game environment. Once an event is triggered, a marker  will be placed on the trail to mark the location, time, and outcome of the event. Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  30. 30. Information Trails Conceptually: A series of agent-detectable markings left by a moving agent within an information ecology. Operationally, it is the long track of information markings  left behind by players’ avatars as they traverse a virtual game environment (i.e. triggered event hooks)  Christian Sebastian LOH [csloh@siu.edu] • Collaboratory for InteractiveUniversity Carbondale, Southern Illinois Learning Research • Southern Illinois University Carbondale Christian Sebastian Loh, Ph.D. USA
  31. 31. Tracking an Avatar Avatar: a representation of self in a game environment (An extension of self that behaves just like a live being in a habitat)  Can a tracking device be attached on to an avatar, so as to track the moving agent? (Just like in the case of a vehicle or a turtle?)  Data (locations, status, behavior) about the avatar can then be transmitted out of the game system, and be received and stored in an external data store for later analysis  (Data logger » Push » Pull » Store) Data can then be represented graphically, where goals and behaviors can be scored as assessment  Christian Sebastian LOH [csloh@siu.edu] • Collaboratory for InteractiveUniversity Carbondale, Southern Illinois Learning Research • Southern Illinois University Carbondale Christian Sebastian Loh, Ph.D. USA
  32. 32. Serious Games & Assessment (Serious) Video Games (Verification Assessment of Learning) Data Data Visualization data (Data Display) Collection Data Reduction Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  33. 33. Serious Games & Assessment Serious Games (with embedded Information Nodes) (Verification Assessment of Learning) Data Collection Data Visualization (Information Trails) data (Data Display) Data Reduction Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  34. 34. Game Development Game Developers’ Toolkit Game Mods (Free, New Contents) Commercial (Serious) Off-The-Shelf (COTS) Video Games Data Store Games Persistent Worlds (Free Multiplayer Games) Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  35. 35. Embedding Information Trails Game Developers’ Toolkit Game Mods (Free, New Contents) (Serious) Commercial Video Games Off-The-Shelf (COTS) Data Store Games Embedding Information Trails Serious Game Developers: into the Game Development Process Persistent Worlds (Free Multiplayer Games) Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  36. 36. In Situ Data Collection Assessment of Digital Game-based Researchers: Learning using Information Trails Game Mods (Free, New Contents) Game Developers’ Toolkit (Serious) Video Games Data Store Commercial Off-The-Shelf (COTS) Games Assessment of Learning (Based on Learners’ Persistent Worlds Behaviors, Decisions) (Free Multiplayer Games) Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  37. 37. Data Visualization Christian Sebastian Loh, Ph.D. Christian Sebastian LOH [csloh@siu.edu] University Southern Illinois • Carbondale, USA
  38. 38. Data Visualization Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
  39. 39. Data Visualization Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA

×