Brian C. Nelson
Arizona State University
August 2016
Design for Learning and Assessment in Virtual Worlds
Why Virtual Worlds?
 Early take: teaching/training is about information
transmission and uptake by individuals
 More rec...
Why Virtual Worlds (2)?
 Virtual Worlds and other digital media fill every
moment of a learner’s life…until they enter th...
Educational Virtual Worlds: A study in
Contrasts
1. Well -designed virtual worlds are good for learning
 Many educational...
My Goals Related to these Issues
1. Virtual Worlds are
poorly designed
2. Virtual Worlds use ill-
suited assessments
1. Th...
My Goals Related to these Issues
1. Virtual worlds are
poorly designed
2. Virtual worlds use ill-
suited assessments
1. Th...
Challenge: Education researchers focus
(understandably) on curricular and pedagogical
issues
Need: Study design of virtu...
Complexity in Virtual Worlds
 Visual and interaction complexity boosts
immersion/embodiment but may hinder learning in
sc...
Chris Dede, Diane Ketelhut, Ed Dieterle, Jody Clarke-
Midura, Cassie Bowman, and a whole bunch more people!
River City
River City
 An Multi-player world to teach scientific inquiry and
content skills to middle school students
 Students wor...
River City Origins
 National policy focus on real-world science
practices
 Push to include science inquiry into the clas...
River City Interface: a mess
Cognitive Design
 Keep learner focus in the 3-d environment
 Reduce reading through “natural narration”
 Use visual and...
Individual investigations of context-based science problems in a virtual world
Example 1: Simlandia
Voice vs. Text Chat collaboration
Ben Erlandson (former ASU PhD student) led study
people learn better when words are pr...
Results
Cognitive load
 Voice-based Chat: lower levels of perceived cognitive
load
Learning
 almost identical performanc...
Diane Ketelhut, Catherine Schifter, Younsu Kim, Uma
Natarajan, Kent Slack, Angela Shelton, (and many more
folks)
SAVE Scie...
SAVE Science
 situated assessment using virtual worlds for
science content and inquiry
 Virtual world-based Game to asse...
SAVE Science Design Studies (so far)
 Visual Signaling
 Avatar Personalization
 Spatial Contiguity
Visual Signaling in Virtual Worlds
 Virtual worlds work well for learning, especially
over long(er) periods of engagement...
Sheep Trouble Module
Sheep Trouble Module
 https://www.youtube.com/watch?v=uurufkuXu3s
 Assess students’ knowledge of beginning
speciation/ad...
Investigating the sheep
Measuring sheep
Visual Signaling: Cognition
 Visual Signaling: using visual cues (such as arrows) to
direct learner attention to relevant...
Signaling Questions
Can the use of visual signaling techniques reduce
perceived extraneous cognitive load in a short game...
Signaling Study
 193 7th graders
 Sheep Trouble: Assessment of Beginning Speciation
 Random assignment: signaling/no si...
Implications and Questions
 Use signaling!
 Why did signaling have a ‘sticky’ effect on
interacting with objects?
 Woul...
My Goals Related to these Issues
1. Virtual Worlds are
poorly designed
2. Virtual Worlds use ill-
suited assessments
1. Th...
Emerging research on:
 Data-mining
 Statistical methods for making sense of learner
actions
Less research on:
 Design...
Evidence Centered Design for Assessment in
Virtual Worlds
…and/or the Presentation Model aspect of ECD
(Robert Mislevy)
Assessment Tasks and Work products
 Assessment in Game-based learning environments:
many researchers and designers focusi...
Assessment tasks in Virtual Worlds
 Virtual world-based tasks support multiple evidence
channels in isolation and in comb...
Example: Global Evidence Channels
Location/Movement (LM) Object Interaction (OI) Communication Activities
(CA)
Location tr...
DATA-MINING BASKETBALL
TROUBLE
Shanshan Zhang and Slobodan Vucetic
Department of Computer and Information Sciences
Temple ...
Basketball Module
 http://youtu.be/hrZVa2i-e5I
 Assess students’ knowledge of gas laws and
related properties as well as...
Basketball Module
Study
1. Create automated grading models that predict the
number of embedded assessment questions a
student will answer co...
Study
 4 important and non-redundant features found:
 Distinct interactions with in-game objects
 Number of NPCs talked...
Example
A graphical illustration of how our Classification Techniques are able to learn to distinguish
between Advanced, P...
Brian C. Nelson
Brian.Nelson@asu.edu
Questions?
Upcoming SlideShare
Loading in …5
×

Design for Learning and Assessment in Virtual Worlds

87 views

Published on

Dr. Brian Nelson, Arizona State University, presentation on “Design for Learning and Assessment in Virtual Worlds” as part of our Cognitive Systems Institute Speaker Series.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
87
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Design for Learning and Assessment in Virtual Worlds

  1. 1. Brian C. Nelson Arizona State University August 2016 Design for Learning and Assessment in Virtual Worlds
  2. 2. Why Virtual Worlds?  Early take: teaching/training is about information transmission and uptake by individuals  More recent view: learning is primarily situated, social activity of collaborating to make sense of and apply content and concepts in specific contexts  Many commercial virtual world-based games are based on social networks and collaborative problem solving…in and outside of the game environment  Virtual world-based games are really good at collecting data about player activities
  3. 3. Why Virtual Worlds (2)?  Virtual Worlds and other digital media fill every moment of a learner’s life…until they enter the classroom  To students in a media-rich world, the classroom can feel like a museum  Virtual worlds engage learners beyond a novelty effect  Virtual worlds may support demonstrations of learning different/beyond that supported by traditional assessments
  4. 4. Educational Virtual Worlds: A study in Contrasts 1. Well -designed virtual worlds are good for learning  Many educational virtual worlds are poorly designed 2. Virtual worlds can support innovative assessment of “21st century skills”  Many educational virtual worlds use traditional measures and approaches to assess learning 3. Virtual worlds can support innovative thinking and multiple ways of knowing  Policy and cultural issues result in virtual worlds that guide learners toward homogenous thinking and simple answers
  5. 5. My Goals Related to these Issues 1. Virtual Worlds are poorly designed 2. Virtual Worlds use ill- suited assessments 1. Theory-based design for learning in Virtual Worlds 2. Embed meaningful assessments Issue Goal
  6. 6. My Goals Related to these Issues 1. Virtual worlds are poorly designed 2. Virtual worlds use ill- suited assessments 1. Theory-based design for learning 2. Embed meaningful assessments Issue Goal
  7. 7. Challenge: Education researchers focus (understandably) on curricular and pedagogical issues Need: Study design of virtual worlds as implemented in educational settings Better Design for Learning in Virtual Worlds
  8. 8. Complexity in Virtual Worlds  Visual and interaction complexity boosts immersion/embodiment but may hinder learning in school settings  large body of literature on design principles from a cognitive processing perspective  Summary: cut the cognitive fat  Yet...successful virtual worlds are frequently highly complex, but players can cope with that complexity and learn  Pragmatic approach: investigate the use of cognitive processing based design to balance complexity, short and long-term engagement, and efficiency
  9. 9. Chris Dede, Diane Ketelhut, Ed Dieterle, Jody Clarke- Midura, Cassie Bowman, and a whole bunch more people! River City
  10. 10. River City  An Multi-player world to teach scientific inquiry and content skills to middle school students  Students work in teams to discover why people in River City are getting sick  Students gather data, form and test hypotheses  More than 20,000 students have taken part
  11. 11. River City Origins  National policy focus on real-world science practices  Push to include science inquiry into the classroom  But…realistic inquiry is difficult to teach and difficult to learn in the classroom  Challenge: Create collaborative, situated inquiry experiences that engage more students in science, particularly those underrepresented in STEM fields
  12. 12. River City Interface: a mess
  13. 13. Cognitive Design  Keep learner focus in the 3-d environment  Reduce reading through “natural narration”  Use visual and audio signalling techniques to focus attention  Apply spatial contiguity principle  Design for “essential complexity”
  14. 14. Individual investigations of context-based science problems in a virtual world Example 1: Simlandia
  15. 15. Voice vs. Text Chat collaboration Ben Erlandson (former ASU PhD student) led study people learn better when words are presented as audio narration rather than as on-screen text (Mayer, 2005)  helps reduce a “split attention” effect Do students completing a science inquiry curriculum in a game using voice chat for collaborative communication... self-report lower levels of cognitive load show better performance on a science learning measure ...than students collaborating via text chat?
  16. 16. Results Cognitive load  Voice-based Chat: lower levels of perceived cognitive load Learning  almost identical performance for both groups Why?  Everyone did well on the pre-test  Assessment-performance mismatch
  17. 17. Diane Ketelhut, Catherine Schifter, Younsu Kim, Uma Natarajan, Kent Slack, Angela Shelton, (and many more folks) SAVE Science
  18. 18. SAVE Science  situated assessment using virtual worlds for science content and inquiry  Virtual world-based Game to assess learning of classroom curriculum in science  Collect data evolving levels of understanding  Enable students who don’t do well with standardized tests to better show understanding
  19. 19. SAVE Science Design Studies (so far)  Visual Signaling  Avatar Personalization  Spatial Contiguity
  20. 20. Visual Signaling in Virtual Worlds  Virtual worlds work well for learning, especially over long(er) periods of engagement  Virtual worlds are initially confusing, especially for novice student gamers  Low efficiency poses challenges to in-school implementations  Low efficiency challenges assessment reliability and validity  Visual signaling is used to guide players to relevant objects and locations
  21. 21. Sheep Trouble Module
  22. 22. Sheep Trouble Module  https://www.youtube.com/watch?v=uurufkuXu3s  Assess students’ knowledge of beginning speciation/adaptation as well as aspects of scientific inquiry.  New and old flocks of sheep  Determine why recently imported sheep are getting sick and dying  Apply understanding of speciation and adaptation
  23. 23. Investigating the sheep
  24. 24. Measuring sheep
  25. 25. Visual Signaling: Cognition  Visual Signaling: using visual cues (such as arrows) to direct learner attention to relevant information on the screen or page (or virtual world)  Visual Signaling may lower extraneous cognitive load and/or increase germane load…letting learners focus on tasks rather than on interface (Merrienboer, 2008)  Mixed record in past studies: often found to reduce self-reported cognitive load, but not always coupled with improved learning (Morozov, 2009; Chen & Fauzy, 2008)
  26. 26. Signaling Questions Can the use of visual signaling techniques reduce perceived extraneous cognitive load in a short game- based assessment? Can use of signaling increase the efficiency in a game- based assessment?
  27. 27. Signaling Study  193 7th graders  Sheep Trouble: Assessment of Beginning Speciation  Random assignment: signaling/no signaling  Lower overall perceived cognitive load (p<.05)  Increased interactions with sheep (p<.01)  More measurements taken (p<.001)  More records entered in notebook (p<.001)
  28. 28. Implications and Questions  Use signaling!  Why did signaling have a ‘sticky’ effect on interacting with objects?  Would the value of signaling for efficiency be greater in a high search environment? (One with more visual objects on the screen at once?)
  29. 29. My Goals Related to these Issues 1. Virtual Worlds are poorly designed 2. Virtual Worlds use ill- suited assessments 1. Theory-based design for learning 2. Embed meaningful assessments Issue Goal
  30. 30. Emerging research on:  Data-mining  Statistical methods for making sense of learner actions Less research on:  Design of tasks and “Work Products” of assessment supported by highly immersive virtual worlds Virtual world-based Assessment
  31. 31. Evidence Centered Design for Assessment in Virtual Worlds …and/or the Presentation Model aspect of ECD (Robert Mislevy)
  32. 32. Assessment Tasks and Work products  Assessment in Game-based learning environments: many researchers and designers focusing mainly on ‘black box’ analysis of data output  Virtual worlds are designed spaces  Need full spectrum design for more meaningful data
  33. 33. Assessment tasks in Virtual Worlds  Virtual world-based tasks support multiple evidence channels in isolation and in combination  Provide complex and interwoven collection of work tools for assessment activities
  34. 34. Example: Global Evidence Channels Location/Movement (LM) Object Interaction (OI) Communication Activities (CA) Location tracking •X location visited •Time spent at X •Coordinates Movement tracking •Direction •Speed •Acceleration/deceleration •Teleporting Movement patterns •Order of movement •Movement as response •Movement strings over time Objects: •View •Select •Click •Manipulate •Pickup •Release Object Types: •Artifacts and inventory •Tools •NPCs •Humans •“intangibles” •Type •Speak •Response selection •Emote •In and out of character •Human and NPC •Goal-oriented vs. social
  35. 35. DATA-MINING BASKETBALL TROUBLE Shanshan Zhang and Slobodan Vucetic Department of Computer and Information Sciences Temple University
  36. 36. Basketball Module  http://youtu.be/hrZVa2i-e5I  Assess students’ knowledge of gas laws and related properties as well as aspects of scientific inquiry.  Mid-winter basketball tournament  Determine why balls at outdoor game don’t bounce well compared to indoor setting  Apply understanding of gas laws (air pressure/temperature link)
  37. 37. Basketball Module
  38. 38. Study 1. Create automated grading models that predict the number of embedded assessment questions a student will answer correctly based on her/his actions in the module 2. 187 students’ records analyzed 3. Analyze correlations between multiple-choice scores, within-game behavior, and free-text answers 4. Correlation of .5 (Pearson’s p) with human graders on predicting performance on in-game multiple choice and short-answer questions
  39. 39. Study  4 important and non-redundant features found:  Distinct interactions with in-game objects  Number of NPCs talked to  Number of objects whose air pressure was measured  Number of temperature measurements recorded in e- notebook  Key task: discover that a decrease in the temperature of several gas systems (basketballs and balloons filled with air) is causing their pressure to decrease.
  40. 40. Example A graphical illustration of how our Classification Techniques are able to learn to distinguish between Advanced, Proficient, and Basic student evaluations (x indicates incorrect prediction) • Good predictors for grades are highlynon-linear • Spherical boundaries approximately indicate the student groups
  41. 41. Brian C. Nelson Brian.Nelson@asu.edu Questions?

×