Dr. Brian Nelson, Arizona State University, presentation on “Design for Learning and Assessment in Virtual Worlds” as part of our Cognitive Systems Institute Speaker Series.
Design for Learning and Assessment in Virtual Worlds
1. Brian C. Nelson
Arizona State University
August 2016
Design for Learning and Assessment in Virtual Worlds
2. Why Virtual Worlds?
Early take: teaching/training is about information
transmission and uptake by individuals
More recent view: learning is primarily situated, social
activity of collaborating to make sense of and apply
content and concepts in specific contexts
Many commercial virtual world-based games are
based on social networks and collaborative problem
solving…in and outside of the game environment
Virtual world-based games are really good at
collecting data about player activities
3. Why Virtual Worlds (2)?
Virtual Worlds and other digital media fill every
moment of a learner’s life…until they enter the
classroom
To students in a media-rich world, the classroom can
feel like a museum
Virtual worlds engage learners beyond a novelty
effect
Virtual worlds may support demonstrations of
learning different/beyond that supported by
traditional assessments
4. Educational Virtual Worlds: A study in
Contrasts
1. Well -designed virtual worlds are good for learning
Many educational virtual worlds are poorly designed
2. Virtual worlds can support innovative assessment of
“21st century skills”
Many educational virtual worlds use traditional measures
and approaches to assess learning
3. Virtual worlds can support innovative thinking and
multiple ways of knowing
Policy and cultural issues result in virtual worlds that guide
learners toward homogenous thinking and simple answers
5. My Goals Related to these Issues
1. Virtual Worlds are
poorly designed
2. Virtual Worlds use ill-
suited assessments
1. Theory-based design
for learning in Virtual
Worlds
2. Embed meaningful
assessments
Issue Goal
6. My Goals Related to these Issues
1. Virtual worlds are
poorly designed
2. Virtual worlds use ill-
suited assessments
1. Theory-based design
for learning
2. Embed meaningful
assessments
Issue Goal
7. Challenge: Education researchers focus
(understandably) on curricular and pedagogical
issues
Need: Study design of virtual worlds as
implemented in educational settings
Better Design for Learning in Virtual
Worlds
8. Complexity in Virtual Worlds
Visual and interaction complexity boosts
immersion/embodiment but may hinder learning in
school settings
large body of literature on design principles from a
cognitive processing perspective
Summary: cut the cognitive fat
Yet...successful virtual worlds are frequently highly
complex, but players can cope with that complexity and
learn
Pragmatic approach: investigate the use of cognitive
processing based design to balance complexity, short
and long-term engagement, and efficiency
9. Chris Dede, Diane Ketelhut, Ed Dieterle, Jody Clarke-
Midura, Cassie Bowman, and a whole bunch more people!
River City
10. River City
An Multi-player world to teach scientific inquiry and
content skills to middle school students
Students work in teams to discover why people in
River City are getting sick
Students gather data, form and test hypotheses
More than 20,000 students have taken part
11. River City Origins
National policy focus on real-world science
practices
Push to include science inquiry into the classroom
But…realistic inquiry is difficult to teach and
difficult to learn in the classroom
Challenge: Create collaborative, situated inquiry
experiences that engage more students in science,
particularly those underrepresented in STEM fields
15. Voice vs. Text Chat collaboration
Ben Erlandson (former ASU PhD student) led study
people learn better when words are presented as audio
narration rather than as on-screen text (Mayer, 2005)
helps reduce a “split attention” effect
Do students completing a science inquiry curriculum in a game using voice chat for
collaborative communication...
self-report lower levels of cognitive load
show better performance on a science learning measure
...than students collaborating via text chat?
16. Results
Cognitive load
Voice-based Chat: lower levels of perceived cognitive
load
Learning
almost identical performance for both groups
Why?
Everyone did well on the pre-test
Assessment-performance mismatch
17. Diane Ketelhut, Catherine Schifter, Younsu Kim, Uma
Natarajan, Kent Slack, Angela Shelton, (and many more
folks)
SAVE Science
18. SAVE Science
situated assessment using virtual worlds for
science content and inquiry
Virtual world-based Game to assess learning
of classroom curriculum in science
Collect data evolving levels of understanding
Enable students who don’t do well with
standardized tests to better show
understanding
20. Visual Signaling in Virtual Worlds
Virtual worlds work well for learning, especially
over long(er) periods of engagement
Virtual worlds are initially confusing, especially
for novice student gamers
Low efficiency poses challenges to in-school
implementations
Low efficiency challenges assessment reliability
and validity
Visual signaling is used to guide players to
relevant objects and locations
22. Sheep Trouble Module
https://www.youtube.com/watch?v=uurufkuXu3s
Assess students’ knowledge of beginning
speciation/adaptation as well as aspects of
scientific inquiry.
New and old flocks of sheep
Determine why recently imported sheep are
getting sick and dying
Apply understanding of speciation and
adaptation
25. Visual Signaling: Cognition
Visual Signaling: using visual cues (such as arrows) to
direct learner attention to relevant information on the
screen or page (or virtual world)
Visual Signaling may lower extraneous cognitive load
and/or increase germane load…letting learners
focus on tasks rather than on interface (Merrienboer,
2008)
Mixed record in past studies: often found to reduce
self-reported cognitive load, but not always coupled
with improved learning (Morozov, 2009; Chen &
Fauzy, 2008)
26. Signaling Questions
Can the use of visual signaling techniques reduce
perceived extraneous cognitive load in a short game-
based assessment?
Can use of signaling increase the efficiency in a game-
based assessment?
27. Signaling Study
193 7th graders
Sheep Trouble: Assessment of Beginning Speciation
Random assignment: signaling/no signaling
Lower overall perceived cognitive load (p<.05)
Increased interactions with sheep (p<.01)
More measurements taken (p<.001)
More records entered in notebook (p<.001)
28. Implications and Questions
Use signaling!
Why did signaling have a ‘sticky’ effect on
interacting with objects?
Would the value of signaling for efficiency be
greater in a high search environment? (One with
more visual objects on the screen at once?)
29. My Goals Related to these Issues
1. Virtual Worlds are
poorly designed
2. Virtual Worlds use ill-
suited assessments
1. Theory-based design
for learning
2. Embed meaningful
assessments
Issue Goal
30. Emerging research on:
Data-mining
Statistical methods for making sense of learner
actions
Less research on:
Design of tasks and “Work Products” of assessment
supported by highly immersive virtual worlds
Virtual world-based Assessment
31. Evidence Centered Design for Assessment in
Virtual Worlds
…and/or the Presentation Model aspect of ECD
(Robert Mislevy)
32. Assessment Tasks and Work products
Assessment in Game-based learning environments:
many researchers and designers focusing mainly on
‘black box’ analysis of data output
Virtual worlds are designed spaces
Need full spectrum design for more meaningful
data
33. Assessment tasks in Virtual Worlds
Virtual world-based tasks support multiple evidence
channels in isolation and in combination
Provide complex and interwoven collection of work
tools for assessment activities
34. Example: Global Evidence Channels
Location/Movement (LM) Object Interaction (OI) Communication Activities
(CA)
Location tracking
•X location visited
•Time spent at X
•Coordinates
Movement tracking
•Direction
•Speed
•Acceleration/deceleration
•Teleporting
Movement patterns
•Order of movement
•Movement as response
•Movement strings over time
Objects:
•View
•Select
•Click
•Manipulate
•Pickup
•Release
Object Types:
•Artifacts and inventory
•Tools
•NPCs
•Humans
•“intangibles”
•Type
•Speak
•Response selection
•Emote
•In and out of character
•Human and NPC
•Goal-oriented vs. social
36. Basketball Module
http://youtu.be/hrZVa2i-e5I
Assess students’ knowledge of gas laws and
related properties as well as aspects of
scientific inquiry.
Mid-winter basketball tournament
Determine why balls at outdoor game don’t
bounce well compared to indoor setting
Apply understanding of gas laws (air
pressure/temperature link)
38. Study
1. Create automated grading models that predict the
number of embedded assessment questions a
student will answer correctly based on her/his
actions in the module
2. 187 students’ records analyzed
3. Analyze correlations between multiple-choice scores,
within-game behavior, and free-text answers
4. Correlation of .5 (Pearson’s p) with human graders
on predicting performance on in-game multiple
choice and short-answer questions
39. Study
4 important and non-redundant features found:
Distinct interactions with in-game objects
Number of NPCs talked to
Number of objects whose air pressure was measured
Number of temperature measurements recorded in e-
notebook
Key task: discover that a decrease in the temperature
of several gas systems (basketballs and balloons
filled with air) is causing their pressure to decrease.
40. Example
A graphical illustration of how our Classification Techniques are able to learn to distinguish
between Advanced, Proficient, and Basic student evaluations (x indicates incorrect prediction)
• Good predictors for grades are highlynon-linear
• Spherical boundaries approximately indicate the student groups