The document discusses leveraging data from digital simulations to assess learning. It notes that simulation data provides detailed, time-sensitive documentation of learner actions, processes, and products. New methods are needed to analyze patterns in this "big data" from things like sensors, and relate patterns to learner activities and performance. The document provides examples of using network graphs and machine learning on simulation data to measure higher-order thinking and infer what learners know based on their in-simulation behaviors and artifacts.
2. The Premise
In an interactive digital simulation, traces of a
learner’s progress, problem-solving attempts,
self-expressions and social communications can
entail highly detailed and time-sensitive
computer-based documentation of the context,
actions, processes and products.
3. New Psychometric Landscape
• A “do over” for performance assessment
• New ways of performing & new methods of
data capture, analysis and display
• Complex tasks and artifacts containing:
– higher order thinking (e.g. decision sequences)
– physical performances demonstrating skills
– emotional responses
7. Research Questions
• What patterns are
found within &
between sensors?
• How do these patterns
relate to baseline and
experimental
activities?
8. Interaction Traces = Evidence
There is a need for new frameworks, concepts
and methods for measuring what someone
knows and can do based on game interactions
and artifacts created during serious play
Why? (It’s a mouthful) Ubiquitous, unobtrusive,
interactive big data (fast, wide variety &
voluminous) created by people working in
digital media performance spaces
9. New Psychometrics
• What are some of the measurement and
analysis considerations needed to address the
challenges of finding patterns and making
inferences based on data from digital learning
experiences?
15. New Space for Performance
• Unfold in time
• Cover a multivariate space of possible actions
• Assets contain both intangible (e.g. value,
meaning, sensory qualities, and emotions)
and tangible components (e.g. media,
materials, time and space)
NOTE: Asset utilization during performance
provides evidence of what a user knows and
can do
16. Example
Clarke-Midura & Gibson, 2013
Students who had
this pattern of
resources were
most likely to
show evidence of
forming a
hypothesis
17. Performance Space Features
• Unconstrained complex multidimensional
stimuli and responses
• Dynamic adaptation of items to user, which
entails interactivity and dependency
• Nonlinear behaviors with both temporal and
spatial components
NOTE: Higher order and creative thinking is
supported in such a space
19. Conclusion
Methods based in data-mining, machine
learning, model-building and complexity theory
form a theoretical foundation for dealing with
the challenges of time sensitivity, spatial
relationships, multiple layers of aggregations at
different scales, and the dynamics of complex
behavior spaces.