Some psychometric and design implications of game-based analytics
Presented from a paper by Clarke-Midura and Gibson, 2013. A game played by 1900 middle school students was analyzed to determine if signatures of scientific reasoning (e.g. forming a hypothesis from data) could be found in the click track data. ABSTRACT The rise of digital game and simulation-based learning applications has led to new approaches in educational measurement that take account of patterns in time, high resolution paths of action, and clusters of virtual performance artifacts. The new approaches, which depart from traditional statistical analyses, include data mining, machine learning, and symbolic regression. This article briefly describes the context, methods and broad findings from two game-based analyses and describes key explanatory constructs use to make claims about the users, as well as the implications for design of digital game-based learning and assessment applications. Conclusion: Highly interactive, high-resolution log file data from virtual performance assessments show promise for documenting in new ways what students know and can do. Data mining, machine learning and symbolic regression techniques are effective tools for analyzing and making sense from the time-based records and for relating those to both automated and human scoring artifacts. New psychometric challenges are emerging due to the dynamics, layered resolution levels, and complex patterning of actions with objects in virtual performance assessment spaces. Learning analytics analyses are helping uncover and articulate the relationship of time-event appraisals, visualization structures and resource utilization constraints on the psychometrics of virtual performance assessments.
Recommended
More Related Content
Similar to Some psychometric and design implications of game-based analytics
Similar to Some psychometric and design implications of game-based analytics (20)
Recently uploaded
Recently uploaded (20)
Some psychometric and design implications of game-based analytics
- 1. Some Psychometric and Design Implications of Game-Based Learning Analytics David Gibson Curtin University Jody Clarke-Midura Harvard & MIT
- 2. Abstract • The rise of digital game and simulation-based learning applications has led to new approaches in educational measurement that take account of patterns in time, high resolution paths of action, and clusters of virtual performance artifacts. • The new approaches, which depart from traditional statistical analyses, include data mining, machine learning, and symbolic regression.
- 3. The Premise In an interactive digital-game, traces of a learner’s progress, problem-solving attempts, self-expressions and social communications can entail highly detailed and time-sensitive computer-based documentation of the context, actions, processes and products.
- 4. Example Virtual Performance Assessment Clarke-Midura & Gibson, 2013 • Contexts: Farm, Playground, Science Lab • Actions: Talking, Testing, Walking to… • Processes & Products: Test Results, Explanations
- 5. Interaction Traces = Evidence There is a need for new frameworks, concepts and methods for measuring what someone knows and can do based on game interactions and artifacts created during serious play Why? Because ubiquitous, unobtrusive, interactive big data (fast, varied, voluminous) is created by people working in digital media performance spaces
- 6. Example of Ecological rationality & Empirical probability • Shifts from prediction to claim indicate that the simulation might be educative Clarke-Midura & Gibson, 2013
- 7. Sensors • Wireless EEG – Facial muscles, emotional clusters, raw EEG • Wireless Galvanic Skin Conductance – Arousal level • Eye Tracker – Gaze-point, duration, mouse-clicks • Haptics – Button presses, head tilt
- 8. Anatomy of the System Helen Chavez & Javier Gomez, ASU
- 9. Data Dashboard at ASU Helen Chavez and Javier Gomez, ASU
- 10. Biometric Sensor Nets • What patterns do we find? • How do they change over time? • How do they relate to baseline and experimental activities?
- 11. Challenge: New Psychometrics • What are some of the measurement and analysis considerations needed to address the challenges of finding patterns and making inferences based on data from digital learning experiences?
- 12. Network Graphs Digraphs illustrate structural relationships in the causative factors during a time slice or event frame.
- 13. Network Analysis AF3 F7 AF3 Adjacency tables Centrality F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 GX GY F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 GX GY
- 14. Symbolic Regression Clarke-Midura & Gibson, 2013 Automated search for algorithms Complex nonlinear relationships can be discovered
- 15. New Space for Performance • Unfold in time • Cover a multivariate space of possible actions • Assets contain both intangible (e.g. value, meaning, sensory qualities, and emotions) and tangible components (e.g. media, materials, time and space) NOTE: Asset utilization during performance provides evidence of what a user knows and can do
- 16. Example of Cluster Analysis One group used far fewer resources labeled as ‘salient strategies’
- 17. Example of a rule-based graph Clarke-Midura & Gibson, 2013 Students who had this pattern of resources were most likely to show evidence of forming a hypothesis
- 18. Performance Space Features • Unconstrained complex multidimensional stimuli and responses • Dynamic adaptation of items to user, which entails interactivity and dependency • Nonlinear behaviors with both temporal and spatial components NOTE: Higher order and creative thinking is supported in such a space
- 19. The Game-Based Psychometric Landscape • A “do over” for performance assessment • New ways of performing = new methods of data capture, analysis and display • Complex tasks and artifacts containing – higher order thinking (e.g. decision sequences) – physical performances demonstrating skills – emotional responses
- 20. Thinking States Rise in uncertainty and interest During thinking Agreement & concentration drop
- 21. What Games & Sims Teach • • • • • • • Understanding big ideas - systems knowledge Dealing with time and scale Practice in decision-making Active problem-solving Concepts, strategies, & tactics Understanding processes beyond experience Practice makes improvement (Aldrich, 2005)
- 22. Conclusion Methods based in data-mining, machine learning, model-building and complexity theory form a theoretical foundation for dealing with the challenges of time sensitivity, spatial relationships, multiple layers of aggregations at different scales, and the dynamics of complex behavior spaces.