Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Developing Number
Sense with Play:
Lessons in Data
Analysis
Ryan Carlson
Overview
• New iPad game to practice & develop number
sense
• In-school user study & pilot study addressing
non-cognitive ...
Outline
• Some Definitions
• NumberACEr
• Designing a configurable game
• Instrumentation
• User Study
• Pilot Study
• Dat...
Who Are We?
• Founded 1998
• Develop K-12 mathematics blended curricula
• 600k students / year across >2,000 schools
• Cog...
What is Number Sense?
“fluidity and flexibility with numbers, the
sense of what numbers mean and ability to
perform mental...
What is Number Sense?
1
2
3
4
What is Number Sense?
4
5
3
4
What is Number Sense?
5
7
2
3
What is Number Sense?
6
13
9
17
Personalization
Cognitive
What you know
(positive results)
• Knowledge Tracing
Non-Cognitive
Everything Else
(less well un...
NumerACEr
NumerACEr
NumerACEr
Configurability
• Content
• Game Parameters
• Level Layout
• Scripting System
• Dynamic runtime engine
• React to game eve...
Configurability
Configurability
Instrumentation
Boring But Important
• Generate usage data & plan analyses before
deployment
• Logging infrastructure & AP...
User Study
• December 2014 in Raleigh, West Virginia
• 6th graders (36)
• 8th graders (7)
• Played for ~10 min
• Exit feed...
User Study
User Study
Aggressive
Neutral
(High Accuracy)
Neutral
(Low Accuracy)
Cautious
User Study
Aggressive
Neutral
(High Accuracy)
Neutral
(Low Accuracy)
Cautious
Pilot Study
• May 2015 in Savannah, Georgia
• 6th graders (200)
• Remote Deployment
• 15 minute play requested
• Non-Cogni...
Pilot Study: Interventions
Self-Efficacy
• Low SE  Motivation
Pilot Study: Interventions
Grit
• High Grit  Increased Difficulty Earning Brains
Learning Orientation
• Low Mastery  Coach Towards Mastery
Pilot Study: Interventions
Pilot Study: Interventions
Game Strategy
• Aggressive  Slowdown
• Reduce maximum speed
• Cautious  Speedup
• Increase mi...
Pilot Study: Interventions
Pilot Study: Interventions
Pilot Study: Game Usage
Pilot Study: Game Usage
Pilot Study: Game Usage
Pilot Study: Game Effectiveness
Pilot Study: Intervention Effectiveness
Relationship b/w Self-Efficacy and how much
students play the game
Self-Efficacy &...
Pilot Study: Intervention Effectiveness
F(1,105)=3.81
p=0.05
Pilot Study: Intervention Effectiveness
χ2 test, p<.01
Pilot Study: Intervention Effectiveness
F(1,176)=3.46
p<0.10
Pilot Study: Intervention Effectiveness
F(1,176)=0.05
p>.10
Pilot Study: Intervention Effectiveness
F(1,176)=4.50
p<.05
Pilot Study: Intervention Effectiveness
F(1,176)=0.03
p>.10
Lessons Learned & Limitations
• Promising directions for adapting to cognitive +
non-cognitive factors into games
• Gamifi...
References
• Geary DC, Hoard MK, Nugent L, Bailey DH (2013) “Adolescents’
Functional Numeracy Is Predicted by Their School...
Thank you!
Questions?
rcarlson@carnegielearning.com
Upcoming SlideShare
Loading in …5
×

Ryan Carlson - Developing Number Sense with Play: Lessons in Data Analysis

382 views

Published on

Presenter: Ryan Carlson, Research Programmer, Carnegie Learning

We developed an iPad racing game to increase students’ math fluency with an architecture that enables rapid prototyping of game features. The game is unique in that it is configurable in both content and behavior, allowing educational researchers, without programming, to quickly build a wide range of experiments to gauge educational gains. For example, researchers can change game parameters, in-game feedback, the reward structure or even make interface changes. Students have been very enthusiastic about the game. Initial tests show big differences in the way students approach the speed/accuracy tradeoff inherent in playing the game. Our hypothesis is that students’ approach reflects both their understanding and motivation. Robust instrumentation enables a deeper understanding of students’ learning and experiences within the game.

  • Be the first to comment

  • Be the first to like this

Ryan Carlson - Developing Number Sense with Play: Lessons in Data Analysis

  1. 1. Developing Number Sense with Play: Lessons in Data Analysis Ryan Carlson
  2. 2. Overview • New iPad game to practice & develop number sense • In-school user study & pilot study addressing non-cognitive factors • Discussion of instrumentation & evaluation of our game
  3. 3. Outline • Some Definitions • NumberACEr • Designing a configurable game • Instrumentation • User Study • Pilot Study • Data Analysis • Lessons Learned
  4. 4. Who Are We? • Founded 1998 • Develop K-12 mathematics blended curricula • 600k students / year across >2,000 schools • Cognitive Tutor • Model student knowledge • Adaptively present new material
  5. 5. What is Number Sense? “fluidity and flexibility with numbers, the sense of what numbers mean and ability to perform mental mathematics to look at the world and make comparisons” (Gersten & Chard, 1999) • Foundation for further mathematics • Early number sense predicts later mathematics ability (Geary et. al., 2013)
  6. 6. What is Number Sense? 1 2 3 4
  7. 7. What is Number Sense? 4 5 3 4
  8. 8. What is Number Sense? 5 7 2 3
  9. 9. What is Number Sense? 6 13 9 17
  10. 10. Personalization Cognitive What you know (positive results) • Knowledge Tracing Non-Cognitive Everything Else (less well understood) • Self-Efficacy • Grit • Learning Orientation
  11. 11. NumerACEr
  12. 12. NumerACEr
  13. 13. NumerACEr
  14. 14. Configurability • Content • Game Parameters • Level Layout • Scripting System • Dynamic runtime engine • React to game events • Add text / images For non- programmers
  15. 15. Configurability
  16. 16. Configurability
  17. 17. Instrumentation Boring But Important • Generate usage data & plan analyses before deployment • Logging infrastructure & API • Developers annotate code • Log messages sent to central storage • Log everything! • Level started? • Speed changes? • iPad tilt? • Device suspend?
  18. 18. User Study • December 2014 in Raleigh, West Virginia • 6th graders (36) • 8th graders (7) • Played for ~10 min • Exit feedback (verbal + written)
  19. 19. User Study
  20. 20. User Study Aggressive Neutral (High Accuracy) Neutral (Low Accuracy) Cautious
  21. 21. User Study Aggressive Neutral (High Accuracy) Neutral (Low Accuracy) Cautious
  22. 22. Pilot Study • May 2015 in Savannah, Georgia • 6th graders (200) • Remote Deployment • 15 minute play requested • Non-Cognitive survey administered prior to play • Self-efficacy • Grit • Mastery learning orientation
  23. 23. Pilot Study: Interventions Self-Efficacy • Low SE  Motivation
  24. 24. Pilot Study: Interventions Grit • High Grit  Increased Difficulty Earning Brains
  25. 25. Learning Orientation • Low Mastery  Coach Towards Mastery Pilot Study: Interventions
  26. 26. Pilot Study: Interventions Game Strategy • Aggressive  Slowdown • Reduce maximum speed • Cautious  Speedup • Increase minimum speed • Neutral (low acc.)  Instruction • Provide in-game feedback on mistakes
  27. 27. Pilot Study: Interventions
  28. 28. Pilot Study: Interventions
  29. 29. Pilot Study: Game Usage
  30. 30. Pilot Study: Game Usage
  31. 31. Pilot Study: Game Usage
  32. 32. Pilot Study: Game Effectiveness
  33. 33. Pilot Study: Intervention Effectiveness Relationship b/w Self-Efficacy and how much students play the game Self-Efficacy & Grit interact in an interesting way
  34. 34. Pilot Study: Intervention Effectiveness F(1,105)=3.81 p=0.05
  35. 35. Pilot Study: Intervention Effectiveness χ2 test, p<.01
  36. 36. Pilot Study: Intervention Effectiveness F(1,176)=3.46 p<0.10
  37. 37. Pilot Study: Intervention Effectiveness F(1,176)=0.05 p>.10
  38. 38. Pilot Study: Intervention Effectiveness F(1,176)=4.50 p<.05
  39. 39. Pilot Study: Intervention Effectiveness F(1,176)=0.03 p>.10
  40. 40. Lessons Learned & Limitations • Promising directions for adapting to cognitive + non-cognitive factors into games • Gamification serves a purpose • Countdown timer adds pressure • Speed forces rapid comparisons • “pretty fun for a school game” • Desire for common game features • Customization, currency • Assumption: the game is a valid measure of number sense
  41. 41. References • Geary DC, Hoard MK, Nugent L, Bailey DH (2013) “Adolescents’ Functional Numeracy Is Predicted by Their School Entry Number System Knowledge”. PLoS ONE 8(1): e54651. • Gersten, R., and D. Chard. “Number Sense: Rethinking Arithmetic Instruction for Students with Mathematical Disabilities.” The Journal of Special Education 33.1 (1999): 18-28
  42. 42. Thank you! Questions? rcarlson@carnegielearning.com

×