Presenter: Ryan Carlson, Research Programmer, Carnegie Learning We developed an iPad racing game to increase students’ math fluency with an architecture that enables rapid prototyping of game features. The game is unique in that it is configurable in both content and behavior, allowing educational researchers, without programming, to quickly build a wide range of experiments to gauge educational gains. For example, researchers can change game parameters, in-game feedback, the reward structure or even make interface changes. Students have been very enthusiastic about the game. Initial tests show big differences in the way students approach the speed/accuracy tradeoff inherent in playing the game. Our hypothesis is that students’ approach reflects both their understanding and motivation. Robust instrumentation enables a deeper understanding of students’ learning and experiences within the game.

1. Developing Number
Sense with Play:
Lessons in Data
Analysis
Ryan Carlson

2. Overview
• New iPad game to practice & develop number
sense
• In-school user study & pilot study addressing
non-cognitive factors
• Discussion of instrumentation & evaluation of
our game

3. Outline
• Some Definitions
• NumberACEr
• Designing a configurable game
• Instrumentation
• User Study
• Pilot Study
• Data Analysis
• Lessons Learned

4. Who Are We?
• Founded 1998
• Develop K-12 mathematics blended curricula
• 600k students / year across >2,000 schools
• Cognitive Tutor
• Model student knowledge
• Adaptively present new material

5. What is Number Sense?
“fluidity and flexibility with numbers, the
sense of what numbers mean and ability to
perform mental mathematics to look at the
world and make comparisons”
(Gersten & Chard, 1999)
• Foundation for further mathematics
• Early number sense predicts later mathematics
ability (Geary et. al., 2013)

6. What is Number Sense?
1
2
3
4

7. What is Number Sense?
4
5
3
4

8. What is Number Sense?
5
7
2
3

9. What is Number Sense?
6
13
9
17

10. Personalization
Cognitive
What you know
(positive results)
• Knowledge Tracing
Non-Cognitive
Everything Else
(less well understood)
• Self-Efficacy
• Grit
• Learning Orientation

11. NumerACEr

12. NumerACEr

13. NumerACEr

14. Configurability
• Content
• Game Parameters
• Level Layout
• Scripting System
• Dynamic runtime engine
• React to game events
• Add text / images
For non-
programmers

15. Configurability

16. Configurability

17. Instrumentation
Boring But Important
• Generate usage data & plan analyses before
deployment
• Logging infrastructure & API
• Developers annotate code
• Log messages sent to central storage
• Log everything!
• Level started?
• Speed changes?
• iPad tilt?
• Device suspend?

18. User Study
• December 2014 in Raleigh, West Virginia
• 6th graders (36)
• 8th graders (7)
• Played for ~10 min
• Exit feedback (verbal + written)

19. User Study

20. User Study
Aggressive
Neutral
(High Accuracy)
Neutral
(Low Accuracy)
Cautious

21. User Study
Aggressive
Neutral
(High Accuracy)
Neutral
(Low Accuracy)
Cautious

22. Pilot Study
• May 2015 in Savannah, Georgia
• 6th graders (200)
• Remote Deployment
• 15 minute play requested
• Non-Cognitive survey administered prior to play
• Self-efficacy
• Grit
• Mastery learning orientation

23. Pilot Study: Interventions
Self-Efficacy
• Low SE  Motivation

24. Pilot Study: Interventions
Grit
• High Grit  Increased Difficulty Earning Brains

25. Learning Orientation
• Low Mastery  Coach Towards Mastery
Pilot Study: Interventions

26. Pilot Study: Interventions
Game Strategy
• Aggressive  Slowdown
• Reduce maximum speed
• Cautious  Speedup
• Increase minimum speed
• Neutral (low acc.)  Instruction
• Provide in-game feedback on mistakes

27. Pilot Study: Interventions

28. Pilot Study: Interventions

29. Pilot Study: Game Usage

30. Pilot Study: Game Usage

31. Pilot Study: Game Usage

32. Pilot Study: Game Effectiveness

33. Pilot Study: Intervention Effectiveness
Relationship b/w Self-Efficacy and how much
students play the game
Self-Efficacy & Grit interact in an interesting way

34. Pilot Study: Intervention Effectiveness
F(1,105)=3.81
p=0.05

35. Pilot Study: Intervention Effectiveness
χ2 test, p<.01

36. Pilot Study: Intervention Effectiveness
F(1,176)=3.46
p<0.10

37. Pilot Study: Intervention Effectiveness
F(1,176)=0.05
p>.10

38. Pilot Study: Intervention Effectiveness
F(1,176)=4.50
p<.05

39. Pilot Study: Intervention Effectiveness
F(1,176)=0.03
p>.10

40. Lessons Learned & Limitations
• Promising directions for adapting to cognitive +
non-cognitive factors into games
• Gamification serves a purpose
• Countdown timer adds pressure
• Speed forces rapid comparisons
• “pretty fun for a school game”
• Desire for common game features
• Customization, currency
• Assumption: the game is a valid measure of
number sense

41. References
• Geary DC, Hoard MK, Nugent L, Bailey DH (2013) “Adolescents’
Functional Numeracy Is Predicted by Their School Entry Number
System Knowledge”. PLoS ONE 8(1): e54651.
• Gersten, R., and D. Chard. “Number Sense: Rethinking Arithmetic
Instruction for Students with Mathematical Disabilities.” The Journal of
Special Education 33.1 (1999): 18-28

42. Thank you!
Questions?
rcarlson@carnegielearning.com