Introductory Computer Science course, Procedurally generated problem sets, Questions based on level and prior performance Earn points by answering question correctly, No punishment for incorrect answers 100% Automated grading. Keep students in flow, It’s OK to fail, Player is in control, Competency, Autonomy, Relatedness, Mixed Practice, Scales Well

1. ‘-
1
Dr. Jesse Hartloff
hartloff@buffalo.edu
USING GAME ELEMENTS TO
TEACH COMPUTER SCIENCE

2. ‘-
2
• Introductory Computer Science course
• Procedurally generated problem sets
• Questions based on level and prior
performance
• Earn points by answering question
correctly
• No punishment for incorrect
answers
• 100% Automated grading
The App

3. ‘-
3
BUT FIRST…
How did we get here?

4. ‘-
4
My [Gaming] Background

5. ‘-
5
My [Gaming] Background

6. ‘-
6
My [Gaming] Background

7. ‘-
7
Can education be as compelling as games?

8. ‘-
8
THE PROBLEM
What are we trying to solve?

9. ‘-
9
• Students are not motivated to learn the material
• Many come to Computer Science for the jobs
• Some non-majors who are “forced” to take our intro
course
• Computer Science can be dry
Low Motivation

10. ‘-
10
• There is very little memorization
• Mostly at the level of Apply on
Bloom’s Taxonomy with implicit
Understanding
• Many students are not prepared
for this in their first semester
• Common complaint
• This is way too hard for a 100-
level course
Difficulty – Application of Material

11. ‘-
11
• Most topics in the course require the students to understand
most of the previous topics
• Ex. You cannot apply data structures in a program without
understanding variables, expressions, and functions
• Similar to a math classes
• You can’t learn algebra without understanding
arithmetic
• Students like to skip hard topics
Difficulty – Cumulative Material

12. ‘-
12
• Typically 4-7% are caught are convicted
each semester
• Presumably many more don’t get caught
• Difficult to access the full scale of this
problem
Academic Integrity

13. ‘-
13
• This Spring there were 350 student in the intro course
• First semester using game elements
• Last Fall: 650 students
• This Fall: 800 students expected across 4 sections
• Fall Support
• Co-taught by 2 faculty members (Dr. Carl Alphonce
and myself)
• 30 undergraduate teaching assistants working 10
hours/week
Scale

14. ‘-
14
POTENTIAL
SOLUTION?
Badges & Leaderboards

15. ‘-
15
• Can be motivating
• Ex. Xbox achievements can be great
• They provide new challenges and goals in games
that you already want to play
• Adding badges to an app does not always motivate a
user
• Should be added with a specific purpose to be effective
Badges
Badges in the Audible mobile app

16. ‘-
16
• I have tried this in upper level course
• Only motivates the top students (~5-
10%)
• Does nothing for the strugling
students
• In some cases it can
demoralize them
• Majority of students just want to get
through the assignment and shift
their focus
Leaderboards
Runtime leaderboard in a 300-level algorithms course

17. ‘-
17
• Where are they?
• Not in my games.
Badges, Leaderboards, etc.

18. ‘-
18
A BETTER
APPROACH
What can we learn from games

19. ‘-
19
• Flow
• Challenging, but not frustrating
• It’s OK to fail
• Try again?
• The game will wait for us
• The player has complete control
• A failure feels like the players shortcoming
• Can overcome with practice
Common Features of Compelling Games

20. ‘-
20
Flow
image: whats-in-a-game.com

21. ‘-
21
• Flow
• Player progresses as far as they are able
• Game then “waits” for the player as the practice with
the game mechanics
• With enough practice the player can overcome the
next obstacle
• OK to fail
• Worst Case: Game over and restart from beginning
• Practice easier levels
• Control
• Responsive control over your sprite
Single-Player Games

22. ‘-
22
• Flow
• Depends on opponent
• Evenly matched opponents can experience flow
• Unbalanced matches diminish fun
• OK to fail
• Worst Case: Find a new opponent
• Control
• Limited by luck
• Over time the more skilled players will win more often
Multi-Player Games

23. ‘-
23
A BETTER
APPROACH
What can we learn from Science?

24. ‘-
24
• A leading theory of motivation
• Motivation depends on:
• Competency
• Autonomy
• Relatedness
Self-Determination Theory
Deci, E. L., & Ryan, R. M. (2008). Self-determination theory: A
macrotheory of human motivation, development, and health.
Canadian Psychology/Psychologie canadienne, 49(3), 182-185.
Núñez, J. L., & León, J. (2015). Autonomy support in the classroom:
A review from self-determination theory. European Psychologist,
20(4), 275-283.

25. ‘-
25
• Competency
• Progressing towards mastery in a task or skill
• Autonomy
• In control
• Can make meaningful decisions
• Relatedness
• Connection to other people
• Does the action matter to others?
Self-Determination Theory

26. ‘-
26
• Mixing concepts together while studying improves long-term memory
• As opposed to practicing 1 topic at a time
Mixed Practice
Shea, J. B., & Morgan, R. L. (1979). Contextual interference effects on the
acquisition, retention, and transfer of a motor skill. Journal of Experimental
Psychology: Human Learning and Memory, 5(2), 179-187.
Hatala, R.M., Brooks, L.R., & Norman, G. (2003). Practice makes perfect: the critical
role of mixed practice in the acquisition of ECG interpretation skills. Advances in
health sciences education: theory and practice, 8(1), 17-26.

27. ‘-
28
 Keep students in flow
 It’s OK to fail
 Player is in control
 Competency
 Autonomy
 Relatedness
 Mixed Practice
 Scales Well
Compiled Checklist of Goals

28. ‘-
29
 Keep students in flow / Competency
 Player is in control / Autonomy
 Relatedness
 It’s OK to fail
 Mixed Practice
 Scales Well
Condensed Checklist of Goals

29. ‘-
30
THE SYSTEM

30. ‘-
31
• Introductory Computer Science course
• Procedurally generated problem sets
• Questions based on level and prior
performance
• Earn points by answering question
correctly
• No punishment for incorrect
answers
• 100% Automated grading
The App

31. ‘-
32
 Keep students in flow / Competency
 Player is in control / Autonomy
 Relatedness
 It’s OK to fail
 Mixed Practice
 Scales Well
Condensed Checklist of Goals
Problem Set Performance?

32. ‘-
33
• Questions depend on their prior performance in the course
• Can’t progress until they practice/study enough to complete the current
question types
• Visual progress bar and level to show their accomplishment thus far
Keep students in flow / Competency

33. ‘-
34
• Player has limited control
• Pros
• Can decide when to use consumable multipliers
• Can complete problem sets any time they’d like
• Can complete the question with any approach they’d like as long
as their code is correct
• Cons
• No choice in the question types or topics
Player is in control / Autonomy

34. ‘-
35
Relatedness

35. ‘-
36
• No punishment for getting a question wrong
• Only reward for answering a question correctly
• If a student can’t answer a question correctly they will get another similar
question
• No game overs
• Try as many problem sets as they’d like
It’s OK to Fail

36. ‘-
37
• Previous questions will come back in later levels
• Especially true if they skip question types
• Level 13 is 6000 point of mixed practice with no new concepts or
question types
Mixed Practice

37. ‘-
38
• 100% Automated
• Scales with ease to our large class sizes
Scales Well

38. ‘-
39
 Keep students in flow / Competency
 Player is in control / Autonomy
 Relatedness
 It’s OK to fail
 Mixed Practice
 Scales Well
Results
✓
✓
✓
✓
−

39. ‘-
40
• Improve autonomy with “skill trees”
• Students can practice questions for any topic as long as they’ve
completed the required prereq topics
• Allows student to choose between different branches in the
course
• Students can switch between branches at any time
• Improved game loop
• Current loop is slow and tedious
• New loop will all be in browser on a single page app
Next Version

40. ‘-
41
MISTAKES
Learning Experiences

41. ‘-
42
• Must have.
• No deadlines last semester
• Students procastinated until the last few weeks
• The course is not [yet] compelling enough to get them to start early
• Difficult to tell if the system was effective when so many students didn’t
even explore it until it was too late
• Will have several deadlines/check points next semester
Deadlines

42. ‘-
43
• Make sure all student understand the mechanics of the system early in
the course
• I had very basic questions very late in the semester (Ex. How do I earn
multipliers)
• One very early deadline could resolve this to make sure everyone
completes a few problem sets
Communication

43. ‘-
44
FUTURE TECH
and random ideas

44. ‘-
45
• Very little of the system, and this talk, is specifically about Computer
Science
• Adding other subject can be as simple as adding questions for that
subject
Beyond Computer Science

45. ‘-
46
• Predict exactly what question will keep the student in flow
• Can personalize the theme of questions to each student based on their
provided interest, or determine their interests by what themes they
answer correct more often
• Warning: Ethics must be considered when adding AI
Artificial Intelligence

46. ‘-
47
• Add a story to the course where student progress by completing sets of
questions
• Occasionally they get story questions that advace the narrative
• Some story questions can have multiple answers that branch the story
depending on the decision/code of the student
• Implemmented story in Spring 17’
Story
Art by Angus Lam

47. ‘-
48
QUESTIONS?

48. ‘-
49
• We have data (timestamps, scores, activity, submissions)
• New system will collect even more data
Data Analysis

49. ‘-
50
• What is the primary loop the students repeat throughout the course
• Check out a problem set, complete the problem set, submit for score and
feedback, check out another problem set
• Passing students went through this game loop on average 73 times
Game Loop

50. ‘-
53
• Apply to problem sets and multiply the score of all correct answers
• Multipliers stack multiplicitivly
• Students doing very well (bored) can quickly progress to a
challenge
Multipliers

51. ‘-
54
• Magnitude: 1.5x
• Purpose: Incentivize students to do a problem set every day
• Result: Less than effective
First Problem Set of the Day Multiplier

52. ‘-
55
• Magnitude: 1.5x
• Purpose: Incentivize students to do a problem set every day
• Result: Less than effective
First Problem Set of the Day Multiplier

53. ‘-
56
• Magnitude: 1.5x
• Purpose: Incentivize students to carefully attempt all 5 questions
• Result: Effective, but added much frustration for small mistakes. Gave
top students an extra challenge. Got students to look over their work
more carefully. Don’t just submit and hope
All Question Correct Multiplier

54. ‘-
57
• Magnitude: 1.2x and 1.5x
• Purpose: Incentivize students to complete goals that do not fit into the
grading structure of the course
• Result: Meh
Consumable Multipliers

55. ‘-
58