Presentation on my research given at the SERA conference 2012 at University of the West of Scotland, Ayr.

1. Evaluation of Computer Games
Developed by Primary School
Children to Gauge Understanding
of Programming Concepts
Amanda Wilson
Prof. Thomas Connolly
Dr Thomas Hainey
amanda.wilson@uws.ac.uk
thomas.connolly@uws.ac.uk
thomas.hainey@uws.ac.uk

2. Presentation structure
• Introduction
• Programming for Children
• Games-based construction
• Scratch
• Methods
• Results
• Discussion
• Further Work

3. Introduction
• New Curriculum in Scottish Schools
• New Computing Science Strand
• GBL encouraged
• Lack of Empirical evidence of GBL in schools

4. Programming for children
• Programming can be taught from an early age.
• Variety of languages aimed at the novice user
such as: Alice, Toontalk, Starlogo, Etoys, Scratch
and Kodu.
• Initiatives such as after school clubs also run to
introduce children to programming.

5. Games-based Construction
• Game construction is one of 3 ways to introduce
games-based learning (Van Eck, 2006).
• Commercial off the Shelf games (COTS)
• Using games specifically created for education
• Use of game construction as an approach to GBL
• Game construction is relatively unexplored within
the primary classroom (Baytak and Land 2010).
• Applications such as Scratch make games
construction more accessible to children.

6. Scratch
• A simple game creation platform using a visual
programming language.
•Developed originally for use in after school computer
clubhouses

7. Scratch

8. Objectives
• Develop coding scheme for Scratch games
• To evaluate games made by primary school
children using Scratch

9. Methods
• 60 children aged between 8 and 11 from 3 classes
(Primary 4, 5/6 and 6/7) in one school participated
• They undertook lessons in pairs (or groups of 3)
• 8 one hour lessons were delivered
▫ Lesson 1 – Introduction to Scratch
▫ Lessons 2-4 – Creating a simple maze game
▫ Lessons 5-8 – Extending maze game/Creating new
game

10. Methods
• A game coding scheme was adapted from Denner,
Werner and Ortiz (2011) and refined based on the
programming concepts that can be learned with Scratch
• The coding was split into 3 main categories and 22
subcategories
▫ Programming Concepts
▫ Code organisation
▫ Designing for Usability
• Each game was coded for the presence of each element
(either 0/1) or in some cases the extent to which that
element was used within the categories using a range
from either 0-2 or 0-3.

11. Methods – Game Coding
• Programming Concepts category focused on the concepts
that Scratch supports that include:
▫ Sequence
▫ Iteration
▫ Variables
▫ Conditional Statements
▫ Lists
▫ Event handling
▫ Threads
▫ Coordination and Synchronisation
▫ Keyboard Input
▫ Random Numbers
▫ Boolean Logic
▫ Dynamic Interaction
▫ User Interface Design

12. Methods – Game Coding
• Code organisation category:
▫ Extraneous blocks
▫ Sprite names (the default is overridden).
▫ Variable names
• Designing for usability category:
▫ Functionality
▫ Goal
▫ Sprite customisation
▫ Stage customisation
▫ Instructions clear
▫ Game originality

13. Results
• During the study 29 games were created by the
groups of children
• The mean game score was 48% and a Kruskal-Wallis
one way analysis of variance test showed no
significant difference in game scores between class
groups (χ 2 = 0.072, p < 0.965) or between gender
groupings (χ 2 = 0.483, p < 0.785).
Class Stick with
maze game
Adapt maze
game (change
background
adapt game)
Adapt maze game
(change
background
adapt game to
two player)
Create new game
(come up with
another idea other
than maze game)
P4 0 3 4 2
P5/6 3 1 0 6
P6/7 2 2 0 6

14. Results
• The games varied in their complexity with over 90%
of them using the keyboard or mouse to control the
game with others either not having been completed
or having the user answer questions instead.
• Kruskal-Wallis tests showed no significant
difference in concepts used between class groups (χ 2
= 0.176, p < 0.916) or between gender groupings (χ 2
= 0.472, p < 0.790). However, Mann-Whitney U test
showed a significant difference between the
concepts used in maze-based games compared to
the original games made by all classes (Z=-2.535, p
< 0.010).

15. Results
Programming Concepts
% of games including programming
concepts
Sequence 93
Event Handling 90
Conditional Statements 86
Threads 83
Variables 72
Coordination and Synchronisation 72
Iteration 55
Keyboard Input 7
Random Numbers 3

16. Results
• 21% of the games included extraneous blocks.
• 72% of the games included meaningful variable
names most games only included a timer,
however some games also implemented a
scoring system as well.
• Only 3% of games had changed the default sprite
name.

17. Results
• Out of the 29 games only 1 had no functionality.
The others had varying degrees with 28% of the
games being fully functional.
Designing for Usability % of games
Functionality 97
Sprite customisation 97
Stage customisation 93
Clear Instructions 86
Game originality 83
Goal 59

18. Game Example 1
Adapted maze game from Primary 5/6 class

19. Game Example 1
• The sprite in the game is controlled by the arrow
keys. Conditions are set that if the sprite touches
the green walls it will bounce off them while the
timer counts down how many seconds left the
user has.

20. Game Example 2
2 Player adapted maze game from the Primary 4 class

21. Game Example 2
• The sprites in the game are controlled by the
arrow keys for player 1 and player 2 used the
w,a,s,d keys.
• There were 2 variables within the game and
while the timer worked the scoring wasn’t
working correctly.

22. Discussion
• Few studies have been undertaken to look at
learning programming concepts through game
construction for children.
• These studies have shown that children are able
to learn programming concepts through game
construction although these have been
conducted mainly after school and little is
known within the classroom setting.

23. Discussion
• The 8 lessons were a basic introduction to game
making with Scratch.
• Most children were successful in creating their
own game.
• With those in the Primary 4 class preferring to
adapt the maze game rather than create their
own.
• 60% of each Primary 5/6 and 6/7 class opted to
create their own game.

24. Discussion
• The concepts most commonly used by the children in their
games were similar to those found in Maloney et al (2008)
▫ Key handling
▫ Iteration
▫ Conditional statements
• Gender groupings did not have a significant effect on the
games created, however the Primary 5/6 class which consisted
of mixed gender groups did have the highest mean score as
well as the most functional games of all 3 classes.
• Similar to the results of Baytak and Land (2011) this study
shows that given a short timeframe children were able to
make progress with Scratch show advances in their
programming skills.

25. Future Work
• Work with more classes (from Primary 4 to
Primary 7) in other schools within Glasgow to
attain further empirical results to produce more
statistically significant evidence.
• Refinement of the instrument of evaluation
through a series of pilot studies.

26. Thank you