2. Games as an educational medium to teach
● Growing research in field of
educational security games to teach
hard to grasp topics in an engaging
way
● These games have potential to attract
wide audience ranging from children
to adults (Druin, 2004; Cone et al.,
2007; Olano et al., 2014)
3. Existing security games
Existing solutions are
great but do not solve
our problem!
Games for absolute beginners Games for Professionals
How do we target interested
parties at an early age?
4. Why firewalls?
● To teach security to the next generation
effectively, we need to start somewhere
● Firewalls as a term that young people have
heard before and associate with security
● Certain curiosity about how firewalls
protect your computer
● VMs are a nightmare to configure and
alienate people starting out
5. What is a firewall?
“ A firewall is a device, software, arrangement, or equipment that limits
network access, be it a software layer or a physical box.” (Cheswick et al.,
2003)
“While there are many types of firewalls, all of them work by examining the
traffic passing across them and applying a set of rules to the traffic to
determine if each packet will be allowed through (accept) or discarded
(drop).” (Gouda & Liu, 2007)
6. Common professional-level firewall errors
● Rule ordering, hard to follow rules, and
keeping rules to a minimum
● Allowing “any service” inbound and outbound,
unencrypted access and using implicit rules
with regard to TCP, UDP and ICMP
● Conceptual errors
● Rise in automation starts to eliminate typos
and simple ordering errors
Even experienced system administrators still make
mistakes and struggle to grasp certain concepts.
Image: http://arstechnica.co.uk/gadgets/2016/01/numbers-dont-lie-its-time-to-build-your-own-router/
7. Requirements & Design Goals
Existing solutions don’t solve our problem for the identified gap, which led us to
formulate the following design goals:
● Accessible to a general audience
● Anyone should be able to play and gain an improved understanding of what a
firewall is and how it works
● Avoid “attack and defense” terminology - security should be welcoming
● Concepts: chains, rule evaluation order, and default policies
● Extend terminology development to build accurate mental models
9. Level 2 Walkthrough - Instructions
Allow user to
switch
between
screens and
progress
Main
character
Roboto
10.
11. Level 2 Walkthrough - Interactive Game Screens
Error message to
help user diagnose
his mistake
Pressing the play button
will play an animation if
user input is correct
Draggable
building
blocks
Ability to reset current
selection
12.
13.
14. Level 1:
● Intro rule building interface
● Different rules for packets coming
in and going out
Level 2:
● Two rules per chain
● Goal: allow traffic for port 80 and
default policy of drop
Levels 3 - 7:
● Introduce protocols: SSH, FTP,
DNS, SMPT and SIP
Level 8:
● Combines knowledge of services
and ports
● Multiple complex rules for a chain
Level 9:
● Default of ACCEPT
● Block a specific IP
Level 10:
● Asks user to construct a sensible
ruleset freely.
● Hints removed
Level Structure
15. Lab Study
● In-person lab study (1 person remote)
● Screen capture software and notes taken by
researcher
● Pre/post questionnaires
○ Demographics
○ System Usability Scale rating (Post test only),
○ Prior firewall knowledge
○ Firewall terminology
○ Read an iptables command
○ Understanding of the rule building interface
16. Demographics
● 5 participants
● 2 female players and 3 male players
● Average game play was 27 min (ranging
from 13 min to 38 min)
● Mix of computer science and/or security
knowledge
10 people filled out the User Feedback Sheet (SUS)
17. Usability
● Average SUS score for 10 participants: 88.25
○ Considered a high degree of usability
● Good understanding of the rule building
interface, especially after playing the game
● One participant didn’t understand that
objects could be dragged and dropped
without instruction
● Easy recovery from errors and every
participant could complete the game
Understanding of the rule building interface before
playing the game (P4)
18. Results
Prior firewall knowledge
Pre Test Post Test
● Term “firewall” sounded familiar to
all participants
● One participant mentioned that
“firewall protects computers from
hackers”
● 2 participants thought they knew
how a firewall operates
● 3 participants had no idea how a
firewall operates
● Every participant provided an answer
as to how a firewall works
● “A firewall has rules and checks
whether packets match these” -
Participant 2 (no CS or Security
knowledge)
19. Results
Knowledge of firewall terminology
IP Address, Hyper Text Transfer Protocol, DNS, Ports, Chains …
Pre Test Post Test
● Even participants that had computer
security knowledge had trouble
identifying correct terminology
● Inexperienced participant did not
know the term “packet”
● No participant was familiar with
“chains”
● In general, more terminology
questions answered correctly after
playing the game
● 3 participants (all with CS
knowledge) accurately described a
“chain” as a set of rules
● One participant indicated that
“chains” didn’t sound familiar
20. Results
Ability to read an iptables command
iptables -A INPUT -i eth0 -p tcp --port 443 -m state --state NEW,ESTABLISHED -j ACCEPT
Pre Test Post Test
● Only one participant (CS and
Security knowledge) could describe
the different parts of the iptables
command
Example answer P2:
P2: “1. IP address tables, 2. Data Input 3.
/ “
● All participants could explain
somewhat what the iptables
command meant, including the
participant who had no previous
knowledge in the field
Example answer P2:
P2: “Input for port 443 is accepted for
new and established connections”
21. Results
Differentiating Input vs Output rules
Pre Test Post Test
● Only two participants understood the two
terms
● The other three participants did not
relate the terms to firewalls or left
question blank
● Participants could all explain the
difference
● Participant who left field blank in pre
test:
“Input rule specifies which packets are
allowed to enter the system. Output
rule specifies what is allowed to leave
the system.”
22. Conclusion
● First results from evaluation promising
● Participants found the game to be fun, engaging
and educational
● Permission Impossible creates an accessible
introduction for beginners to learn about firewalls
● Addresses gap between interest improving games
and competitive level games
● Participants responded well to positive language:
“helping Roboto”
23. Thank you
Q & A Session
Sibylle Katharina Sehl
MSc Computer Science Graduate from University of Edinburgh
E-mail: sibylle.sehl@googlemail.com
LinkedIn: https://www.linkedin.com/in/sibyllekatharinasehl/
24. Limitations & Future Research
Limitations
● Small sample of users in study
● Research can only serve as first investigation into the field
Future Work
● More complex levels should be developed and build on users’ knowledge
● Design improvements in terms of different screen sizes
● Motivational screens for congratulating the user to keep up engagement
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38. More Screenshots
Level 10: Colour hints removed
Extra building blocks that
aren’t needed for
completion of level
Teaching about
implications of default
policy of accept
