Talk at the International 2017 IEEE Requirements Engineering Conference, Lisbon, Portugal

1. A Gamified Tutorial for Learning about
Security Requirements Engineering
Daniel Alami and Fabiano Dalpiaz
Requirements Engineering Lab
Utrecht University, the Netherlands
September 7, 2017

2. 1. The problem
@2017 Fabiano Dalpiaz2

3. 1. The problem
 Ignoring security in SE may lead to security breaches
 Conducting Security in the early phases (RE) is essential!
… but …
 So far, low industrial uptake of existing Security RE methods
 Hypothesis: this may also be a learning problem
@2017 Fabiano Dalpiaz3

4. 1. The learning problem
@2017 Fabiano Dalpiaz4
Traditional education
(about Security RE)
Disengaged students
Too busy professionals

5. 1. The learning problem
@2017 Fabiano Dalpiaz5
Traditional education
(about Security RE)
Disengaged students
Too busy professionals
Security reqs?
Hmm… nah!

6. 1. Research about the learning problem
@2017 Fabiano Dalpiaz6
 Growing interest in innovative learning techniques
 Dedicated venues: CSEE&T, ICSE SEET, REET
 Techniques: improv, role-playing, project simulations
 Focus on high-level, soft skills
 a few exceptions exist, e.g., Zowghi RE’03
 How about procedural learning (e.g., a modeling method,
a specification language)?

7. 2. Solution overview (spoiler alert!)
@2017 Fabiano Dalpiaz7

8. 2. Solution overview
@2017 Fabiano Dalpiaz8
 STS-Tooltorial: an interactive, gamified tutorial
embedded within a security requirements modeling tool
(STS-Tool)
 Bite-sized lessons
 Self-learning
 Gamification
 Immediate
feedback

9. 2. Solution overview
@2017 Fabiano Dalpiaz9
 Baseline: the STS-ml modeling language used within
STS-Tool
 Why?
 Convenience (co-author)
 Access to the source code
 Learning modeling languages is hard!
 Some details
 Goal-oriented
 Multi-view: social, information, authorization
 For socio-technical systems

10. 3. Design framework
@2017 Fabiano Dalpiaz10

11. 3. Design framework
@2017 Fabiano Dalpiaz11
STS-Tooltorial
Interactive tutorial design (33) Game elements (17)
Game mechanics (12)
Curriculum design

12. 3. Design framework
@2017 Fabiano Dalpiaz12
 Review of the interactive tutorials design field
 Learner centricity
 Effectiveness of video tutorials
 Use of contextual assistance
33 Guidelines for designing interactive tutorials

13. 3. Design framework
@2017 Fabiano Dalpiaz13
 Review of the gamification of learning field
A. 17 game elements consistently used
 Points, Badges, Leaderboards, Levels, Progress bars,Avatars,
Onboarding, …
B. Game mechanics
that combine the
game elements into
a game experience

14. 3. Design framework
@2017 Fabiano Dalpiaz14
 Intended learning outcomes
1. Identify the different activities and deliverables of security RE

15. 3. Design framework
@2017 Fabiano Dalpiaz15
 Intended learning outcomes
1. Identify the different activities and deliverables of security RE
2. Recognize the modeling elements and their meaning in a model
3. Choose the most appropriate elements when creating a model

16. 3. Design framework
@2017 Fabiano Dalpiaz16
 Intended learning outcomes
1. Identify the different activities and deliverables of security RE
2. Recognize the modeling elements and their meaning in a model
3. Choose the most appropriate elements when creating a model
4. Express interest in pursuing further learning in the field of
(socio-technical) SRE

17. 4. STS-Tooltorial explained
@2017 Fabiano Dalpiaz17

18. 4. STS-Tooltorial explained
@2017 Fabiano Dalpiaz18
 Lectures: YouTube videos and text
 After each lecture, a quiz takes place
 Interaction with the models
 Immediate feedback on the quizzes
 Multiple levels of increasing difficulty

19. 4. STS-Tooltorial explained
@2017 Fabiano Dalpiaz19
 Some game elements
 Narrative: science fiction, the learner is a rookie security
requirements analyst hired to protect the Empire’s interests
 Point system + Leaderboard + Rank
 Rewards marketplace
 Time restriction
 Avatar

20. 4. STS-Tooltorial explained
@2017 Fabiano Dalpiaz20

21. 5. Preliminary results
@2017 Fabiano Dalpiaz21

22. 5. Preliminary results
@2017 Fabiano Dalpiaz22
 Conceptual model describing our hypotheses
+
+
+
+

23. 5. Preliminary results
@2017 Fabiano Dalpiaz23
 Evaluation set-up
 Two countries (Spain, Netherlands)
 Two groups (8 postgraduate students, 5 IT professionals)
 Data gathering
 Post-test survey
 Analytics module of the interactive platform
 Follow-up interviews

24. 5. Preliminary results
@2017 Fabiano Dalpiaz24
 Evaluation set-up
 Two countries (Spain, Netherlands)
 Two groups (8 postgraduate students, 5 IT professionals)
 Data gathering
 Post-test survey
 Analytics module of the interactive platform
 Follow-up interviews

25. 5. Preliminary results
@2017 Fabiano Dalpiaz25
 H1: Learners exposed to a gamified interactive tutorial
will report high levels of engagement
Above-average engagement results (4.5 out of 7), superior
to other similar studies (Lombriser et al., REFSQ 2016)
Tentatively accepted

26. 5. Preliminary results
@2017 Fabiano Dalpiaz26
 H2: The higher the engagement of learning during
their participation in the gamified tutorial,the higher
their performance will be
On average, 80-100% tutorial completion rate
Quiz accuracy 48% (students), 53% (professionals)
Cannot be tested (no baseline)

27. 5. Preliminary results
@2017 Fabiano Dalpiaz27
 H3: Learners exposed to a gamified interactive tutorial
will report a positive satisfaction towards the use of
the system
SUS score = 80,“A” Platform (90th percentile)
Tentatively accepted

28. 5. Preliminary results
@2017 Fabiano Dalpiaz28
 H4: Deploying a gamified interactive tutorial positively
influences intention to learn more about SRE and the
STS Method
Neutral-to-low intention to learn more
Not accepted

29. 6. Discussion and outlook
@2017 Fabiano Dalpiaz29

30. 6. Discussion and outlook
 A first-of-a-kind attempt
 Positive results about engagement and completion rate
 Issues and open avenues
 The modeling language was hard to learn quickly
 We tested only the social view of STS-ml
 Controlled experiments are needed!
 Replicate with other languages and/or SRE approaches?
@2017 Fabiano Dalpiaz30

31. Important dates
 Abstracts September 25, 2017
 Papers October 2, 2017
 Conference March 19-22, 2018
www.refsq.org