Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish
•Download as PPT, PDF•
0 likes•280 views
Slides from 2007 on the design and evaluation of Anti-Phishing Phil, a game that teaches people how to avoid phishing attacks. In this paper we describe the design and evaluation of Anti-Phishing Phil, an online game that teaches users good habits to help them avoid phishing attacks. We used learning science principles to design and iteratively refine the game. We evaluated the game through a user study: participants were tested on their ability to identify fraudulent web sites before and after spending 15 minutes engaged in one of three anti-phishing training activities (playing the game, reading an anti-phishing tutorial we created based on the game, or reading existing online training materials). We found that the participants who played the game were better able to identify fraudulent web sites compared to the participants in other conditions. We attribute these effects to both the content of the training messages presented in the game as well as the presentation of these materials in an interactive game format. Our results confirm that games can be an effective way of educating people about phishing and other security attacks. Authors are Steve Sheng, Bryant Magnien, Ponnurangam Kumaraguru, Alessandro Acquisti, Lorrie Cranor, Jason Hong, and Elizabeth Nunge
Recommended
More Related Content
Similar to Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish
Similar to Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish (20)
Recently uploaded
Recently uploaded (20)
Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish
- 1. CMU Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ Anti-Phishing Phil The Design and Evaluation of a Game That Teaches People Not to Fall for Phish S. Sheng, B. Maginien, P. Kumaraguru, A. Acquisti, L. Cranor, J. Hong, E. Nunge
- 2. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Online game • http://cups.cs.cmu.edu/antiphishing_phil/ Teaches people how to protect themselves from phishing attacks • Identify phishing URLs • Use web browser cues • Find legitimate sites with search engines Anti-Phishing Phil
- 3. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Why a game? Security is a secondary task Learning by doing Fun and engaging Better strategies
- 10. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ More about the game Four rounds • Increasing difficulty • Two minutes in each round Eight URL “worms” in each round • Four phishing and four legitimate URLs • Users must correctly identify 6 out of 8 URLs to advance
- 12. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ User Study Test participants’ ability to identify phishing web sites before and after training • 10 URLs before training, 10 after, randomized • Up to 15 minutes of training Training conditions: • Web-based phishing education • Tutorial • Game 14 participants in each condition • Screened out security experts • Younger, college students
- 13. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
- 14. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
- 15. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Falling for Phishing 0.43 0.34 0.12 0.19 0.17 0.38 0 0.1 0.2 0.3 0.4 0.5 Existing training materials Tutorial Game FalseNegativeRate Pre test Post test
- 16. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Misidentifying Legitimate Sites 0.30 0.27 0.30 0.41 0.21 0.14 0 0.1 0.2 0.3 0.4 0.5 Existing training material Tutorial Game FalsePositiveRate Pre test Post test
- 17. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Results Game group had the best performance overall Game group had fewest false positives No significant difference in false negatives among the three groups
- 18. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Field Study
- 20. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Study Set-up Test participants’ ability to identify phishing web sites after training and the ability to retain the knowledge • 6 URL quiz before training, after training, one week later Conditions: • Control • Game Completed training • 423 in training group • 292 in control group
- 21. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Preliminary Results 31% 60% 92% 75% 81% 93% 0% 20% 40% 60% 80% 100% Novice Intermediate Expert Pretest Post test
- 22. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Comments
- 23. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Press
- 24. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Deployment We’ve released Phil under a Creative Commons non-commercial license Over the past few weeks we’ve been contacted by several banks, retailers, other companies, and government agencies who are interested in using Phil in their employee training programs • Can’t get employees to read security memos, but think they will be willing to play a game and learn something We’re working on setting up a commercial licensing program, customized versions
- 25. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Portuguese Version
- 26. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Future Plans Analyze field study results to understand how game can be further improved Continue to update game and use data from public usage to evaluate and improve Consider adding new modules to teach different skills or reinforce skills through alternate approaches Consider special versions for kids, elderly, specific brands, etc.
- 27. • CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ Acknowledgements Members of Supporting Trust Decision research group Members of CUPS Lab
- 28. CMU Usable Privacy and Security Laboratory http://cups.cs.cmu.edu/ Play Anti-Phishing Phil: http://cups.cs.cmu.edu/antiphishing_phil/
Editor's Notes
- Good afternoon everyone, I am Steve Sheng from Carnegie Mellon University, I am part of the CUPS lab at CMU. Today, I will be talking about some of the work that we did at CUPS lab in order to find solutions to train users about phishing attacks. The work that I will be presenting today was jointly done with Bryant Maginien, Ponguru Kumaragu, Alessandro Acquisti, Lorrie Cranor, Jason Hong and Elizabeth Nunge.
- Educating user have some constraints, The first constraint is that security is a secondary task, people are not visiting a website to look at its security features, they go to the website to complete transactions. Another constraint is people like learning by doing, they don’t like to sit down and read training materials. Education is more effective when users learn by doing rather than by learning the classroom instructions.
- The scene: is sea, we have a small fish called Phil, her job is to eat all the worms.
- So today, Phil swim by a worm, the worm is identified by a URL. A good worm is a legitimate URL, whereas a bad worm is a bait dropped by the phishers.
- Phil needs to score 6 / 8 to move on to the next rounds, and the end of the round, phil got a chance to reflect what he missed.
- In between rounds, we also have short tutorials to teach Phil better strategies to identify phishing. In this example, Phil’s father teaches Phil how to use a search engine.
- STUDY WAS A THINK ALOUD STUDY that lasts 45 - 60 MINS.. WE CAREFULLY RECRUITED NON-EXPERTS USING THREE SPECIFIC QUESTIONS - THE DEIFNITION OF NON EXPERT IS THE SAME AS IN PREVIOUS STUDY THAT I SAID. It aimed at testing the participants’ ability to identify phishing websites. We presented them 10 websites before training, followed by a 15 minute break where users perform one of the three tasks: they read webased phishing education, they read the game tutorial, or they played the game. Users are randomly assigned in each of the conditions. There are fourteen non-expert participants in each condition, for a total of 42 participants.
- All of them are statistical significant, there is no statistical difference between them in Either pre test or post test.
- There are statistically different.
- To summarize, there are -- No significant difference in false negatives among the three groups - Game group performed best in false positives - Game condition performed best in total correctness Effect between the tutorial and the game conditions not statistically significant. The next question we want to answer, is that is the increase in performance due to learning or raising awareness.
- http://www.pcworld.com/article/id,137868-c,cybercrime/article.html http://www.news.com/8301-10784_3-9787549-7.html?tag=nefd.only http://www.cbc.ca/technology/story/2007/09/26/phil-phish.html http://www.pcpro.co.uk/news/126386/phishers-caught-hook-line-and-sinker.html http://www.businessweek.com/the_thread/blogspotting/archives/2007/09/play_with_anti-.html