Jason Hong
Carnegie Mellon University
Wombat Security Technologies
Teaching Johnny
Not to Fall for Phish
What are Effective Ways of Teaching
People not to Fall for Phish?
PhishGuru Embedded Training
• Use embedded training to teach people how
to avoid phishing in regular use of email
– People...
Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information
Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information
Please login and enter yo...
Tells people why they are
seeing this message, uses
engaging character
Tells people why they are
seeing this message, uses...
Tells a story about what
happened and what the
risks are
Tells a story about what
happened and what the
risks are
Gives concrete examples of
how to protect oneself
Gives concrete examples of
how to protect oneself
Explains how criminals conduct
phishing attacks
Explains how criminals conduct
phishing attacks
Series of User Studies
Studies Results
Lab study I • Security notices are ineffective
• Users educated with PhishGuru made...
First lab study results
• Are security notices
effective?
– Ineffective for training
• Is embedded training
effective?
– E...
Second lab study results
• Can people retain what they learned?
– Users educated with PhishGuru
retained knowledge after s...
Real world study: Portuguese ISP
• Does PhishGuru training extend to real world?
– Did reduce rate of falling for phishing...
Real world study: CMU
• Replicate previous study at larger scale
• Investigate retention after 1 week, 2 weeks,
and 4 week...
Study design
• Sent email to all CMU students, faculty and
staff to recruit participants (opt-in)
• 515 participants in th...
Our Simulated Spear Phish
URL is not hiddenURL is not hidden
Plain text email
without graphics
Plain text email
without gr...
Our Simulated Phishing Website
http://andrewwebmail.org/password/change.htm?ID=9009http://andrewwebmail.org/password/chang...
Our Simulated Phishing Website
http://andrewwebmail.org/password/thankyou.html?ID=9009http://andrewwebmail.org/password/th...
Effect of PhishGuru Training
Condition N % who
clicked on
Day 0
% who
clicked on
Day 28
Control 172 52.3 44.2
Trained 343 ...
Results conditioned on participants
who clicked on day 0
TestTest
Results conditioned on participants
who clicked on day 0
Trained
participants
less likely to
fall for phish
Trained
partic...
Results conditioned on participants
who clicked on day 0 and day 14
Two-train participants less likely
than one-train part...
Results conditioned on participants
who clicked on day 0 and day 14
Two-train participants less likely
than one-train part...
Does PhishGuru Affect Clicking
on Legitimate Emails?
Condition N Day 0 Day 7 Day 28
Clicked % Clicked % Clicked
%
Control ...
Students Most Vulnerable
• Students significantly more likely to fall for
phish than staff before training
• No significan...
Most Participants Liked PhishGuru,
Wanted More
• 280 post study responses
• 80% recommended that CMU continue
PhishGuru tr...
Summary
• People trained with PhishGuru far less likely
to click on phishing links than not trained
• People retained trai...
For More Information
• Forthcoming SOUPS 2009 paper
• White paper on Wombat Security web site
• PhishGuru commercialized
b...
Acknowledgments
• Supporting Trust Decisions group
• CyLab Usable Privacy and Security Lab
• CMU’s Information Security Of...
Study schedule
Day of the
study
Control One training
message
Two training
messages
Day 0 Test and real Train and real Trai...
Why is Teaching People Hard?
Why is Teaching People Hard?
• Problems
– Existing materials good, but could be better
• Not many opportunities for testin...
Legitimate emails
No difference between the three
conditions on day 0, 7, and 28
No difference between the three
condition...
Teaching Johnny not to Fall for Phish, at APWG CeCOS 2009
Teaching Johnny not to Fall for Phish, at APWG CeCOS 2009
Teaching Johnny not to Fall for Phish, at APWG CeCOS 2009
Upcoming SlideShare
Loading in …5
×

Teaching Johnny not to Fall for Phish, at APWG CeCOS 2009

104 views
89 views

Published on

An overview of our group's work on teaching people not to fall for phishing attacks, using simulated phish. The summary is that simulated phish work surprisingly well, in terms of learning and retention.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
104
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • THE USER WILL SEE THIS INTERVENTION… WHICH TELLS THEM HOW TO AVOID FALLING FROM PHISHING EMAILS… I WILL DESCRIBE IN DETAIL WHAT INFORMATION IS IN THIS INTERVENTION IN A COUPLE OF MINUTES. You have the printout of this intervention…
  • THE USER WILL SEE THIS INTERVENTION… WHICH TELLS THEM HOW TO AVOID FALLING FROM PHISHING EMAILS… I WILL DESCRIBE IN DETAIL WHAT INFORMATION IS IN THIS INTERVENTION IN A COUPLE OF MINUTES. You have the printout of this intervention…
  • THE USER WILL SEE THIS INTERVENTION… WHICH TELLS THEM HOW TO AVOID FALLING FROM PHISHING EMAILS… I WILL DESCRIBE IN DETAIL WHAT INFORMATION IS IN THIS INTERVENTION IN A COUPLE OF MINUTES. You have the printout of this intervention…
  • THE USER WILL SEE THIS INTERVENTION… WHICH TELLS THEM HOW TO AVOID FALLING FROM PHISHING EMAILS… I WILL DESCRIBE IN DETAIL WHAT INFORMATION IS IN THIS INTERVENTION IN A COUPLE OF MINUTES. You have the printout of this intervention…
  • THE USER WILL SEE THIS INTERVENTION… WHICH TELLS THEM HOW TO AVOID FALLING FROM PHISHING EMAILS… I WILL DESCRIBE IN DETAIL WHAT INFORMATION IS IN THIS INTERVENTION IN A COUPLE OF MINUTES. You have the printout of this intervention…
  • THE USER WILL SEE THIS INTERVENTION… WHICH TELLS THEM HOW TO AVOID FALLING FROM PHISHING EMAILS… I WILL DESCRIBE IN DETAIL WHAT INFORMATION IS IN THIS INTERVENTION IN A COUPLE OF MINUTES. You have the printout of this intervention…
  • Mention why are these questions important? We showed that embedded works when u test participants immediately, but we don’t know how they will perform after 7 days? BECAUSE USERS HAVE REMEMBER WHAT YOU TEACH FOR SOME TIME…Knowledge retention (KR): The ability to apply the knowledge gained after a time period ANOTHER QUESTION IS, WHETHER USERS HAVE TO FALL FOR PHISHING TO GET TRAINED, THIS IS TO ADDRESS THE DELIVERY CHANNEL QUESTION… IF IT TURNS OUT TO BE SO, THEN WE DON’T HAVE TO MAKE THEM FALL FOR PHISHING.. We also don’t know how they will perform in a different situation…. GIVEN THE EARLIER RESEARCH RESULTS THAT USERS DON’T GENARALIZE, WE WANTED TO SEE CAN USERS TRANSFER… Knowledge transfer (KT): The ability to transfer the knowledge gained from one situation to another situation
  • TO ADDRESS SOME OF THE LIMITATIONS IN THIS STUDY, I AM CURRENTLY DOING THIS EXCITING STUDY AMONG CMU STUDENTS/FACULTY/STAFF WHERE I AM PHISHING THEM FOR THE LAST 4 WEEKS… I WAS INTERESTED IN STUDYING LONG TERM RETENTION .. MORE THAN 1 WEEK.. SO IN THIS STUDY WE ARE STUDYING 4 WEEK RETENTION.. IN PREVIOUS STUDY WE STUDIED 1 TRAINING MATERIAL… HERE WE ARE STUDYING 2 MESSAGES… THIS STUDY IS REALLY IN THE WILD AND WE ARE COLLECTING LOT OF DATA…. I M STILL IN THE DATA COLLECTION MODE IN A FEW WEEKS, I SHOULD HAVE SOME RESULTS FROM THIS STUDY…
  • TO ADDRESS SOME OF THE LIMITATIONS IN THIS STUDY, I AM CURRENTLY DOING THIS EXCITING STUDY AMONG CMU STUDENTS/FACULTY/STAFF WHERE I AM PHISHING THEM FOR THE LAST 4 WEEKS… I WAS INTERESTED IN STUDYING LONG TERM RETENTION .. MORE THAN 1 WEEK.. SO IN THIS STUDY WE ARE STUDYING 4 WEEK RETENTION.. IN PREVIOUS STUDY WE STUDIED 1 TRAINING MATERIAL… HERE WE ARE STUDYING 2 MESSAGES… THIS STUDY IS REALLY IN THE WILD AND WE ARE COLLECTING LOT OF DATA…. I M STILL IN THE DATA COLLECTION MODE IN A FEW WEEKS, I SHOULD HAVE SOME RESULTS FROM THIS STUDY…
  • Spear phishing emails are targetted phishing emails COLLECTING VARIETY OF INFORMATION (HR, COMPLAINTS THAT ARE BEING LOGGED TO HELP CENTERS AND ISO) COUNTERBALANCING THE EMAILS COLLECTING DATA FOR LEGITIMATE EMAILS TO SEE WHETHER TRAIING INCREASES CONCERN
  • Some email clients don’t show the html and so we used this way
  • The idea in this slide is to show that training conditions did better than control conditions and it was significantdifferenc… There is an improvement of 50% among people in PhihsGuru training
  • Graph is people who clicked on day 0 (trained in the training conditions). People in the training conditions retained knowledge until day 28
  • Graph is people who clicked on day 0 (trained in the training conditions). People in the training conditions retained knowledge until day 28
  • People who clicked on day 0 and day 14. This is to find how participants in two training conditions compare with participants in one training condition. Shows a significant difference on day 16, day 21 (next slide)
  • People who clicked on day 0 and day 14. This is to find how participants in two training conditions compare with participants in one training condition. Shows a significant difference on day 16, day 21 (next slide)
  • Similar effect for gave information too
  • WALK THROUGH THE TABLE POINTING THE DIFFERENCES DEFINE REAL, TEST, AND TRAIN
  • Similar effect for gave information too
  • Teaching Johnny not to Fall for Phish, at APWG CeCOS 2009

    1. 1. Jason Hong Carnegie Mellon University Wombat Security Technologies Teaching Johnny Not to Fall for Phish
    2. 2. What are Effective Ways of Teaching People not to Fall for Phish?
    3. 3. PhishGuru Embedded Training • Use embedded training to teach people how to avoid phishing in regular use of email – People get simulated phishing email from good guys – Teach how to protect self in engaging format – Applies learning science for training • Motivating users – “teachable moment” • Started as research at CMU, product by Wombat Security Technologies
    4. 4. Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information
    5. 5. Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information Please login and enter your informationPlease login and enter your information http://www.amazon.com/exec/obidos/sign-in.htmlhttp://www.amazon.com/exec/obidos/sign-in.html
    6. 6. Tells people why they are seeing this message, uses engaging character Tells people why they are seeing this message, uses engaging character
    7. 7. Tells a story about what happened and what the risks are Tells a story about what happened and what the risks are
    8. 8. Gives concrete examples of how to protect oneself Gives concrete examples of how to protect oneself
    9. 9. Explains how criminals conduct phishing attacks Explains how criminals conduct phishing attacks
    10. 10. Series of User Studies Studies Results Lab study I • Security notices are ineffective • Users educated with PhishGuru made better decisions Lab study II • Users in embedded condition retain and transfer knowledge more effectively than other conditions even after 7 days Real-world study I • PhishGuru is effective in training people in the real world • Trained participants retained knowledge after 7 days of training Real-world study II • People trained with PhishGuru were less likely to click on phishing links than those not trained • People retained their training for 28 days • Two training messages are better than one • PhishGuru training does not make people less likely to click on legitimate links
    11. 11. First lab study results • Are security notices effective? – Ineffective for training • Is embedded training effective? – Embedded training condition made better decisions than those sent security notices Kumaraguru, P., Rhee, Y., Acquisti, A., Cranor, L. F., Hong, J., and Nunge, E. Protecting people from phishing: the design and evaluation of an embedded training email system. CHI ’07, pp. 905- 914.
    12. 12. Second lab study results • Can people retain what they learned? – Users educated with PhishGuru retained knowledge after seven days • Do people have to fall for phish? – Users trained with embedded did better than users trained with non-embedded Kumaraguru, P., Rhee, Y., Sheng, S., Hasan, S., Acquisti, A., Cranor, L. F., and Hong, J. Getting users to pay attention to anti-phishing education: Evaluation of retention and transfer. e-Crime Researchers Summit, Anti-Phishing Working Group (2007).
    13. 13. Real world study: Portuguese ISP • Does PhishGuru training extend to real world? – Did reduce rate of falling for phishing – Trained participants retained knowledge after 7 days of training – Don’t have to train all people in organization Kumaraguru, P., Sheng, S., Acquisti, A., Cranor, L. F., and Hong, J. Lessons from a real world evaluation of anti-phishing training. e-Crime Researchers Summit, 2008
    14. 14. Real world study: CMU • Replicate previous study at larger scale • Investigate retention after 1 week, 2 weeks, and 4 weeks • Compare effectiveness of 2 training messages vs 1 training message • Examine demographics and phishing P. Kumaraguru, J. Cranshaw, A. Acquisti, L. Cranor, J. Hong, M. A. Blair, and T. Pham. School of Phish: A Real-World Evaluation of Anti-Phishing Training. 2009. SOUPS 2009.
    15. 15. Study design • Sent email to all CMU students, faculty and staff to recruit participants (opt-in) • 515 participants in three conditions – Control / One training message / Two messages • Emails sent over 28 day period – 7 simulated spear-phishing messages – 3 legitimate (cyber security scavenger hunt) • Campus help desks and all spoofed departments notified before messages sent
    16. 16. Our Simulated Spear Phish URL is not hiddenURL is not hidden Plain text email without graphics Plain text email without graphics
    17. 17. Our Simulated Phishing Website http://andrewwebmail.org/password/change.htm?ID=9009http://andrewwebmail.org/password/change.htm?ID=9009
    18. 18. Our Simulated Phishing Website http://andrewwebmail.org/password/thankyou.html?ID=9009http://andrewwebmail.org/password/thankyou.html?ID=9009
    19. 19. Effect of PhishGuru Training Condition N % who clicked on Day 0 % who clicked on Day 28 Control 172 52.3 44.2 Trained 343 48.4 24.5
    20. 20. Results conditioned on participants who clicked on day 0 TestTest
    21. 21. Results conditioned on participants who clicked on day 0 Trained participants less likely to fall for phish Trained participants less likely to fall for phish Trained participants remember what they learned 28 days later Trained participants remember what they learned 28 days later Test + train Test + train TestsTests TestsTests
    22. 22. Results conditioned on participants who clicked on day 0 and day 14 Two-train participants less likely than one-train participants to click on days 16 and 21 Two-train participants less likely than one-train participants to click on days 16 and 21
    23. 23. Results conditioned on participants who clicked on day 0 and day 14 Two-train participants less likely than one-train participants to click on days 16 and 21 Two-train participants less likely than one-train participants to click on days 16 and 21 Two-train participants less likely than one-train participants to provide information on day 28 Two-train participants less likely than one-train participants to provide information on day 28
    24. 24. Does PhishGuru Affect Clicking on Legitimate Emails? Condition N Day 0 Day 7 Day 28 Clicked % Clicked % Clicked % Control 90 50.0 41.1 38.9 One-train 89 39.3 42.7 32.3 Two-train 77 48.1 44.2 35.1 For Cyber Security Scavenger Hunt No difference between the three conditions on days 7 and 28 For Cyber Security Scavenger Hunt No difference between the three conditions on days 7 and 28
    25. 25. Students Most Vulnerable • Students significantly more likely to fall for phish than staff before training • No significant differences based on student year, department, or gender • 18-25 age group most vulnerable Age group Day 0 Day 28 18-25 62% 36% 26-35 48% 16% 36-45 33% 18% 45 and older 43% 10%
    26. 26. Most Participants Liked PhishGuru, Wanted More • 280 post study responses • 80% recommended that CMU continue PhishGuru training – “I really liked the idea of sending CMU students fake phishing emails and then saying to them, essentially, HEY! You could've just gotten scammed! You should be more careful - here's how....” – “I think the idea of using something fun, like a cartoon, to teach people about a serious subject is awesome!”
    27. 27. Summary • People trained with PhishGuru far less likely to click on phishing links than not trained • People retained training for 28 days • Two training messages better than one • PhishGuru training does not make people less likely to click on legitimate links
    28. 28. For More Information • Forthcoming SOUPS 2009 paper • White paper on Wombat Security web site • PhishGuru commercialized by Wombat Security
    29. 29. Acknowledgments • Supporting Trust Decisions group • CyLab Usable Privacy and Security Lab • CMU’s Information Security Office • APWG • Supported by National Science Foundation, Army Research Office, CyLab, ISP in Portugal
    30. 30. Study schedule Day of the study Control One training message Two training messages Day 0 Test and real Train and real Train and real Day 2 Test Day 7 Test and real Day 14 Test Test Train Day 16 Test Day 21 Test Day 28 Test and real Day 35 Post-study survey
    31. 31. Why is Teaching People Hard?
    32. 32. Why is Teaching People Hard? • Problems – Existing materials good, but could be better • Not many opportunities for testing skills – Most people don’t proactively look for security training materials – “Security notice” emails tend to be ignored • Too much to read • People don’t consider them relevant
    33. 33. Legitimate emails No difference between the three conditions on day 0, 7, and 28 No difference between the three conditions on day 0, 7, and 28 No difference within the three conditions for the three emails No difference within the three conditions for the three emails Condition N Day 0 Day 7 Day 28 Clicked % Clicked % Clicked % Control 90 50.0 41.1 38.9 One-train 89 39.3 42.7 32.3 Two-train 77 48.1 44.2 35.1

    ×