©2009CarnegieMellonUniversity:1
Leveraging Human Factors
for Effective Security Training
FISSEA 2012
Jason Hong
jasonh@cs....
©2012CarnegieMellonUniversity:2
About the Speaker
• Associate Prof, Carnegie Mellon
University, School of Comp Science
• R...
©2012CarnegieMellonUniversity:3
About this Talk
• Useful for people interested in:
– How to effectively train people?
– Ho...
©2012CarnegieMellonUniversity:4
Human Element of Security
• People are key part of computer
security for every organizatio...
©2012CarnegieMellonUniversity:5
What is Human-Computer
Interaction?
• Field that seeks to understand the
relationship betw...
©2012CarnegieMellonUniversity:6
Interactions Can Be Successful
©2012CarnegieMellonUniversity:7
Interactions Can Also Fail
©2012CarnegieMellonUniversity:8
Design Principles in 5 Minutes
• How do people believe
how things work?
• Mental models de...
©2012CarnegieMellonUniversity:9
Mental Models Example:
Refrigerator
Freezer
(temperature too cold)
Fresh food
(temperature...
©2012CarnegieMellonUniversity:10
Refrigerator Controls
What is a typical conceptual model?
Normal Settings C and 5
Colder ...
©2012CarnegieMellonUniversity:11
7 6 5 4 3
A B C D E
Most people think of
independent controls
Cooling
Unit
Cooling
Unit
A...
©2012CarnegieMellonUniversity:12
• Now can you fix the problem?
• Two general solutions:
– make controls map to user’s men...
©2012CarnegieMellonUniversity:13
Nissan Maxima Gear Shift
©2012CarnegieMellonUniversity:14
• Users create a model from what they hear
from others, past experiences, and usage
– int...
©2012CarnegieMellonUniversity:15
Mental Models
• People inevitably build models of how
things work
– Ex. children and comp...
©2012CarnegieMellonUniversity:16
Example: Phishing Attacks
• Interviewed 40 people as part of an
“email study” (Downs et a...
©2012CarnegieMellonUniversity:17
Example: Phishing Attacks
• 55% of participants reported being
cautious when email asks f...
©2012CarnegieMellonUniversity:18
Can We Educate End-Users?
• Users not motivated to learn
• Security is a secondary task
•...
©2012CarnegieMellonUniversity:19
Yes, End-Users Are Trainable
• Our research demonstrates users can
learn how to protect t...
©2012CarnegieMellonUniversity:20
How Do We Get People Trained?
• Create “teachable moments”
• Micro-games for training (fu...
©2012CarnegieMellonUniversity:21
PhishGuru Embedded Training
• Send simulated phishing emails
• If recipient falls for it,...
©2012CarnegieMellonUniversity:22
Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Info...
©2012CarnegieMellonUniversity:23
Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Info...
©2012CarnegieMellonUniversity:24
©2012CarnegieMellonUniversity:25
Learning Science
• Area of research examining learning,
retention, and transfer of skills...
©2012CarnegieMellonUniversity:26
Evaluation of PhishGuru
• Is embedded training effective?
– We’ve conducted 4 peer-review...
©2012CarnegieMellonUniversity:27
Results of One Study
• Tested 500+ people in one month
– 1 simulated phish at beginning o...
©2012CarnegieMellonUniversity:28
Can Browser Interfaces Help?
• Modern web browsers come with
blacklists and special inter...
©2012CarnegieMellonUniversity:29
Screenshots
Internet Explorer 7 – Passive Warning
©2012CarnegieMellonUniversity:30
Screenshots
Internet Explorer 7 – Active Block
©2012CarnegieMellonUniversity:31
Screenshots
Mozilla Firefox – Active Block
©2012CarnegieMellonUniversity:32
How Effective are these
Warnings?
• Tested four conditions
– FireFox Active Block
– IE Ac...
©2012CarnegieMellonUniversity:33
How Effective are these
Warnings?
Almost everyone clicked, even those
with strong technic...
©2012CarnegieMellonUniversity:34
How Effective are these
Warnings?
• No one in Firefox condition fell for our phish
• Peop...
©2012CarnegieMellonUniversity:35
Discussion of Phish Warnings
• Nearly everyone will fall for highly
targeted and contextu...
©2012CarnegieMellonUniversity:36
Screenshots
Internet Explorer – Passive Warning
©2012CarnegieMellonUniversity:37
Discussion of Phish Warnings
• Active IE warnings
– Most saw the warning, but many did no...
©2012CarnegieMellonUniversity:38
Screenshots
Internet Explorer – Active Block
©2012CarnegieMellonUniversity:39
MSIE8 Re-design Based on
our Work
MSIE8 Redesign Based on our Work
©2012CarnegieMellonUniversity:40
A Science of
Warnings
• C-HIP model
for real-world
warnings
– See the warning?
– Understa...
©2012CarnegieMellonUniversity:41
Designing for Path of Least
Resistance
• Where possible, make the
default behavior safe
–...
©2012CarnegieMellonUniversity:42
Summary
• Human element most overlooked
aspect of computer security
– Ex. phishing scams,...
©2012CarnegieMellonUniversity:43
More of Our Research
• Our team does research on:
– Better password policies
– Alternativ...
©2012CarnegieMellonUniversity:44
More of Our Research
• http://cups.cs.cmu.edu
• http://mcom.cs.cmu.edu
• http://cmuchimps...
©2012CarnegieMellonUniversity:45
Thanks, where can
I learn more?
Find more at
wombatsecurity.com
jasonh@cs.cmu.edu
©2012CarnegieMellonUniversity:46
©2012CarnegieMellonUniversity:47
Micro-Games for Cyber
Security
• Training doesn’t have to be long & boring
• Micro game f...
©2012CarnegieMellonUniversity:48
Case Study: Anti-Phishing Phil
• Tested Anti-Phishing Phil with ~4500
people
– Huge impro...
©2012CarnegieMellonUniversity:49
©2012CarnegieMellonUniversity:50
©2012CarnegieMellonUniversity:51
©2012CarnegieMellonUniversity:52
©2012CarnegieMellonUniversity:53
©2012CarnegieMellonUniversity:54
©2012CarnegieMellonUniversity:55
False negatives for users who played Anti-Phishing Phil (“game condition”). False negativ...
©2012CarnegieMellonUniversity:56
False positives for users who played the Anti-Phishing Phil game. False positives are sit...
©2012CarnegieMellonUniversity:57
©2012CarnegieMellonUniversity:58
Example Topic: Email Security
©2012CarnegieMellonUniversity:59
Example Topic: Passwords
©2012CarnegieMellonUniversity:60
Other Training: Social
Networks
©2012CarnegieMellonUniversity:61
Measurable
©2012CarnegieMellonUniversity:62
Measurable
©2012CarnegieMellonUniversity:63
Case Study #1: PhishGuru
• Canadian healthcare organization
• Three-month embedded traini...
©2012CarnegieMellonUniversity:64
Simulated Phishing Email
©2012CarnegieMellonUniversity:65
Case Study
©2012CarnegieMellonUniversity:66
Measurable Reduction in
Falling for Phish
Viewed
Email
Only %
Viewed
Email and
Clicked
Li...
©2012CarnegieMellonUniversity:67
0 10 20 30 40
Campaign3
Campaign2
Campaign1
ViewedEmail and Clicked
Link
ViewedEmail Only
©2012CarnegieMellonUniversity:68
Can We Educate End-Users?
• Users not motivated to learn
• Security is a secondary task
•...
Upcoming SlideShare
Loading in …5
×

Leveraging Human Factors for Effective Security Training, at FISSEA Mar2012

239 views
206 views

Published on

I discuss a range of human factors issues for cybersecurity, in particular cybersecurity awareness and education. Topics include mental models, user interfaces, and simulated attacks.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
239
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • On the left is Nissan Maxima gear shift. It turns out my brother was driving in 3 rd gear for over a year before I pointed out to him that 3 and D are separate. The older Nissan Maxima gear shift on the right makes it hard to make this mistake.
  • These findings led us to think about how to educate and train people about phishing attacks…
  • http://news.cnet.com/21007350_361252132.html
  • These findings led us to think about how to educate and train people about phishing attacks…
  • These findings led us to think about how to educate and train people about phishing attacks…
  • ASSUME THAT THIS IS YOUR EMAIL INBOX AND AMONG OTHER EMAILS.. YOU THIS EMAIL FROM AMAZON THAT JUST LOOKS LIKE THE LEGITIMATE EMAIL FROM AMAZON. WHEN YOU OPEN THE EMAIL ….
  • YOU WILL SEE THIS.. WHICH LOOKS LEGITIMATE.. AND WITH THE DATA THAT WE HAVE .. WE KNOW THAT MOST OF THE USERS WILL CLICK ON THE LINK.. WHEN THEY CLICK ON THE LINK THEY WILL SEE ….
  • P. Kumaraguru et al. Protecting People from Phishing: The Design and Evaluation of an Embedded Training Email System. CHI 2007. P. Kumaraguru et al. Getting Users to Pay Attention to Anti-Phishing Education: Evaluation of Retention and Transfer . eCrime 2007.
  • Our evaluation of several blacklists show they catch ~80% of phish after 24 hours, not very good in first few hours Also only catch “shotgun phish” rather than spear-phish
  • S. Egelman, L. Cranor, and J. Hong. You've Been Warned: An Empirical Study of the Effectiveness of Web Browser Phishing Warnings. CHI 2008.
  • http://wombatsecurity.com/file_download/6/PhishGuru%20White%20Paper.pdf http://wombatsecurity.com/file_download/8/Anti-Phishing%20Phil%20whitepaper.pdf
  • http://news.cnet.com/21007350_361252132.html
  • Leveraging Human Factors for Effective Security Training, at FISSEA Mar2012

    1. 1. ©2009CarnegieMellonUniversity:1 Leveraging Human Factors for Effective Security Training FISSEA 2012 Jason Hong jasonh@cs.cmu.edu
    2. 2. ©2012CarnegieMellonUniversity:2 About the Speaker • Associate Prof, Carnegie Mellon University, School of Comp Science • Research and teaching interests: – Usable privacy and security – Mobile computing • Co-author • Startup
    3. 3. ©2012CarnegieMellonUniversity:3 About this Talk • Useful for people interested in: – How to effectively train people? – How to effectively design better user interfaces for privacy and security? • Two case studies from my research Embedded Training Evaluating Warnings
    4. 4. ©2012CarnegieMellonUniversity:4 Human Element of Security • People are key part of computer security for every organization – Keeping passwords strong and secure – Avoiding social engineering – Avoiding malware – Appropriate use of social networking – Keeping mobile devices secure • Overlooking human element is most common mistake in computer security
    5. 5. ©2012CarnegieMellonUniversity:5 What is Human-Computer Interaction? • Field that seeks to understand the relationship between people & computers – Designing useful, usable, desirable artifacts – Understanding how people use systems – Expanding the ways we can use computers • Combines behavioral sciences, interaction design, and computer science
    6. 6. ©2012CarnegieMellonUniversity:6 Interactions Can Be Successful
    7. 7. ©2012CarnegieMellonUniversity:7 Interactions Can Also Fail
    8. 8. ©2012CarnegieMellonUniversity:8 Design Principles in 5 Minutes • How do people believe how things work? • Mental models describe how a person thinks something works • Incorrect mental models can make things very hard to understand and use
    9. 9. ©2012CarnegieMellonUniversity:9 Mental Models Example: Refrigerator Freezer (temperature too cold) Fresh food (temperature just right)
    10. 10. ©2012CarnegieMellonUniversity:10 Refrigerator Controls What is a typical conceptual model? Normal Settings C and 5 Colder Fresh Food C and 6-7 Coldest Fresh Food B and 8-9 Colder Freezer D and 7-8 Warmer Fresh Food C and 4-1 OFF (both) 0 A B C D E 7 6 5 4 3
    11. 11. ©2012CarnegieMellonUniversity:11 7 6 5 4 3 A B C D E Most people think of independent controls Cooling Unit Cooling Unit A Common Conceptual Model
    12. 12. ©2012CarnegieMellonUniversity:12 • Now can you fix the problem? • Two general solutions: – make controls map to user’s mental model – foster a more accurate mental model 7 6 5 4 3 A B C D E Cooling Unit Actual Conceptual Model Controls amount of cold air Controls amount air vectored up and down
    13. 13. ©2012CarnegieMellonUniversity:13 Nissan Maxima Gear Shift
    14. 14. ©2012CarnegieMellonUniversity:14 • Users create a model from what they hear from others, past experiences, and usage – interactions with system image Three Different Models Design Model (How you intend the system to work) Design Model (How you intend the system to work) User Model (How users think the system works) User Model (How users think the system works) System Image (Your implementation) System Image (Your implementation) User Interactions System feedback
    15. 15. ©2012CarnegieMellonUniversity:15 Mental Models • People inevitably build models of how things work – Ex. children and computers – Ex. you and your car – Ex. how hackers work (and why) – Ex. visibility in social networking sites – Ex. app stores (all apps vetted by Google?) • Two options: – Make the system match people’s models – Foster a better mental model
    16. 16. ©2012CarnegieMellonUniversity:16 Example: Phishing Attacks • Interviewed 40 people as part of an “email study” (Downs et al, SOUPS 2006) • Only 55% of participants said they had ever noticed an unexpected or strange- looking URL – Most did not consider them to be suspicious
    17. 17. ©2012CarnegieMellonUniversity:17 Example: Phishing Attacks • 55% of participants reported being cautious when email asks for sensitive financial info – But very few reported being suspicious of email asking for passwords • Knowledge of financial phish reduced likelihood of falling for these scams – But did not transfer to other scams, such as an amazon.com password phish
    18. 18. ©2012CarnegieMellonUniversity:18 Can We Educate End-Users? • Users not motivated to learn • Security is a secondary task • Difficult to teach people right decisions without increasing false positives • Basically, educating users is as hard as herding cats
    19. 19. ©2012CarnegieMellonUniversity:19 Yes, End-Users Are Trainable • Our research demonstrates users can learn how to protect themselves… if you can get them to pay attention to training • Problem is that today’s training often boring, time consuming, and ineffective – All day lecture, no chance to practice skills – Or passively watching videos – Or posters and mugs and calendars – Raise awareness, but little on what to actually do
    20. 20. ©2012CarnegieMellonUniversity:20 How Do We Get People Trained? • Create “teachable moments” • Micro-games for training (fun) • Use learning science principles throughout Embedded Training Micro-Game on Phishing
    21. 21. ©2012CarnegieMellonUniversity:21 PhishGuru Embedded Training • Send simulated phishing emails • If recipient falls for it, show intervention that teaches what cues to look for in succinct and engaging format – Useful for people who don’t know that they don’t know • Multiple user studies have demonstrated that PhishGuru is effective • Delivering training via direct email not effective
    22. 22. ©2012CarnegieMellonUniversity:22 Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information
    23. 23. ©2012CarnegieMellonUniversity:23 Subject: Revision to Your Amazon.com InformationSubject: Revision to Your Amazon.com Information Please login and enter your informationPlease login and enter your information
    24. 24. ©2012CarnegieMellonUniversity:24
    25. 25. ©2012CarnegieMellonUniversity:25 Learning Science • Area of research examining learning, retention, and transfer of skills • Example principles – Learning by doing – Immediate feedback – Conceptual-procedural – Reflection – … many others
    26. 26. ©2012CarnegieMellonUniversity:26 Evaluation of PhishGuru • Is embedded training effective? – We’ve conducted 4 peer-reviewed studies showing embedded training works well – Studies showed significant decrease in falling for phish and ability to retain what they learned P. Kumaraguru et al. Protecting People from Phishing: The Design and Evaluation of an Embedded Training Email System. CHI 2007. P. Kumaraguru et al. School of Phish: A Real-Word Evaluation of Anti-Phishing Training. SOUPS 2009.
    27. 27. ©2012CarnegieMellonUniversity:27 Results of One Study • Tested 500+ people in one month – 1 simulated phish at beginning of month, testing done at end of month • ~50% reduction in falling for phish – 68 out of 85 surveyed said they recommend continuing doing this sort of training in the future – “I really liked the idea of sending [organization] fake phishing emails and then saying to them, essentially, HEY! You could've just gotten scammed! You should be more careful – here's how...”
    28. 28. ©2012CarnegieMellonUniversity:28 Can Browser Interfaces Help? • Modern web browsers come with blacklists and special interfaces for identifying phish – Our evaluation of several blacklists show they catch ~80% of phish after 24 hours, not very good in first few hours • Are these browser interfaces effective? – And, what can we learn from them? – Science of Warnings from human factors
    29. 29. ©2012CarnegieMellonUniversity:29 Screenshots Internet Explorer 7 – Passive Warning
    30. 30. ©2012CarnegieMellonUniversity:30 Screenshots Internet Explorer 7 – Active Block
    31. 31. ©2012CarnegieMellonUniversity:31 Screenshots Mozilla Firefox – Active Block
    32. 32. ©2012CarnegieMellonUniversity:32 How Effective are these Warnings? • Tested four conditions – FireFox Active Block – IE Active Block – IE Passive Warning – Control (no warnings or blocks) • “Shopping Study” – Setup phishing pages and added to blacklists – Phished users after real purchases (2 phish) – Used real email accounts and personal info S. Egelman, L. Cranor, and J. Hong. You've Been Warned: An Empirical Study of the Effectiveness of Web Browser Phishing Warnings. CHI 2008.
    33. 33. ©2012CarnegieMellonUniversity:33 How Effective are these Warnings? Almost everyone clicked, even those with strong technical backgrounds
    34. 34. ©2012CarnegieMellonUniversity:34 How Effective are these Warnings? • No one in Firefox condition fell for our phish • People in Firefox condition not more technically savvy
    35. 35. ©2012CarnegieMellonUniversity:35 Discussion of Phish Warnings • Nearly everyone will fall for highly targeted and contextualized phish • Passive IE warning failed for many reasons – Didn’t interrupt the main task – Can be slow to appear (up to 5 seconds) – Not clear what the right action was – Looked too much like other ignorable warnings (habituation) – Bug, any keystroke dismissed
    36. 36. ©2012CarnegieMellonUniversity:36 Screenshots Internet Explorer – Passive Warning
    37. 37. ©2012CarnegieMellonUniversity:37 Discussion of Phish Warnings • Active IE warnings – Most saw the warning, but many did not believe it • “Since it gave me the option of still proceeding to the website, I figured it couldn’t be that bad” – Some element of habituation (looks like other warnings) – Saw two pathological cases
    38. 38. ©2012CarnegieMellonUniversity:38 Screenshots Internet Explorer – Active Block
    39. 39. ©2012CarnegieMellonUniversity:39 MSIE8 Re-design Based on our Work MSIE8 Redesign Based on our Work
    40. 40. ©2012CarnegieMellonUniversity:40 A Science of Warnings • C-HIP model for real-world warnings – See the warning? – Understand it? – Believe it? – Motivated? – Can and will act?
    41. 41. ©2012CarnegieMellonUniversity:41 Designing for Path of Least Resistance • Where possible, make the default behavior safe – Ex. The two pathological cases – Assume people won’t see, read, believe, or be motivated • Active warnings over passive warnings – Interrupt people if warning is important – Need to balance this with habituation • Make important warnings look very different
    42. 42. ©2012CarnegieMellonUniversity:42 Summary • Human element most overlooked aspect of computer security – Ex. phishing scams, passwords, mobile • Mental models important to design – Mismatched models can cause failures • Security training can work if done right – Learning sciences • C-HIP model for security warnings – Do people see, understand, believe, and can act on warnings?
    43. 43. ©2012CarnegieMellonUniversity:43 More of Our Research • Our team does research on: – Better password policies – Alternatives to passwords – Mobile apps, privacy and security – Location-based services and privacy – Social networking and privacy – Configuring firewalls
    44. 44. ©2012CarnegieMellonUniversity:44 More of Our Research • http://cups.cs.cmu.edu • http://mcom.cs.cmu.edu • http://cmuchimps.org/
    45. 45. ©2012CarnegieMellonUniversity:45 Thanks, where can I learn more? Find more at wombatsecurity.com jasonh@cs.cmu.edu
    46. 46. ©2012CarnegieMellonUniversity:46
    47. 47. ©2012CarnegieMellonUniversity:47 Micro-Games for Cyber Security • Training doesn’t have to be long & boring • Micro game format, play for short time • Two-thirds of Americans played a video game in past six months • Not just young people – Average game player 35 years old – 25% of people over 50 play games • Not just males – 40% of casual gamers are women
    48. 48. ©2012CarnegieMellonUniversity:48 Case Study: Anti-Phishing Phil • Tested Anti-Phishing Phil with ~4500 people – Huge improvement by novices in identifying phishing URLs – Also dramatically lowered false positives
    49. 49. ©2012CarnegieMellonUniversity:49
    50. 50. ©2012CarnegieMellonUniversity:50
    51. 51. ©2012CarnegieMellonUniversity:51
    52. 52. ©2012CarnegieMellonUniversity:52
    53. 53. ©2012CarnegieMellonUniversity:53
    54. 54. ©2012CarnegieMellonUniversity:54
    55. 55. ©2012CarnegieMellonUniversity:55 False negatives for users who played Anti-Phishing Phil (“game condition”). False negatives are situations where people incorrectly label a phishing site as legitimate. Novices saw the greatest reduction in false negatives, and retained what they had learned.
    56. 56. ©2012CarnegieMellonUniversity:56 False positives for users who played the Anti-Phishing Phil game. False positives are situations where people incorrectly label a legitimate site as phishing. Again, novices saw the greatest improvement in reducing false positives, and retained what they had learned.
    57. 57. ©2012CarnegieMellonUniversity:57
    58. 58. ©2012CarnegieMellonUniversity:58 Example Topic: Email Security
    59. 59. ©2012CarnegieMellonUniversity:59 Example Topic: Passwords
    60. 60. ©2012CarnegieMellonUniversity:60 Other Training: Social Networks
    61. 61. ©2012CarnegieMellonUniversity:61 Measurable
    62. 62. ©2012CarnegieMellonUniversity:62 Measurable
    63. 63. ©2012CarnegieMellonUniversity:63 Case Study #1: PhishGuru • Canadian healthcare organization • Three-month embedded training campaign – 190 employees – Security assessment and effective training in context
    64. 64. ©2012CarnegieMellonUniversity:64 Simulated Phishing Email
    65. 65. ©2012CarnegieMellonUniversity:65 Case Study
    66. 66. ©2012CarnegieMellonUniversity:66 Measurable Reduction in Falling for Phish Viewed Email Only % Viewed Email and Clicked Link % Employees Campaign 1 20 10.53% 35 18.42% 190 Campaign 2 37 19.47% 23 12.11% 190 Campaign 3 7 3.70% 10 5.29% 189
    67. 67. ©2012CarnegieMellonUniversity:67 0 10 20 30 40 Campaign3 Campaign2 Campaign1 ViewedEmail and Clicked Link ViewedEmail Only
    68. 68. ©2012CarnegieMellonUniversity:68 Can We Educate End-Users? • Users not motivated to learn • Security is a secondary task • Difficult to teach people right decisions without increasing false positives “User education is a complete waste of time. It is about as much use as nailing jelly to a wall… They are not interested…they just want to do their job.” -- An IBM security specialist

    ×