The presentation is based on research paper titled Alice in warningland: A Large Scale Study of Browser Security Warnings, Experimenting At Scale With Google Chrome’s SSL Warning

1. The Effectiveness of Browser
Security Warnings and
reducing SSL Click-through
Rates
Presented By:
Ruchir Dhiman
Meghna Singhal

2. Base Paper Details
Cristian Bravo-Lill o, Lorrie Faith Cranor, Julie S. Downs and Saranga
Komanduri: Bridging the Gap in Computer Security Warnings: A Mental
Model Approach, 17 December 2010, doi: 10.1109/MSP.2010.198,
Security & Privacy, IEEE (Volume:9 , Issue: 2 )

3. Given a choice between dancing pigs
and security, the user will pick
dancing pigs every time.
Felton
&
Mcgraw

4. Evidence from experimental studies indicates that
most people
don’t read computer warnings,
don’t understand them, or simply
don’t heed them,
even when the situation is clearly hazardous.

5. Introduction
• Warnings are a form of communication designed to
protect people from harm.
• An effective physical warning clearly communicates
risk, consequences of not complying, and instructions
to comply (although some of this information can be
omitted if the risk is obvious or the consequences can
be deduced from the warning).
• Many of the most common computer alerts fail to
follow one or more of these guidelines.

6. Introduction
• Web browsers show warnings to users when an attack
might be occurring.
• If the browser is certain that an attack is occurring, it will
show an error page that the user cannot bypass.
• If there is a chance that the perceived attack is a false
positive, the browser will show a bypassable
warning that discourages the user from continuing.

7. Example
• Consider a hazardous broken sidewalk. You could
repair (design the risk out) or put a barricade
around it (guard against the risk). You could post
warning signs as an interim solution, but they
shouldn’t be the only safeguard. However, in some
situations, designing out a hazard or guarding
against it might not be feasible.
• Similarly, the risk of being phished by a malicious
website can’t be completely designed out, although
users could employ guarding strategies such as
automatically detecting and removing suspicious
links from email.

8. The warning dialog doesn’t
explain the risk (the file
might be infected with
malware) or consequences
(information might get
corrupted, erased, or
disclosed to third parties),
and it doesn’t instruct users
on how to avoid the risk
(either delete attachment
or save it on your hard disk
and scan it with your
antivirus software)

9. Problem Statement
Computer security warnings are intended to
protect users and their computers. However,
research suggests that users frequently ignore
these warnings. The authors describe a study
designed to gain insight into how users perceive
and respond to computer alerts.

10. Study Methodology
• They collected examples of 29 security warnings from
popular operating systems and application software
and categorized them into four warning types:
information deletion or loss, information disclosure,
execution of malicious code, and trust in malicious
third parties.
• They picked one to two warnings from each category: a
disk space warning, an email-encryption warning, an
address book disclosure warning, an email attachment
warning (see Figure 1), and a certificate warning.
• They created at least one scenario per warning in which
they briefly described a situation that provided context
for the warning’s appearance.

11. • To improve users’ understanding of warnings, we first need
to determine how users process the information in them,
that is, how they think about warnings. For this purpose,
they conducted 30 interviews—10 with advanced users in
security and privacy and 20 with novice users.
• Interviews had seven segments: a brief general section
about computer use, five sections that asked about warning
reactions, and a final segment about demographics. In each
warning segment, we showed a warning dialog and read
aloud a brief scenario that described a nontechnically savvy
friend asking the participant for help. Then the following
main questions were asked.
 Could you tell me what this message is?
 What do you think will happen if your friend clicks on X?
(We asked for all the options present in the warning.)
 What do you think your friend should do?

12. In one study, 32 percent of people who heeded a
phishing warning attributed the warning to a Web
problem and still believed that phishing emails
sent to them were legitimate.

13. Click-through Rate
#warnings ignored
#warnings shown

14. When a user clicks through a warning, the
user has
• Ignored the warning because she did not
read or understand it or,
• made an informed decision to proceed because
she believes that the warning is a false
positive or her computer is safe against these
attacks (e.g. due to an antivirus).

15. What is the ideal click
through rate of effective
warnings?
0%

16. How was it
measured?

17. Browser Telemetry
• A mechanism for browsers to collect pseudonymous
performance and quality data from end users
• Users opt-in to sharing data with the browser
vendors
• Data collected: May 2013 (Akhawe D. and Felt A.P.)

18. Types of Browser Warnings
•Malware & Phishing
•SSL Warnings

19. Malware & Phishing Warnings
• If a malware or phishing warning is a true positive,
clicking through exposes the user to a dangerous
situation.
• The browsers routinely fetch a list of suspicious
(i.e., malware or phishing) sites from Safe Browsing
servers. If a user tries to visit a site that is on the
locally cached list, the browser checks with the Safe
Browsing service that the URL is still on the
malware or phishing list. If the site is still on one of
the lists, the browser presents a warning.

20. Malware & Phishing Warnings (Cont.)
• Google Chrome stops the page load and replaces the page with
a warning.
• Mozilla Firefox blocks the third-party resource with no
warning.
• Mozilla Firefox users can see fewer warnings than Google
Chrome users, despite both browsers using the same Safe
Browsing list.
• When a browser presents the user with a malware or phishing
warning, she has three options:
leave the page via the warning’s escape button
leave the page by closing the window or typing a new URL
click through the warning and proceed to the page

21. Malware warning for Google Chrome
Chrome users who want to bypass the warning need to click
twice: first on the “Advanced” link, and then on “Proceed at
your own risk”.

22. Malware warning for Mozilla Firefox
Users who want to bypass the warning need to click one button:
the “Ignore this warning”

23. SSL
• SSL is a Secure Sockets Layer and
• SSL is the standard security technology for
establishing an encrypted link between a web server
and a browser.
• This link ensures that all data passed between the
web server and browsers remain private and
integral.

24. Step 1: Client accesses website
Client
Browser connects to website
Web Server
Step 2: Server responds with Certificate
Client
Server responds with
Certificate and key Web Server
Step 3: Client verifies with CA
Client Web Server
CA
Client verifies certificate
with CA

25. Step 4: Client sends random key to server
Client Web Server
Random
Key
Client sends a random key to server
encrypted with the public key
Step 5: All communications are now encrypted with the
Random key
Client Web Server
Random
Key

26. SSL Warnings
• The validation will fail in the presence of a man-in-the-
middle (MITM) attack.
• Authentication failures can also occur in a wide variety
of benign scenarios, such as server misconfigurations.
Browsers usually cannot distinguish these benign
scenarios from real MITM attacks. Instead, browsers
present users with a warning; users have the option to
bypass the warning, in case the warning is a false
positive.
• A 0% click through rate for SSL warnings is desired.
However, many SSL warnings may be false positives
(e.g. server misconfigurations).

27. SSL Warnings (Cont.)
• There are two competing views regarding SSL
false positives.
In the first, warning text should discourage users
from clicking through both true and false
positives, in order to incentivize developers to get
valid SSL certificates.
In the other, warning text should provide users
with enough information to correctly identify and
dismiss false positives.

28. SSL Warnings (Cont.)
• The desired click through rates for false-positive
warnings would be 0% and 100%, respectively.
• In either case, false positives are undesirable for the
user experience because we do not want to annoy
users with invalid warnings.
• Therefore the goal is 0% click through rate for all
SSL warnings:
users should heed all valid warnings
the browser should minimize the number of false
positives

29. SSL warning for Google Chrome

30. SSL warning for Mozilla Firefox

31. Year Description
2006 15 out of 22 clicked through without reading it. Only one user was later
able to tell the researchers what the warning had said
2007 53% of the total 57 participants clicked through
2009 409 people were asked about Firefox 2, Firefox 3, and
Internet Explorer 7 warnings. Less than half of respondents said they
would continue to the website after seeing the warning
2009 The clickthrough rates were 90%, 55%, and 90% when participants tried
to access their bank websites in Firefox 2, Firefox 3, and Internet
Explorer 7, respectively.
The clickthrough rates increased to 95%, 60%, and 100% when
participants saw an SSL warning while trying to visit the university
library website.

32. Malware and Phishing Warnings
• Click through rates for malware warnings were
7.2% and 23.2% in stable versions of Mozilla
Firefox and Google Chrome respectively.
• For phishing warnings the click through rates
were 9.1% and 18.0% for the two browsers.

33. Malware Rates by Date
• Malware warning click-through rates for Chrome vary
widely as rates ranging from 11.2% to 24.9% were
observed depending on the week.
• In contrast, the Mozilla Firefox malware warnings vary
within one percentage point of the month-long average.
• Such variations weren’t observed in phishing of SSL
warning click-through rates.

34. Malware/Phishing Rates by Demographics
• Linux users have significantly higher clickthrough
rates than Mac and Windows users combined.
• Early adopters have comparatively higher
clickthrough rates when compared to users of the
stable versions (in most cases).
• One possible explanation is the greater technical
skill of both Linux users and the early-version
adopters.

35. User Operating System vs. click-through
rates for malware and phishing warnings
Operating
System
Malware Phishing
Firefox Chrome Firefox Chrome
Windows 7.1% 23.5% 8.9% 17.9%
Mac OS 11.2% 16.6% 12.5% 17.0%
Linux 18.2% 13.9% 34.8% 31.0%

36. Release channel vs. click-through rates for
malware and phishing warnings, for all operating
systems.
Channel Malware Phishing
Firefox Chrome Firefox Chrome
Stable 7.2% 23.2% 9.1% 18.0%
Beta 8.7% 22.0% 11.2% 28.1%
Dev 9.4% 28.1% 11.6% 22.0%
Nightly 7.1% 54.8% 25.9% 20.4%
As given by Akhawe D. and Felt A.P.

37. Malware/Phishing Rates by Browser
• Google Chrome users clickthrough phishing warnings
more often than Mozilla Firefox stable users. If iframes
are excluded Firefox Beta users still bypass warnings at a
lower rate 9.6% for malware and 10.8% for phishing.
• One explanation can be that the warnings of Firefox are
more frightening therefore more convincing.
• The other possibility being that the two have different
levels of risk tolerance and different demographics.

38. SSL Warnings
• The click-through rates for SSL Warnings were
33% for Mozilla Firefox (Beta Channel) and
70.2% for Google Chrome (Stable) as given by
Akhawe D. and Felt A.P.
• In this study the click-through rate for Chrome
was found to be 67.9% and the change is
attributed to fluctuation over time.

39. SSL Rates by Demographics
• Unlike malware and phishing clickthrough
variations in SSL warnings the difference w.r.t user
operating systems is less pronounced.
• In early adopters Nightly users have higher
clickthrough rates for both browsers.
• In Chrome, the Windows users are likely to bypass
SSL warnings whereas in Firefox, Linux users are
likely to bypass them when compared to the other
operating systems.

40. User Operating System vs. Click-
through Rates for SSL Warnings
Operating
System
SSL Warnings
Firefox Chrome
Windows 32.5% 71.1%
Mac OS 39.3% 68.8%
Linux 58.7% 64.2%
As given by Akhawe D. and Felt A.P.

41. Channel vs. Click-Through Rates for
SSL Warnings
Channel SSL Warnings
Firefox Chrome
Nightly 43.0% 74.0%
Dev 35.0% 75.9%
Beta 32.2% 73.3%
Stable NA 70.2%
As given by Akhawe D. and Felt A.P.

42. SSL Rates by Browser
• Chrome users are almost twice as likely as Firefox users to
bypass SSL warnings.
• Number of Clicks : Chrome users need to click one button
to dismiss SSL warnings whereas Firefox users have to click
three. But this isn’t the reason for the rate gap.
• Demographics : The differences in demographics may be
the case but as there was a very small difference in
malware/phishing rates this has a small effect.
• Warning Appearance : The warnings are displayed
previously.

43. SSL Warnings by Browser (Cont.)
• Certificate Pinning : Chrome ships a list of “pinned”
certificates to HSTS Sites (HTTP Strict Transport Security).
Users cannot clickthrough these sites.
• In contrast Firefox doesn’t come with many preloaded
“pinned” certificates on any HSTS Site.
• So, Chrome has almost 20% non-bypassable warnings as
compared to 1% for Firefox.
• Based on this, it is safe to say that Firefox encounters more
warnings on critical sites and hence, clickthrough rate will be
low.

44. SSL Warnings by Browser (Cont.)
• Remebering Exceptions : Due to “Permanently Store this
Exception” feature Firefox users see SSL Warnings for sites
without saved exceptions.
• And so after time a user may encounter the same rate of
warnings in both browsers.
• Assuming that users visit same sites often, two things are
possible. One, that the error is a false-positive and so lack of
exception storing raises the rate for Chrome.
• And two, if Chrome users are posed to more warnings they
may pay less attention to the warnings they may encounter.

45. SSL Rates by Certificate Error Type
• Google Chrome:
Clearly the results
are different from
what the
expectations are.
• We may assume
that untrusted
warnings occur for
unimportant sites but the data from Mozilla Firefox
suggests otherwise.
Certificate
Error
Percentage in
Total
Click-
Through Rate
Untrusted
Issuer
56.0% 81.8%
Name
Mismatch
25.0% 62.8%
Expired 17.6% 57.4%
Other Error 1.4% -
All Error
Types
100% 70.2%
As given by Akhawe D. and Felt A.P.

46. SSL Rates by Certificate Error Type (Cont.)
• Mozilla Firefox : The user is informed about the
specific error type in the secondary “Add Exception”
dialog box. To proceed this dialog must be confirmed.
• As the following table suggests that the error type does
not greatly influence confirmation rates and we can say
that the “Add Exception” dialog box does not do its job
properly.
• This also proves that we cannot attribute differences in
error as if that were the case then the same would be
seen for Chrome as well.

47. Confirmation Rates for different errors in “Add
Exception” Dialog Box
Certificate Error Percentage in Total Confirmation Rate
Untrusted Issuer 38.0% 87.1%
Untrusted and Name
Mismatch
26.4% 87.9%
Name Mismatch 15.7% 80.3%
Expired 10.2% 80.7%
All the three 4.7% 87.6%
Expired and
Untrusted
4.1% 83.6%
Expired and Name-
Mismatch
0.7% 85.2%
None of these <0.1% 77.9%
All Errors 100.0% 85.4%
As given by Akhawe D. and Felt A.P.

48. Time Spent on SSL Warnings
• In addition to MITM attacks, SSL warnings can occur due to server
misconfigurations which result in false-warnings, which are safe to
bypass.
• Time spent on SSL warning pages was measured and was recorded
into two categories.
Time by Outcome : 47% of the users ignoring the warning take 1.5s
whereas 47% of the leavers take 3.5 s which shows us that users who
click through do so after less consideration.
Time by Error Type : 49% of the untrusted issuer warnings were
clicked-through within 1.7s but took 2.2s and 2.7s on name and data
error warnings. This shows that users click through more-frequent
errors faster.

49. Graphs for Click-Through Rates
Click-through time by outcome (ms) Click-through time by error-type (ms)

50. Implications of Alice in Warningland
(Akhawe D. and Felt A.P.)
• Browser warnings can be effective security mechanisms but with
varying effectiveness.
• Clickthrough Rates : Contrary to popular belief this study shows
that browser security warnings can be highly effective at preventing
users from visiting websites.
• Google Chrome’s SSL has an undesirably high click-through rate at
70.2%. But other findings suggest room for improvement.
• User Attention : The following results suggest that users pay
attention to the warnings :
a. 24.4% difference in rates for untrusted issuer and expired certificate
errors.
b. 21.3% users un-check the default “Permanently Store Exception” option.

51. • Default Chrome Warning modified by adding
images.
• Firefox’s warning replicated in Chrome (Mock
Firefox).
• Mock Firefox warning without image.
• Mock Firefox warning with corporate styling.
Suggestions added in this study

52. Mock Firefox SSL warning

53. Firefox SSL warning with
Google styling

54. Click-through Rates for Conditions
No. Condition CTR
1. Default Chrome Warning 67.9%
2. Chrome warning with policeman 68.9%
3. Chrome warning with criminal 66.5%
4. Chrome warning with traffic light 68.8%
5. Mock Firefox 56.1%
6. Mock Firefox, no image 55.9%
7. Mock Firefox with corporate styling 55.8%

55. Implications
• Changing the appearance of the default warning by adding
images did not have any impact on the CTR.
• Adding a mock Firefox warning did reduce the CTR but 98%
of the people who clicked the first button also clicked the 2nd
and so we can say that adding such an easy extra step did not
effect the CTR at all.
• Modifying the warning using a different style guide does not
have a significant effect on the CTR.
• The pop-up menu of the Add exception may be the reason for
less CTR in Firefox but the effect produced will be
comparatively less (around 10%).

56. Suggestions
• The “Add exception dialog box” of Firefox
deterred only 15% of the users from going
through to the site and so, improving it should
lead to a less CTR.
• Google Chrome does not have a “Permanently
Add Exception” option and adding such a
feature should reduce the CTR by reducing the
click-throughs for repeated false-positive
warnings.

57. Improving The Add Exception Dialog
Box (Mozilla Firefox)
• Once users entered the “Add Exception” dialog box the
confirmation rate was almost same for all error types.
• The reason for this ineffectiveness of the dialog box can be its
very basic appearance and so we propose that if the
appearance of the dialog box is improved there may be an
increase in the user attention.
• The “Add Permanent Exception” should be changed to un-
ticked by default. The reason for implementing this change is
that in case a user confirms by mistake, s/he will get a chance
to rectify it when the site is re-visited in the future.

58. Improved Appearance of The “Add
Exception” Dialog Box

59. Improved Appearance of The “Add
Exception” Dialog Box
• The reason why these changes
should work is that we’ve noticed
that in Google Chrome such
warning messages did have an effect
on the click-through rates and so we
think that such detailed warnings
will bring down the confirmation
rate in some cases.

60. Reducing the SSL Click-Through rate in
Google Chrome
• Assuming that users visit same sites often, we can say that Chrome’s
high SSL warning click-through rate is because users may have to
click-through the warning for same sites multiple times.
• The lack of “Permanently store exception” option in Chrome may
cause repetition of SSL warnings for a site with a false warning.
• So, to reduce the click-through rate of Chrome’s SSL warnings we
propose an addition of this option in Google Chrome as well.
• The reason is that exceptions will be stored for frequently visited
sites and hence, there will be decrease in the click-through rate as
for the same site there will only be one instance.

61. Modified SSL Warning for Google
Chrome
• Here, the “Permanently Add Exception” check-box is un-checked by
default for added security because, as already specified, Chrome
uses HSTS sites for non-bypassable warnings and so, we assume
less false-warnings would occur in Chrome.

62. Thank
You