1. 10 Years Later:
Are We There Yet?
Carsten Eiram
Risk Based Security
@CarstenEiram

2. Quick Bio – VDB Work Experience
Involved with VDBs for 10+ years
• Currently, CRO at Risk Based Security – commercial arm of
Open Security Foundation (runs OSVDB and DatalossDB) – and
responsible for the VulnDB service.
• Chief Security Specialist at Secunia, running the Research team.
• Security Team Lead at Danish Verisign affiliate, running a
customer-only accessible vulnerability database.
NOT JUST SECURITY , THE RIGHT SECURITY

3. Quick Bio – Vulnerability Research
Officially been doing vulnerability research since 2003
• Focused on a static analysis / reverse engineering approach
• Jokingly refer to myself as a "vulnerability connoisseur"
- I enjoy analyzing vulnerabilities and their root causes.
• Critical vulnerabilities discovered in products from many major
software vendors.
NOT JUST SECURITY , THE RIGHT SECURITY

4. INTRODUCTION
What will be discussed?

5. Reason for Talk
After 10+ years of VDB work,
I felt it was time to reflect
on certain areas related to
vulnerabilities
NOT JUST SECURITY , THE RIGHT SECURITY

6. Considerations
NOT JUST SECURITY , THE RIGHT SECURITY

7. Metrics and their Usage
NOT JUST SECURITY , THE RIGHT SECURITY

8. Code Quality
NOT JUST SECURITY , THE RIGHT SECURITY

9. Advisory Quality
VENDORS MAKE BAD DECISIONS
NOT JUST SECURITY , THE RIGHT SECURITY

10. Vulnerability Handling / Bug Bounties
NOT JUST SECURITY , THE RIGHT SECURITY

11. Million Dollar (or Leu) Question
NOT JUST SECURITY , THE RIGHT SECURITY

12. Quick Show of Hands
NOT JUST SECURITY , THE RIGHT SECURITY

13. Vulnerability
A Quick Overview To Set The Stage
Statistics

14. Currently Oldest Recorded Vulnerabilities
Vulnerabilities have been around for a very long time
- And will continue to be...
• Oldest entries in OSVDB are 79399 and 79400
• Marconi wireless telegraph
• Dated November 1902
• Message spoofing and message disclosure
NOT JUST SECURITY , THE RIGHT SECURITY

15. Guglielmo Marconi
http://www.newscientist.com/article/mg21228440.700-dotdashdiss-the-gentleman-hackers-1903-lulz.html
NOT JUST SECURITY , THE RIGHT SECURITY

16. First Ever Unbreakable Claim!
http://www.newscientist.com/article/mg21228440.700-dotdashdiss-the-gentleman-hackers-1903-lulz.html
NOT JUST SECURITY , THE RIGHT SECURITY

17. Nevil Maskelyne Ruins Demo
http://www.newscientist.com/article/mg21228440.700-dotdashdiss-the-gentleman-hackers-1903-lulz.html
NOT JUST SECURITY , THE RIGHT SECURITY

18. No Wire-Cutting Please
While not providing
the privacy and
security as promised,
the wireless telegraph
still had one significant
advantage over the
wired telegraph:
Not possible to cut the
wires!
NOT JUST SECURITY , THE RIGHT SECURITY

19. Have We Improved?
Obviously, we have progressed a fair bit
technically since then, but have we gotten
significantly better?
NOT JUST SECURITY , THE RIGHT SECURITY

20. Bringing The Internet Down – Old Lady Style
Article: http://news.softpedia.com/news/Old-Lady-Cuts-Off-Internet-in-Armenia-193640.shtml
NOT JUST SECURITY , THE RIGHT SECURITY

21. 10 Year Vulnerability Trend
12000
10000
8000
6000
# Vulns
4000
2000
0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
NOT JUST SECURITY , THE RIGHT SECURITY

22. All Datasets Are Incomplete!
All datasets are incomplete - some just more than others
Many love taking CVE content that’s free and do random
conclusions based on it, but since the dataset is severely
lacking, the conclusions are as well
NOT JUST SECURITY , THE RIGHT SECURITY

23. 2006 – 2013 Vulnerability Type Trend
NOT JUST SECURITY , THE RIGHT SECURITY

24. 2012 Data Breaches due to SQL Injection
NOT JUST SECURITY , THE RIGHT SECURITY

25. Companies affected by XSS in 2012
Source: CWN - http://www.cyberwarnews.info/2012/07/04/300000-personal-details-leaked-38-sites-hacked-for-projectdragonfly/
NOT JUST SECURITY , THE RIGHT SECURITY

26. Companies Impacted By Hacking In 2012
NOT JUST SECURITY , THE RIGHT SECURITY

27. Vulnerability Metrics
Usage

28. Which is more secure?
Product A
10 Vulnerabilities
Product B
20 Vulnerabilities
NOT JUST SECURITY , THE RIGHT SECURITY

29. Security State != Number of Vulnerabilities
Previously, the security state of a product was considered
to be equal to the number of vulnerabilities.
Flawed conclusion!
Today, people understand that the number of vulnerabilities
!=
security state
NOT JUST SECURITY , THE RIGHT SECURITY

30. Some Apparently Still Don’t Know...
“The problem with Java is that a lot of vulnerabilities are
constantly being reported in it, and when a lot of
vulnerabilities are reported, then there are a lot of hackers
using these to access programs built on Java“
- Morten Stengaard, CTO, Secunia
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY

31. Dissecting the Statement – Part 1
”... then there are a lot of hackers
using these to access programs built on Java”
Most vulnerabilities in Java are not used to target Java
applications, but the Java Runtime Environment to
compromise the system.
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY

32. Dissecting the Statement – Part 2
”... when a lot of vulnerabilities are reported,
then there are a lot of hackers using these…”
Just because a lot of vulnerabilities are reported in a
product, a lot of hackers may not be exploiting them.
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY

33. Dissecting the Statement – Part 3
”The problem with Java is that a lot of vulnerabilities are
constantly being reported in it…”
The security state of a product is not defined by the number
of vulnerabilities reported in it.
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY

34. We Should All Stop Using Popular Software Then
400
350
300
250
Java
200
Chrome
Firefox
150
Internet Explorer
100
50
0
Vulnerabilities (2013 - Nov 10th)
NOT JUST SECURITY , THE RIGHT SECURITY

35. Facewall!
NOT JUST SECURITY , THE RIGHT SECURITY

36. Microsoft Argument For SDL (Windows)
NOT JUST SECURITY , THE RIGHT SECURITY

37. Microsoft Argument For SDL (SQL Server)
NOT JUST SECURITY , THE RIGHT SECURITY

38. Microsoft Office Vulnerability Trend
Vulnerabilities in Office versions one year after product release
(based on Microsoft security bulletins)
14
12
10
8
6
4
2
0
Office 2000
Office 2007
NOT JUST SECURITY , THE RIGHT SECURITY
Office 2010

39. Microsoft Security Bulletin Trend
350
300
250
200
Bulletins
150
CVEs
100
50
0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
NOT JUST SECURITY , THE RIGHT SECURITY

40. Researcher Focus and SCADA
NOT JUST SECURITY , THE RIGHT SECURITY

41. Stop Drawing Conclusions on Vulnerability Counts...
NOT JUST SECURITY , THE RIGHT SECURITY

42. There are so many other aspects to consider!
More things to consider incl.
Patched vs. Unpatched
Vulnerability Type
Impact
Time-To-Patch
Time-To-Vendor-Response
Security Mechanisms
...
NOT JUST SECURITY , THE RIGHT SECURITY

43. Vulnerability Metrics
Severity

44. Severity Metrics
Many different severity metrics – both public and internal
Most popular and hated is CVSS,
which currently has problems reflecting real risk
Many concerns raised about CVSSv2 by many people
e.g. myself and Brian Martin of OSVDB in our open letter:
"The CVSSv2 Shortcomings, Faults, and Failures Formulation"
http://www.riskbasedsecurity.com/reports/CVSS-ShortcomingsFaultsandFailures.pdf
NOT JUST SECURITY , THE RIGHT SECURITY

45. Limitations of Severity Metrics
Reflecting the threat of vulnerability-dependent issues
(e.g. sandbox bypass, ASLR bypass related to memory
disclosure etc.)
By themselves and from a scoring point-of-view, these issues
are pretty minor, but when combined with code execution...
Jackpot!
Ability to disclose a few memory addresses was in the past
pretty much a non-issue – today it’s very useful.
NOT JUST SECURITY , THE RIGHT SECURITY

46. Pick A Vuln... Any Vuln...
If I’d offer you one vulnerability
in e.g. Google Chrome, which
would you pick?
1) Code execution within
sandbox
CVSSv2:
6.8
2) Sandbox bypass
CVSSv2:
2.6
NOT JUST SECURITY , THE RIGHT SECURITY

47. Severity Metrics and Sandbox Bypasses
If we conclude that exploiters are more interested in the
sandbox bypass and system administrators should focus on
fixing such a vulnerability over a code execution
vulnerability within the sandbox, why are we not rating
them higher?
Case of reality not being reflected well by severity metrics
NOT JUST SECURITY , THE RIGHT SECURITY

48. Severity Metrics and Vulnerability Chains
And once these issues start occuring in chains, which is
becoming more and more common, then it really gets
complex...
You can have a lot of independent minor issues that when
combined suddenly are very serious
NOT JUST SECURITY , THE RIGHT SECURITY

49. Google Chrome Pwn2Own Example
CVSSv2:
6.8
CVSSv2:
5.1
OSVDB 89734
IPC channel missing
listener process validation
OSVDB 80007
Plugin blocking logic not run
for NaCl in pre-rendering
CVSSv2:
7.6
OSVDB 80293
Unpacked NPAPI extension
installation without
confirmation
CVSSv2:
OSVDB 81645
5.1
GPU command decoding
integer underflow
CVSSv2:
2.6
OSVDB 80741
CVSSv2:
2.6
OSVDB 89736
Too permissive
LoadExtension
bindings for
extension manager
Unprivileged renderer
can navigate to
privileged URLs
http://blog.chromium.org/2012/05/tale-of-two-pwnies-part-1.html
NOT JUST SECURITY , THE RIGHT SECURITY

50. When Severity Metrics Met Reality
NOT JUST SECURITY , THE RIGHT SECURITY

51. Vulnerability Metrics
Exploitability

52. Microsoft Severity Ratings
Source: http://technet.microsoft.com/en-us/security/gg309177.aspx
NOT JUST SECURITY , THE RIGHT SECURITY

53. Exploitability Index Ratings
NOT JUST SECURITY , THE RIGHT SECURITY

54. Microsoft Approach: Pros and Cons
Pros
Cons
Gives an realistic evaluation of
the technical requirements to
exploit a given vulnerability and
how feasible it is
Requires significant technical
skills and resources to get right
Makes it clear which are
theoretical and which are
plausible
Still requires a bit of
guesstimation
NOT JUST SECURITY , THE RIGHT SECURITY

55. No Granularity Really Added...
NOT JUST SECURITY , THE RIGHT SECURITY

56. How Does Adobe Do It?
NOT JUST SECURITY , THE RIGHT SECURITY

57. How Does Adobe Do It?
NOT JUST SECURITY , THE RIGHT SECURITY

58. Adobe Approach: Pros and Cons
Pros
...Cons
Allows understanding which
Does not factor in technical
products, versions, and
requirements and the nature of
architectures are most critical to the vulnerability i.e. does not
prioritize
differentiate between theoretical
issues and straight-forward issues
to exploit
Dynamic approach that can be
easily tweaked
Requires very little resources –
just an understanding of historical
exploitation
NOT JUST SECURITY , THE RIGHT SECURITY

59. How Does CVSSv2 Do It?
NOT JUST SECURITY , THE RIGHT SECURITY

60. CVSSv2 Approach: Pros and Cons
Pros
Most reliable of all the
approaches: If an exploit is
available, a vulnerability is clearly
exploitable.
Requires very little resources –
just knowledge of availability of
PoCs and exploits
Cons
Purely reactive, requiring very
fast response times
Only takes into account when the
availability of an exploit is
publicly known i.e. may be
exploited long before being
flagged as such
NOT JUST SECURITY , THE RIGHT SECURITY

61. No information about code quality
NOT JUST SECURITY , THE RIGHT SECURITY

62. Code Quality
... And How To Measure It

63. Code Quality – Why Measure It?
NOT JUST SECURITY , THE RIGHT SECURITY

64. Code Maturity Metric – The Idea
The idea of code maturity is that by evaluating
the prevalence of the different vulnerability
classes being discovered in a product, we can
conclude the maturity of that product.
We, naturally, focus on it from a security
perspective.
NOT JUST SECURITY , THE RIGHT SECURITY

65. Code Maturity Metric – Scoring
• Each vulnerability can be scored based on
type, and how easy it is to discover.
• Researchers find simple vulnerabilities first - as
simple vulnerabilities are eliminated, researchers
move on to finding more complex vulnerabilities.
• When a vendor secures the code, basic
vulnerabilities are easier to spot and remedy or
never introduce compared to more complex
vulnerabilities.
NOT JUST SECURITY , THE RIGHT SECURITY

66. Code Maturity Metric – Scoring Example
Level
Vulnerability Classes
0
Classic buffer overflows due to
e.g. strcpy, sprintf, sscanf and
format string issues.
1
Buffer overflows due to incorrect
size being used e.g. strncpy,
memcpy and array-indexing
issues
2
Arithmetic errors i.e. Integer
overflows/underflows, type
conversion, signedness.
3
Uninitialized variable, use-afterfree, bad cast, complex logic
errors.
NOT JUST SECURITY , THE RIGHT SECURITY

67. Schneider Modbus Serial Driver Buffer Overflow
Source: http://www.riskbasedsecurity.com/research/RBS-2013-003.pdf
NOT JUST SECURITY , THE RIGHT SECURITY

68. Schneider Modbus Serial Driver Buffer Overflow
NOT JUST SECURITY , THE RIGHT SECURITY

69. Schneider Modbus Serial Driver Buffer Overflow
Code Maturity
Level: 1
NOT JUST SECURITY , THE RIGHT SECURITY

70. Schneider Modbus Serial Driver Buffer Overflow
NOT JUST SECURITY , THE RIGHT SECURITY

71. Schneider Modbus Serial Driver Buffer Overflow
NOT JUST SECURITY , THE RIGHT SECURITY

72. ActiveX Control Vulnerability
Code Maturity
Level: 3
NOT JUST SECURITY , THE RIGHT SECURITY

73. Office Vulnerabilities Analysed
Office 2000:
Office XP:
62
103
Office 2003:
90
Office 2007:
47
Office 2010:
14
NOT JUST SECURITY , THE RIGHT SECURITY

74. Office Product Code Maturity Scores
Office 2010
Office 2007
Office 2003
Code Maturity
Office XP
Office 2000
0
0.5
1
1.5
2
2.5
NOT JUST SECURITY , THE RIGHT SECURITY
3

75. Office Vulnerability Type Prevalence
Office 2010
Office 2007
Uninitialised Variable
Object Type Confusion
Use-after-free
Office 2003
Arithmetic
Array Indexing
Incorrect Size Copy
Office XP
Classic Buffer Overflow
Office 2000
0%
5%
10%
15%
20%
25%
30%
35%
NOT JUST SECURITY , THE RIGHT SECURITY

76. Measuring the Efforts Taken By Vendors
With this we can put more focus on the code security
improvement efforts taken by vendors by being able to
measure them.
Allows system administrators to know which software to steer
clear from... and researcher to understand which types of
vulnerabilities they can expect to find in a given product.
NOT JUST SECURITY , THE RIGHT SECURITY

77. Advisory Quality
Or Lack Thereof...

78. Information Needs To Be Publicly Available
Most vendors have also acknowledged that publishing
vulnerability information is beneficial
Juniper recently joined the party
Still some black sheep like SAP, trying to keep it a secret…
NOT JUST SECURITY , THE RIGHT SECURITY

79. Needs To Include Vulnerability Type
Either clearly descripting the vulnerability type in the advisory
description
or
alternatively including CWEs
NOT JUST SECURITY , THE RIGHT SECURITY

80. Everything Is Memory Corruption These Days
NOT JUST SECURITY , THE RIGHT SECURITY

81. Microsoft MS12-037 vs MS13-080
----
NOT JUST SECURITY , THE RIGHT SECURITY

82. Rise In Usage Of Memory Corruption Term
NOT JUST SECURITY , THE RIGHT SECURITY

83. No requirements to include proper info
Various standards and formats e.g. CVRF are being
proposed, but these deal with required fields – not the
content of these.
Primary focus is to ensure a structure that is easy to parse in
an automated manner.
Completely up to the vendors how much information they
feel like sharing. Up to customers to raise their voice, if they
want/need more.
NOT JUST SECURITY , THE RIGHT SECURITY

84. Vulnerability
... And Bug Bounties
Handling

85. Bug Bounties
When I started reporting vulnerabilities to vendors, I was
stoked each time I actually got a response - and it wasn't a
threat from a lawyer.
Had any of you told me back then that vendors today would
be offering bug bounties, I'd have smiled and shook my head.
NOT JUST SECURITY , THE RIGHT SECURITY

86. Bug Bounties
A few interesting ones are of course Google's bounty, which is
one of the more serious vendor bounties, and especially their
latest twist: Bounties for other software!
Microsoft's bounty for vulnerabilities, but specifically
bypassing security mechanisms is very interesting
NOT JUST SECURITY , THE RIGHT SECURITY

87. Shockwave Player Vulnerability Trend
90
80
70
60
50
40
30
20
10
0
2003
2004
2005
2006
2007
2008
2009
2010
2011
NOT JUST SECURITY , THE RIGHT SECURITY
2012
2013

88. Bug Bounties
There has definitely been a shift in how vendors perceive bug
bounties.
It’s clear to me that if a vendor wants to encourage
researchers to look at their code and report it in a
coordinated manner, then bug bounties are very effective
when done right.
NOT JUST SECURITY , THE RIGHT SECURITY

89. Conclusion
Are We There Yet?

90. Security Software and Shiny Appliances
NOT JUST SECURITY , THE RIGHT SECURITY

91. Everything Is Vulnerable – Even Security Software!
About 2.2% of all entries in
OSVDB cover vulnerabilities in
security software
NOT JUST SECURITY , THE RIGHT SECURITY

92. The Security Software Paradox
Reducing attack surface by adding an even greater
attack surface is a paradox
NOT JUST SECURITY , THE RIGHT SECURITY

93. Code Quality Improvements(?)
Microsoft, Google, and Adobe are examples of vendors
noticeably improving their security efforts.
Oracle may be on their way after everyone finally realized that
Java is a mess...
NOT JUST SECURITY , THE RIGHT SECURITY

94. How Do We Force Vendors To Improve?
NOT JUST SECURITY , THE RIGHT SECURITY

95. Grand Demonstrations!
We need
that ordinary people can
relate to!
NOT JUST SECURITY , THE RIGHT SECURITY

96. FTC vs. TRENDnet
After demonstrating how network cameras were easily
publicly accessible and e.g. allowing spying on people in their
homes, the FTC (Federal Trade Commision) in USA went after
TRENDnet.
Eventually agreed that TRENDnet was ”prohibited from
misrepresenting the security of its cameras”, will establish a
comprehensive IS program, and hire outside consulting to
review security every two years for 20 years...
http://www.ftc.gov/opa/2013/09/trendnet.shtm
NOT JUST SECURITY , THE RIGHT SECURITY

97. Is TRENDnet worse than the rest?
This is really something every single software vendor should
do – but definitely don’t!
Is TRENDnet really that much worse than other embedded
device vendors?
NOT JUST SECURITY , THE RIGHT SECURITY

98. TRENDnet Product Vulnerabilities
NOT JUST SECURITY , THE RIGHT SECURITY

99. D-Link Product Vulnerabilities
NOT JUST SECURITY , THE RIGHT SECURITY

100. D-Link User-Agent Backdoor
Source: http://www.devttys0.com/2013/10/reverse-engineering-a-d-link-backdoor/
NOT JUST SECURITY , THE RIGHT SECURITY

101. Is Legislation The Answer?
NOT JUST SECURITY , THE RIGHT SECURITY

102. Software Will Always Have Vulns?
Vendors claim that they provide software ”as-is” and have
long EULAs to exempt them from liability
We seemingly accept that software will always have vulns...
... but the types of vulnerabilities matter as well as how the
vendor proactively reduces risk and reactively deals with
them.
NOT JUST SECURITY , THE RIGHT SECURITY

103. Conclusion
Of all the areas, vulnerability coordination/handling is the
biggest improvement and continuing in the right direction.
Advisory quality overall seems static with some vendors
improving and others devolving.
Only a few major vendors really seem to have solid SDLs and
can show an improvement in code quality.
People are beginning to understand metrics better, and we’re
seeing attempts at providing more granularity.
NOT JUST SECURITY , THE RIGHT SECURITY

104. The Good News: There is Room for Improvement
NOT JUST SECURITY , THE RIGHT SECURITY

105. Discussion!
NOT JUST SECURITY , THE RIGHT SECURITY