#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
DefCamp 2013 - Are we there yet?
1. 10 Years Later:
Are We There Yet?
Carsten Eiram
Risk Based Security
@CarstenEiram
2. Quick Bio – VDB Work Experience
Involved with VDBs for 10+ years
• Currently, CRO at Risk Based Security – commercial arm of
Open Security Foundation (runs OSVDB and DatalossDB) – and
responsible for the VulnDB service.
• Chief Security Specialist at Secunia, running the Research team.
• Security Team Lead at Danish Verisign affiliate, running a
customer-only accessible vulnerability database.
NOT JUST SECURITY , THE RIGHT SECURITY
3. Quick Bio – Vulnerability Research
Officially been doing vulnerability research since 2003
• Focused on a static analysis / reverse engineering approach
• Jokingly refer to myself as a "vulnerability connoisseur"
- I enjoy analyzing vulnerabilities and their root causes.
• Critical vulnerabilities discovered in products from many major
software vendors.
NOT JUST SECURITY , THE RIGHT SECURITY
5. Reason for Talk
After 10+ years of VDB work,
I felt it was time to reflect
on certain areas related to
vulnerabilities
NOT JUST SECURITY , THE RIGHT SECURITY
14. Currently Oldest Recorded Vulnerabilities
Vulnerabilities have been around for a very long time
- And will continue to be...
• Oldest entries in OSVDB are 79399 and 79400
• Marconi wireless telegraph
• Dated November 1902
• Message spoofing and message disclosure
NOT JUST SECURITY , THE RIGHT SECURITY
16. First Ever Unbreakable Claim!
http://www.newscientist.com/article/mg21228440.700-dotdashdiss-the-gentleman-hackers-1903-lulz.html
NOT JUST SECURITY , THE RIGHT SECURITY
17. Nevil Maskelyne Ruins Demo
http://www.newscientist.com/article/mg21228440.700-dotdashdiss-the-gentleman-hackers-1903-lulz.html
NOT JUST SECURITY , THE RIGHT SECURITY
18. No Wire-Cutting Please
While not providing
the privacy and
security as promised,
the wireless telegraph
still had one significant
advantage over the
wired telegraph:
Not possible to cut the
wires!
NOT JUST SECURITY , THE RIGHT SECURITY
19. Have We Improved?
Obviously, we have progressed a fair bit
technically since then, but have we gotten
significantly better?
NOT JUST SECURITY , THE RIGHT SECURITY
20. Bringing The Internet Down – Old Lady Style
Article: http://news.softpedia.com/news/Old-Lady-Cuts-Off-Internet-in-Armenia-193640.shtml
NOT JUST SECURITY , THE RIGHT SECURITY
21. 10 Year Vulnerability Trend
12000
10000
8000
6000
# Vulns
4000
2000
0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
NOT JUST SECURITY , THE RIGHT SECURITY
22. All Datasets Are Incomplete!
All datasets are incomplete - some just more than others
Many love taking CVE content that’s free and do random
conclusions based on it, but since the dataset is severely
lacking, the conclusions are as well
NOT JUST SECURITY , THE RIGHT SECURITY
23. 2006 – 2013 Vulnerability Type Trend
NOT JUST SECURITY , THE RIGHT SECURITY
24. 2012 Data Breaches due to SQL Injection
NOT JUST SECURITY , THE RIGHT SECURITY
25. Companies affected by XSS in 2012
Source: CWN - http://www.cyberwarnews.info/2012/07/04/300000-personal-details-leaked-38-sites-hacked-for-projectdragonfly/
NOT JUST SECURITY , THE RIGHT SECURITY
28. Which is more secure?
Product A
10 Vulnerabilities
Product B
20 Vulnerabilities
NOT JUST SECURITY , THE RIGHT SECURITY
29. Security State != Number of Vulnerabilities
Previously, the security state of a product was considered
to be equal to the number of vulnerabilities.
Flawed conclusion!
Today, people understand that the number of vulnerabilities
!=
security state
NOT JUST SECURITY , THE RIGHT SECURITY
30. Some Apparently Still Don’t Know...
“The problem with Java is that a lot of vulnerabilities are
constantly being reported in it, and when a lot of
vulnerabilities are reported, then there are a lot of hackers
using these to access programs built on Java“
- Morten Stengaard, CTO, Secunia
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY
31. Dissecting the Statement – Part 1
”... then there are a lot of hackers
using these to access programs built on Java”
Most vulnerabilities in Java are not used to target Java
applications, but the Java Runtime Environment to
compromise the system.
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY
32. Dissecting the Statement – Part 2
”... when a lot of vulnerabilities are reported,
then there are a lot of hackers using these…”
Just because a lot of vulnerabilities are reported in a
product, a lot of hackers may not be exploiting them.
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY
33. Dissecting the Statement – Part 3
”The problem with Java is that a lot of vulnerabilities are
constantly being reported in it…”
The security state of a product is not defined by the number
of vulnerabilities reported in it.
http://www.dr.dk/tv/se/tv-avisen/tv-avisen-827#!/
NOT JUST SECURITY , THE RIGHT SECURITY
34. We Should All Stop Using Popular Software Then
400
350
300
250
Java
200
Chrome
Firefox
150
Internet Explorer
100
50
0
Vulnerabilities (2013 - Nov 10th)
NOT JUST SECURITY , THE RIGHT SECURITY
38. Microsoft Office Vulnerability Trend
Vulnerabilities in Office versions one year after product release
(based on Microsoft security bulletins)
14
12
10
8
6
4
2
0
Office 2000
Office 2007
NOT JUST SECURITY , THE RIGHT SECURITY
Office 2010
39. Microsoft Security Bulletin Trend
350
300
250
200
Bulletins
150
CVEs
100
50
0
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
NOT JUST SECURITY , THE RIGHT SECURITY
42. There are so many other aspects to consider!
More things to consider incl.
Patched vs. Unpatched
Vulnerability Type
Impact
Time-To-Patch
Time-To-Vendor-Response
Security Mechanisms
...
NOT JUST SECURITY , THE RIGHT SECURITY
44. Severity Metrics
Many different severity metrics – both public and internal
Most popular and hated is CVSS,
which currently has problems reflecting real risk
Many concerns raised about CVSSv2 by many people
e.g. myself and Brian Martin of OSVDB in our open letter:
"The CVSSv2 Shortcomings, Faults, and Failures Formulation"
http://www.riskbasedsecurity.com/reports/CVSS-ShortcomingsFaultsandFailures.pdf
NOT JUST SECURITY , THE RIGHT SECURITY
45. Limitations of Severity Metrics
Reflecting the threat of vulnerability-dependent issues
(e.g. sandbox bypass, ASLR bypass related to memory
disclosure etc.)
By themselves and from a scoring point-of-view, these issues
are pretty minor, but when combined with code execution...
Jackpot!
Ability to disclose a few memory addresses was in the past
pretty much a non-issue – today it’s very useful.
NOT JUST SECURITY , THE RIGHT SECURITY
46. Pick A Vuln... Any Vuln...
If I’d offer you one vulnerability
in e.g. Google Chrome, which
would you pick?
1) Code execution within
sandbox
CVSSv2:
6.8
2) Sandbox bypass
CVSSv2:
2.6
NOT JUST SECURITY , THE RIGHT SECURITY
47. Severity Metrics and Sandbox Bypasses
If we conclude that exploiters are more interested in the
sandbox bypass and system administrators should focus on
fixing such a vulnerability over a code execution
vulnerability within the sandbox, why are we not rating
them higher?
Case of reality not being reflected well by severity metrics
NOT JUST SECURITY , THE RIGHT SECURITY
48. Severity Metrics and Vulnerability Chains
And once these issues start occuring in chains, which is
becoming more and more common, then it really gets
complex...
You can have a lot of independent minor issues that when
combined suddenly are very serious
NOT JUST SECURITY , THE RIGHT SECURITY
49. Google Chrome Pwn2Own Example
CVSSv2:
6.8
CVSSv2:
5.1
OSVDB 89734
IPC channel missing
listener process validation
OSVDB 80007
Plugin blocking logic not run
for NaCl in pre-rendering
CVSSv2:
7.6
OSVDB 80293
Unpacked NPAPI extension
installation without
confirmation
CVSSv2:
OSVDB 81645
5.1
GPU command decoding
integer underflow
CVSSv2:
2.6
OSVDB 80741
CVSSv2:
2.6
OSVDB 89736
Too permissive
LoadExtension
bindings for
extension manager
Unprivileged renderer
can navigate to
privileged URLs
http://blog.chromium.org/2012/05/tale-of-two-pwnies-part-1.html
NOT JUST SECURITY , THE RIGHT SECURITY
54. Microsoft Approach: Pros and Cons
Pros
Cons
Gives an realistic evaluation of
the technical requirements to
exploit a given vulnerability and
how feasible it is
Requires significant technical
skills and resources to get right
Makes it clear which are
theoretical and which are
plausible
Still requires a bit of
guesstimation
NOT JUST SECURITY , THE RIGHT SECURITY
56. How Does Adobe Do It?
NOT JUST SECURITY , THE RIGHT SECURITY
57. How Does Adobe Do It?
NOT JUST SECURITY , THE RIGHT SECURITY
58. Adobe Approach: Pros and Cons
Pros
...Cons
Allows understanding which
Does not factor in technical
products, versions, and
requirements and the nature of
architectures are most critical to the vulnerability i.e. does not
prioritize
differentiate between theoretical
issues and straight-forward issues
to exploit
Dynamic approach that can be
easily tweaked
Requires very little resources –
just an understanding of historical
exploitation
NOT JUST SECURITY , THE RIGHT SECURITY
59. How Does CVSSv2 Do It?
NOT JUST SECURITY , THE RIGHT SECURITY
60. CVSSv2 Approach: Pros and Cons
Pros
Most reliable of all the
approaches: If an exploit is
available, a vulnerability is clearly
exploitable.
Requires very little resources –
just knowledge of availability of
PoCs and exploits
Cons
Purely reactive, requiring very
fast response times
Only takes into account when the
availability of an exploit is
publicly known i.e. may be
exploited long before being
flagged as such
NOT JUST SECURITY , THE RIGHT SECURITY
63. Code Quality – Why Measure It?
NOT JUST SECURITY , THE RIGHT SECURITY
64. Code Maturity Metric – The Idea
The idea of code maturity is that by evaluating
the prevalence of the different vulnerability
classes being discovered in a product, we can
conclude the maturity of that product.
We, naturally, focus on it from a security
perspective.
NOT JUST SECURITY , THE RIGHT SECURITY
65. Code Maturity Metric – Scoring
• Each vulnerability can be scored based on
type, and how easy it is to discover.
• Researchers find simple vulnerabilities first - as
simple vulnerabilities are eliminated, researchers
move on to finding more complex vulnerabilities.
• When a vendor secures the code, basic
vulnerabilities are easier to spot and remedy or
never introduce compared to more complex
vulnerabilities.
NOT JUST SECURITY , THE RIGHT SECURITY
66. Code Maturity Metric – Scoring Example
Level
Vulnerability Classes
0
Classic buffer overflows due to
e.g. strcpy, sprintf, sscanf and
format string issues.
1
Buffer overflows due to incorrect
size being used e.g. strncpy,
memcpy and array-indexing
issues
2
Arithmetic errors i.e. Integer
overflows/underflows, type
conversion, signedness.
3
Uninitialized variable, use-afterfree, bad cast, complex logic
errors.
NOT JUST SECURITY , THE RIGHT SECURITY
67. Schneider Modbus Serial Driver Buffer Overflow
Source: http://www.riskbasedsecurity.com/research/RBS-2013-003.pdf
NOT JUST SECURITY , THE RIGHT SECURITY
74. Office Product Code Maturity Scores
Office 2010
Office 2007
Office 2003
Code Maturity
Office XP
Office 2000
0
0.5
1
1.5
2
2.5
NOT JUST SECURITY , THE RIGHT SECURITY
3
75. Office Vulnerability Type Prevalence
Office 2010
Office 2007
Uninitialised Variable
Object Type Confusion
Use-after-free
Office 2003
Arithmetic
Array Indexing
Incorrect Size Copy
Office XP
Classic Buffer Overflow
Office 2000
0%
5%
10%
15%
20%
25%
30%
35%
NOT JUST SECURITY , THE RIGHT SECURITY
76. Measuring the Efforts Taken By Vendors
With this we can put more focus on the code security
improvement efforts taken by vendors by being able to
measure them.
Allows system administrators to know which software to steer
clear from... and researcher to understand which types of
vulnerabilities they can expect to find in a given product.
NOT JUST SECURITY , THE RIGHT SECURITY
78. Information Needs To Be Publicly Available
Most vendors have also acknowledged that publishing
vulnerability information is beneficial
Juniper recently joined the party
Still some black sheep like SAP, trying to keep it a secret…
NOT JUST SECURITY , THE RIGHT SECURITY
79. Needs To Include Vulnerability Type
Either clearly descripting the vulnerability type in the advisory
description
or
alternatively including CWEs
NOT JUST SECURITY , THE RIGHT SECURITY
80. Everything Is Memory Corruption These Days
NOT JUST SECURITY , THE RIGHT SECURITY
82. Rise In Usage Of Memory Corruption Term
NOT JUST SECURITY , THE RIGHT SECURITY
83. No requirements to include proper info
Various standards and formats e.g. CVRF are being
proposed, but these deal with required fields – not the
content of these.
Primary focus is to ensure a structure that is easy to parse in
an automated manner.
Completely up to the vendors how much information they
feel like sharing. Up to customers to raise their voice, if they
want/need more.
NOT JUST SECURITY , THE RIGHT SECURITY
85. Bug Bounties
When I started reporting vulnerabilities to vendors, I was
stoked each time I actually got a response - and it wasn't a
threat from a lawyer.
Had any of you told me back then that vendors today would
be offering bug bounties, I'd have smiled and shook my head.
NOT JUST SECURITY , THE RIGHT SECURITY
86. Bug Bounties
A few interesting ones are of course Google's bounty, which is
one of the more serious vendor bounties, and especially their
latest twist: Bounties for other software!
Microsoft's bounty for vulnerabilities, but specifically
bypassing security mechanisms is very interesting
NOT JUST SECURITY , THE RIGHT SECURITY
87. Shockwave Player Vulnerability Trend
90
80
70
60
50
40
30
20
10
0
2003
2004
2005
2006
2007
2008
2009
2010
2011
NOT JUST SECURITY , THE RIGHT SECURITY
2012
2013
88. Bug Bounties
There has definitely been a shift in how vendors perceive bug
bounties.
It’s clear to me that if a vendor wants to encourage
researchers to look at their code and report it in a
coordinated manner, then bug bounties are very effective
when done right.
NOT JUST SECURITY , THE RIGHT SECURITY
91. Everything Is Vulnerable – Even Security Software!
About 2.2% of all entries in
OSVDB cover vulnerabilities in
security software
NOT JUST SECURITY , THE RIGHT SECURITY
92. The Security Software Paradox
Reducing attack surface by adding an even greater
attack surface is a paradox
NOT JUST SECURITY , THE RIGHT SECURITY
93. Code Quality Improvements(?)
Microsoft, Google, and Adobe are examples of vendors
noticeably improving their security efforts.
Oracle may be on their way after everyone finally realized that
Java is a mess...
NOT JUST SECURITY , THE RIGHT SECURITY
94. How Do We Force Vendors To Improve?
NOT JUST SECURITY , THE RIGHT SECURITY
96. FTC vs. TRENDnet
After demonstrating how network cameras were easily
publicly accessible and e.g. allowing spying on people in their
homes, the FTC (Federal Trade Commision) in USA went after
TRENDnet.
Eventually agreed that TRENDnet was ”prohibited from
misrepresenting the security of its cameras”, will establish a
comprehensive IS program, and hire outside consulting to
review security every two years for 20 years...
http://www.ftc.gov/opa/2013/09/trendnet.shtm
NOT JUST SECURITY , THE RIGHT SECURITY
97. Is TRENDnet worse than the rest?
This is really something every single software vendor should
do – but definitely don’t!
Is TRENDnet really that much worse than other embedded
device vendors?
NOT JUST SECURITY , THE RIGHT SECURITY
102. Software Will Always Have Vulns?
Vendors claim that they provide software ”as-is” and have
long EULAs to exempt them from liability
We seemingly accept that software will always have vulns...
... but the types of vulnerabilities matter as well as how the
vendor proactively reduces risk and reactively deals with
them.
NOT JUST SECURITY , THE RIGHT SECURITY
103. Conclusion
Of all the areas, vulnerability coordination/handling is the
biggest improvement and continuing in the right direction.
Advisory quality overall seems static with some vendors
improving and others devolving.
Only a few major vendors really seem to have solid SDLs and
can show an improvement in code quality.
People are beginning to understand metrics better, and we’re
seeing attempts at providing more granularity.
NOT JUST SECURITY , THE RIGHT SECURITY
104. The Good News: There is Room for Improvement
NOT JUST SECURITY , THE RIGHT SECURITY
We don’t specifically ”access programs built on Java”, but whatever we feel like that is accessible with the gained privileges.
When a vulnerability database / security companyfails like this, how can we expect the rest to understand?
MS forgot to show the Office stats, so I generated them based on their bulletins. Applying the same “logic”, then SDL apparently made Office worse?
And overall more vulnerabilities have been reported since SDL’s introduction? So it all went downhill?
October 21st SCADA vulnerabilities from 2005 to 2013. So does this trend mean that SCADA security was better years ago than it is today? Or does it rather demonstrate that researchers suddenly started focusing on it, as they could get their 5 min of fame + money?
While counting vulns is one of the most popular ways to ”document” the security state of a product, it’s terribly uninteresting in an isolated manner, as various factors affect it -most importantly researcher focus.
Any seasoned exploiter in this crowd would pick #2 without blinking even with a lesser severity rating. Turning the question around to system admins: Which would (should) you fix first, if you could apply just one patch?
Makes sense that these consider worst-case impacts, but as a system administrator you want more granularity.
Introduced October 2008 and improved May 2011. Evaluates how likely a vulnerability is exploited within 30 days. Evaluation is done by Microsoft and unspecified key partners.
Keyword is ”realistic”. Cannot be too conservative nor downplay issues or value is lost.
35 ”critical” bulletins and all but one scored 1. >80% of all 85 2012 bulletins had at least vulnerability scored as 1. Problem is that MS are too conservative and afraid of getting it wrong. Need to address that or reduce the 30 day window to e.g. 14 days or a week to add granularity.
Instead of assessing the technical feasability, Adobe assesses the likelihood based product and on historical data.
Actually not intended to be considered a metric, but commonly used as such.
Allows understanding the current security state of a products code – even down to a component level. Since code is code, it would allow comparing the code quality of different products.
Researchers (generally) find simple vulnerabilities first - as simple vulnerabilities are eliminated, researchers move on to finding more complex vulnerabilities. Same for fuzzing and static analysis.
You can still find basic vulnerabilities today. List of affected products
Another unbounded copy. Overwrite return address and flawless victory since not compiled with /GS flags.
ActiveX control’s dispatching function
Part of code maturity work done while at Secunia. Presented more in-depth at RVAsec 2012.
There is an improvement. May not seem significant, but due to the manner issues were scored.
Digging into numbers reveals significance. We can see the types changed a lot e.g. no more classic buffer overflows since Office 2007.
Interesting metric – especially combined with existing metrics to understand security of product.
Question vendors with such antiquated approaches like SAP, and if you find vulnerabilities in their products, make sure to publish the details.
Generic term when a vendor doesn’t want to disclose root cause and a researcher doesn’t know the root cause...
1996 to August 1st 2013. About 1.8% - not staggering, but actually 4% of disclosures made today are ”memory corruption”.
Believing in an evidence-based approach, we obviously prefer as much information as possible to make the best decisions.
Also good way for Microsoft to get focus on their beta software prior to stable release to ensure less customers are impacted
Starts receiving focus late 2009 / early 2010. Picks up momentum in 2010 after initial disclosures. Mid-2011 ZDI stops accepting submissions. Reports have dropped since then.
Security spending is increasing and this is where most spend their money. But can you trust these security devices?
Security software has vulnerabilities too. Don't automatically trust that software vendors developing security products know security nor secure coding!
Message is not: “Don’t use security products”, but instead: “Consider what security software you really need – and certainly which features you need enabled. Apply the same healthy skepticism to security products as any other products”.
Microsoft started improving security, because it started hurting their image. Adobe now recently started doing the same.
Marconi case is an example. Barnaby Jack was amazing at this with his medical device and ATM hacks. Show how it impacts people’s lives. Surveillance case was effective and people could relate to it.
Also lots of vulnerabilities – same basic type.
And that’s where code maturity metrics may help...
