This document discusses using topic models to analyze trends in vulnerability data from Common Vulnerabilities and Exposures (CVE). Latent Dirichlet allocation is used to identify topics in CVE summaries from 2000-2009. Key trends identified include a decline in buffer overflows and format strings but an increase in SQL injection, cross-site scripting, and application servers. The topics identified generally align well with the Common Weakness Enumeration classifications.

1. © Microsoft Corporation
Security Trend Analysis
with CVE Topic Models
Stephan Neuhaus
Universita degli Studi di Trento, Italy
Thomas Zimmermann
Microsoft Research, Redmond, USA
ISSRE 2010, San Jose, CA, USA

2. © Microsoft Corporation
Background
http://www.microsoft.com/security/sir/default.aspx
Steve Christey, Robert A. Martin
http://cve.mitre.org/docs/vuln-trends/index.html
http://www.sans.org/top-cyber-security-risks/

3. © Microsoft Corporation
Can we automate the trend analysis of
security reports?

4. © Microsoft Corporation
Trend Analysis on Vulnerability Data
Raw Data Cleaning Topic Models Trends

5. © Microsoft Corporation
Data Sources
• Common Vulnerabilities and Exposures
– Hosted by MITRE (large US research company
and defense contractor)
– Clearinghouse for vulnerabilities: Assigns IDs to
vulnerabilities and collects descriptions
• National Vulnerability Database
– Annotated version of the CVE data
– Downloadable from NIST

6. © Microsoft Corporation
CVE Overview
• Earliest CVE has a published date of
October 1, 1988 (CVE 1999-0095)
– Reports the sendmail DEBUG hole that was
exploited by the Morris worm
• Latest CVEs in our dataset are from
December 31, 2009
• Total of 39,743 entries, 350 duplicates
• 39,393 unique CVEs remain

7. © Microsoft Corporation
Number of CVEs

8. © Microsoft Corporation
Summary
Impact
Classification
Date
ID

9. © Microsoft Corporation
Document Processing
Stack-based buffer overflow in vpnconf.exe in TheGreenBow
IPSec VPN Client 4.51.001, 4.65.003, and possibly other
versions, allows user-assisted remote attackers to execute
arbitrary code via a long OpenScriptAfterUp parameter in a
policy (.tgb) file, related to "phase 2."

10. © Microsoft Corporation
Document Processing
Stack-based buffer overflow in vpnconf.exe in TheGreenBow
IPSec VPN Client 4.51.001, 4.65.003, and possibly other
versions, allows user-assisted remote attackers to execute
arbitrary code via a long OpenScriptAfterUp parameter in a
policy (.tgb) file, related to "phase 2."
STOP WORDS
Remove
common
words

11. © Microsoft Corporation
Stack-based buffer overflow in vpnconf.exe in TheGreenBow
IPSec VPN Client 4.51.001, 4.65.003, and possibly other
versions, allows user-assisted remote attackers to execute
arbitrary code via a long OpenScriptAfterUp parameter in a
policy (.tgb) file, related to "phase 2."
Stack-based buffer overflow in vpnconf.exe in TheGreenBow
IPSec VPN Client 4.51.001, 4.65.003, and possibly other
versions, allows user-assisted remote attackers to execute
arbitrary code via a long OpenScriptAfterUp parameter in a
policy (.tgb) file, related to "phase 2."
i
i
Document Processing
STOP WORDS
Remove
common
words
STEMMER
Remove
word suffixes

12. © Microsoft Corporation
Length of CVE summary

13. © Microsoft Corporation
Topic Analysis
CVE
Application Server
Buffer Overflow
Cross-site
Scripting

14. © Microsoft Corporation
Topic Analysis
• Based on “Latent Dirichlet Allocation”
• Documents (CVEs) are bags of words
– Word order does not matter
• Words are assigned to (different) topics
– Probabilistic assessment
• We compute topics for all years 2000-2009
– Use post-hoc probabilities to find the fraction of
CVEs about a given topic in a given year

15. © Microsoft Corporation
Topic Example
overflow 6.29 execut 6.15 buffer 5.74 arbitrari 5.55 code 4.77
remot 4.29 command 3.05 long 2.83 craft 1.31 file 1.30
script 14.31 html 7.26 cross-sit 7.15 xss 6.92 web 6.73
inject 6.62 vulner 6.48 arbitrari 6.34 remot 5.73 paramet 4.47
11.3%
9.2%
Buffer Overflow
Cross-site Scripting

16. © Microsoft Corporation
28 Topics
http://www.wordle.net/show/wrdl/2674704/Relative_Importance_of_Security_Topics_identified_from_CVEs
2009

17. © Microsoft Corporation
Trends
Topic Trend 2000 2009
Application Servers 1% 5%
Arbitrary Code (PHP) 0% 2%
Buffer Overflow 19% 11%
Cross-Site Scripting 0% 10%
Link Resolution 6% 1%
Privilege Escalation 12% 2%
Resource Management 14% 10%
SQL Injection 1% 10%

18. © Microsoft Corporation
Cause and Impact
“x allows someone to y”
BEA WebLogic Portal 10.0 and 9.2 through MP1,
when an administrator deletes a single instance of a
content portlet, removes entitlement policies for
other content portlets
bypass intended access restrictions.
which allows attackers to

19. © Microsoft Corporation
Cause and Impact
“x allows someone to y”
BEA WebLogic Portal 10.0 and 9.2 through MP1,
when an administrator deletes a single instance of a
content portlet, removes entitlement policies for
other content portlets
bypass intended access restrictions.
which allows attackers to
CAUSE

20. © Microsoft Corporation
Cause and Impact
“x allows someone to y”
BEA WebLogic Portal 10.0 and 9.2 through MP1,
when an administrator deletes a single instance of a
content portlet, removes entitlement policies for
other content portlets
bypass intended access restrictions.
which allows attackers to
CAUSE
IMPACT

21. © Microsoft Corporation
24 Topics for “Cause”
http://www.wordle.net/show/wrdl/2674788/Relative_Importance_of_Security_%22Cause%22_Topics_identified_from_CVEs
2009

22. © Microsoft Corporation
Trends for “Cause”
Topic Trend 2000 2009
Buffer Overflow 17% 10%
Cross-Site Scripting 11% 17%
PHP 5% 8%
SQL Injection 10% 21%

23. © Microsoft Corporation
12 Topics for “Impact”
http://www.wordle.net/show/wrdl/2674920/Relative_Importance_of_Security_%22Impact%22_Topics_identified_from_CVEs
2009

24. © Microsoft Corporation
Trends for “Impact”
Topic Trend 2000 2009
Arbitrary Code 15% 24%
Arbitrary Script 17% 35%
Denial of Service 30% 11%
Information Leak 22% 15%
Privilege Escalation 11% 7%

25. © Microsoft Corporation
Common Weakness Enumeration
• Supposed to be a complete dictionary for
software weaknesses
• There are lots of available CWEs (659)
• But only 19 are used in CVE entries,
73% of CVE entries have a CWE field
• How well do our LDA topics align with the
manual CWE classification?
Classification

26. © Microsoft Corporation
Alignment with CWEs
Precision Recall LDA Topic Name
Only those that mapped to CWEs

27. © Microsoft Corporation
Alignment with CWEs
Precision Recall LDA Topic Name
SQL Injection
Cross-Site Scripting
Directory Traversal
Link Resolution
Format String
Buffer Overflow
Resource Management
Cross-Site Request Forgery
Information Leak
Cryptography
Credentials Management
Arbitrary Code

28. © Microsoft Corporation
Alignment with CWEs
Precision Recall LDA Topic Name
97.8 94.6 SQL Injection
98.1 85.4 Cross-Site Scripting
93.1 85.6 Directory Traversal
57.6 80.1 Link Resolution
51.8 75.3 Format String
60.1 57.6 Buffer Overflow
29.7 49.3 Resource Management
24.9 54.5 Cross-Site Request Forgery
33.1 18.6 Information Leak
28.0 18.0 Cryptography
12.1 38.7 Credentials Management
14.2 8.7 Arbitrary Code

29. © Microsoft Corporation
Buffer Overflow
• CVE 2008-0090:
– “A certain ActiveX control in npUpload.dll in DivX
Player 6.6.0 allows remote attackers to cause a
denial of service (Internet Explorer 7 crash) via a
long argument to the SetPassword method.”
– Possible classifiers: input validation, resource
management, credentials management
– Assigned classifier: buffer overflow

30. © Microsoft Corporation
Buffer Overflow
• Vulnerability descriptions are sometimes
not very specific
• CWEs are not mutually exclusive (there is
“buffer overflow” and “arbitrary code”)
• CWE assignment is not quality-checked
• Only one CWE can be assigned, even
when the CVE is about multiple issues

31. © Microsoft Corporation
Get the Data and Scripts!
http://tomz.me/issre2010cve
(Or follow the link in the paper.)

32. © Microsoft Corporation
PHP: declining, with occasional SQL injection.
Buffer Overflows: flattening out after decline.
Format Strings: in steep decline.
SQL Injection and XSS: remaining strong, and rising.
Cross-Site Request Forgery: a sleeping giant perhaps, stirring.
Application Servers: rising steeply.
http://msrconf.org

33. © Microsoft Corporation
Thank you!

34. © Microsoft Corporation
Mining Software Repositories 2011
http://msrconf.org