The document discusses metrics for prioritizing security efforts. It proposes using attack surfaces, which are all paths for data and commands in software, to identify code most likely to have vulnerabilities. The approach analyzes crash dump stack traces to determine which code is involved in crashes, representing active parts of the attack surface. Analysis of Windows, Firefox, and Fedora crashes found this approach covered the majority of known vulnerabilities. Undergraduate research further classified vulnerability types covered and visualized the data. The approach was found to outperform alternative models using software metrics or text mining alone or in combination.

1. METRICS FOR
SECURITY EFFORT
PRIORITIZATION
Christopher Theisen

2. AGENDA
◦ Motivate Security Metrics
◦ Solution: Attack Surfaces
◦ Results
◦ Undergraduate Contributions
◦ Future Work

4. Motivation | Solution | Results | Undergraduates | Future Work

5. Where are the
vulnerabilities?
Motivation | Solution | Results | Undergraduates | Future Work

6. Motivation | Solution | Results | Undergraduates | Future Work

7. Motivation | Solution | Results | Undergraduates | Future Work

8. Motivation | Solution | Results | Undergraduates | Future Work

9. [2] https://arstechnica.com/information-technology/2017/02/microsoft-hosts-the-windows-source-in-a-monstrous-300gb-git-repository/
Windows: 300GB repository, millions of source code files
Motivation | Solution | Results | Undergraduates | Future Work

10. 780,000
Cybersecurity Jobs in the U.S. - 2017
3,500,000
Estimated Cybersecurity openings worldwide - 2021
350,000
Cybersecurity openings worldwide - 2017
https://cybersecurityventures.com/jobs/
Motivation | Solution | Results | Undergraduates | Future Work

11. Motivation | Solution | Results | Undergraduates | Future Work
Tons of code…
Rare, expensive vulnerabilities…
Not enough people to find them…
How do we prioritize code for security testing/inspection?
http://www.classroomnook.com/2017/08/dealing-with-teacher-overwhelm.html

12. ATTACK SURFACE
Definition:
◦ All paths for data and commands in a software
system
◦ The data that travels these paths
◦ The code that implements and protects both
Concept used for security effort prioritization.
Hard to measure in practice…
Motivation | Solution | Results | Undergraduates | Future Work

13. RISK-BASED ATTACK SURFACE APPROXIMATION
Crashes represent activity that put the system under stress.
Stack Traces tell us what happened.
foo!foobarDeviceQueueRequest+0x68
foo!fooDeviceSetup+0x72
foo!fooAllDone+0xA8
bar!barDeviceQueueRequest+0xB6
bar!barDeviceSetup+0x08
bar!barAllDone+0xFF
center!processAction+0x1034
center!dontDoAnything+0x1030
Pull out individual code
artifacts from traces.
If code appears on a
crash dump stack
trace, it’s on the attack
surface.
Motivation | Solution | Results | Undergraduates | Future Work

14. Motivation | Solution | Results | Undergraduates | Future Work
Crashes are used by attackers!
Great source of forensic data.

15. HYPOTHESIS
◦ Crashes are empirical evidence of…
▫ Paths through the system with flaws
▫ Data paths through software
◦ Code appearing on crashes is therefore…
▫ More likely to have vulnerabilities
▫ Vulnerabilities more likely to be exploited
Expectation: High percentage of code with
vulnerabilities also crashes.
Motivation | Solution | Results | Undergraduates | Future Work

16. Motivation | Solution | Results | Undergraduates | Future Work

17. Motivation | Solution | Results | Undergraduates | Future Work

18. Motivation | Solution | Results | Undergraduates | Future Work

19. Motivation | Solution | Results | Undergraduates | Future Work
Use known vulnerabilities as an “oracle” to measure effectiveness

20. CRASHES – WINDOWS, FIREFOX, FEDORA
◦ Covering majority of vulnerable files
▫ Focus testing and review effort on crashing files
▫ Large amount of code that is irrelevant for
security review
Code
Coverage
Vulnerability
Coverage
Windows (Binaries) 48.4% 94.8%
Firefox (Source Files) 14.8% 85.6%
Fedora (Packages) 8.9% 63.3%
Motivation | Solution | Results | Undergraduates | Future Work

21. DATA SCALE
◦ Windows – 10 million crashes
◦ Firefox – 1.2 million crashes
◦ Fedora – 47 million crashes
◦ Common complaint:
▫ “We don’t have that much data!”
◦ How does this approach scale down?
Motivation | Solution | Results | Undergraduates | Future Work

22. DATA SCALE
71%
72%
73%
74%
75%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
PercentofVulnerabilitiesonaStackTrace
Random Sample Size

23. COMPARING VULNERABILITY PREDICTION MODELS
What about other approaches?
◦ Crash Dump Stack Traces
◦ Software Metrics
▫ Lines of Code, Code Churn, # of Developers…
◦ Text Mining
▫ Do specific words/strings mean more
vulnerabilities?
Motivation | Solution | Results | Undergraduates | Future Work

24. COMPARING VULNERABILITY PREDICTION MODELS
True
Positives
Vulnerability
Coverage
Crashes 5% 86%
Text Mining 1% 74%
Software Metrics 13% 42%
True
Positives
Vulnerability
Coverage
Crashes+Text Mining 22% 34%
Crashes+Software Metrics 18% 33%
Text Mining+Software Metrics 1% 85%
Crashes+Text+S. Metrics 23% 36%
Motivation | Solution | Results | Undergraduates | Future Work

25. CONCLUSIONS
◦ Other approaches need an “oracle” of vulnerabilities
to build their model.
▫ Known set of vulnerabilities so the model “knows”
what a vulnerability looks like.
◦ Crashes have no such restriction; single metric beats
out or equals models with 10’s or 100’s of metrics
◦ Better to optimize for Vulnerability Coverage.
▫ Professionals at Microsoft, etc. agree!
Motivation | Solution | Results | Undergraduates | Future Work

26. UNDERGRADUATE RESEARCH

27. UNDERGRADUATE RESEARCH
Motivation | Solution | Results | Undergraduates | Future Work
What kinds of vulnerabilities are we covering?
Dawson Tripp – 3rd year undergraduate from NCSU
◦ Learned some basic exploits
◦ Developed classification scheme (CWE w/ caveats)
◦ Mined vulnerabilities from Firefox
◦ Classified vulnerabilities with two graduate students

28. UNDERGRADUATE CONTRIBUTIONS
Motivation | Solution | Results | Undergraduates | Future Work
What kinds of vulnerabilities are we covering?
Dawson Tripp – 3rd year undergraduate from NCSU
Still using his code today!

29. UNDERGRADUATE CONTRIBUTIONS
Motivation | Solution | Results | Undergraduates | Future Work
Practical application: visualize the data
Dalisha Rodriguez – 2nd year undergraduate from
Hanover (now Fayetteville State)

30. Motivation | Solution | Results | Undergraduates | Future Work

31. NEXT STEPS
Motivation | Solution | Results | Undergraduates | Future Work
◦ More public vulnerability datasets
◦ We need metrics that model attacker behavior
◦ Dynamic (runtime) metrics!
▫ Think of crashes as dynamic: generated at
runtime
▫ Count “messages” to/from/between objects
(AspectJ)
▫ Combine with techniques like fuzzing
◦ Chamoli et al. – dynamic metrics for defect prediction
▫ Better on vulnerabilities…?

32. Motivation | Solution | Results | Undergraduates | Future Work
Tons of code…
Rare, expensive vulnerabilities…
Not enough people to find them…
http://www.classroomnook.com/2017/08/dealing-with-teacher-overwhelm.html

33. MASSIVELY OPEN ONLINE COURSES
◦ “Flipped” NCSU’s Software Security course
▫ Iteration 1: ~400 students
▫ Iteration 2: ~1100 students
◦ Lessons Learned:
▫ Amplifies gulf between veterans and newbies
▫ Assume no more than 2-3 hours of work/week
▫ Important to “seed” discussions
Motivation | Solution | Results | Undergraduates | Future Work

34. MASSIVELY OPEN ONLINE COURSES
◦ “Flipped” NCSU’s Software Security course
▫ Iteration 1: ~400 students
▫ Iteration 2: ~1100 students
◦ Lessons Learned:
▫ Amplifies gulf between veterans and newbies
▫ Assume no more than 2-3 hours of work/week
▫ Important to “seed” discussions
Motivation | Solution | Results | Undergraduates | Future Work

35. SoftSec Materials publicly available!
https://tinyurl.com/ncsu-softsec

36. SoftSec Materials publicly available!
https://tinyurl.com/ncsu-softsec
theisen.cr@gmail.com
theisencr.github.io

37. RELATED WORK
Data in this talk:
In Submission: [SECDEV ‘18] Chris Theisen, Hyunwoo Song, Dawson Tripp, Laurie Williams,
“What Are We Missing? Vulnerability Coverage by Type”
In Submission: [FSE ‘18] Chris Theisen, Hyunwoo Song, Dalisha Rodriguez, Laurie Williams,
“Better Together: Comparing Vulnerability Prediction Models”
[ICSE – SEIP ‘17] Chris Theisen, Kim Herzig, Laurie Williams, “Risk-Based Attack Surface
Approximation: How Much Data is Enough?”
[ICSE - SEIP ‘15] Chris Theisen, Kim Herzig, Pat Morrison, Brendan Murphy, and Laurie
Williams, “Approximating Attack Surfaces with Stack Traces”, in Companion Proceedings of the
37th International Conference on Software Engineering
Other works:
Revising: [IST] Chris Theisen, Nuthan Munaiah, Mahran Al-Zyoud, Jeffery C. Carver, Laurie
Williams, “Attack Surface Definitions”
[HotSoS ’17] Chris Theisen and Laurie Williams, “Advanced Metrics for Risk-Based Attack
Surface Approximation”, Proceedings of the Symposium and Bootcamp on the Science of Security.
[FSE – SRC ’15 – 1st Place] Chris Theisen, “Automated Attack Surface Approximation”, in the
23rd ACM SIGSOFT International Symposium on the Foundations of Software Engineering -
Student Research Competition, 2015
[SRC Grand Finals ’15 – 3rd Place] Chris Theisen, “Automated Attack Surface Approximation”, in
the 23rd ACM SIGSOFT International Symposium on the Foundations of Software Engineering -
Student Research Competition, 2015

38. IDENTIFYING VULNERABLE CODE
#include <stdio.h>
int main(int argc, char **argv)
{
char buf[8]; // buffer for eight characters
gets(buf); // read from stdio (sensitive!)
printf("%sn", buf); // print out data in buf
return 0; // 0 as return value
}
https://www.owasp.org/index.php/Buffer_overflow_attack
Motivation | Solution | Results | Undergraduates | Future Work