Talk given at BSides LV on trends in SQL injection flaw introduction over time. A look into the data to see that everyone makes mistakes, but when developers are given access to a consequence free environment to scan their applications, and the tools and resources to learn how to fix the flaws, they do.
2. Allow me to introduce myself
So, I work at Veracode.
We scan a lot of applications.
I’ve been working with scan results, of one kind of another, for
the past five year.
3. Also about a year ago
I needed a research topic for Ming Chow’s security class at
Tufts University.
Thought about “fixing” WebGoat.
He asked me to think about something else.
4. State of Software Security
Veracode has a lot of data
Mostly annual report on trends across applications we scan.
Some findings are surprisingly consistent over time.
https://www.veracode.com/sites/default/files/Resources/Reports/state-of-
software-security-volume-7-veracode-report.pdf
5. SQL Injection
Flaw type I wanted to fix in WebGoat.
Fairly prevalent.
Veracode finds it in static scans, and can detect fixes.
6. I had made some assumptions
Looked at flaw prevalence data over previous SOSS reports.
The number didn’t change very much.
Why was it so flat? For years?
7. Measuring the prevalence of SQLi
The measurement refers to the presence of at least one SQL
injection flaw on first static scan.
It is one of the standard measurements that gets reported in
the SOSS.
9. SQLi prevalence on first scan
All first static scans between
2013 and first half of 2017.
Mean: 31.9%
SD: 0.36%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2013 2014 2015 2016 2017
With SQLi Without SQLi
10. SQLi prevalence rate across orgs
Same data, but looking across
17 randomly selected
organizations.
11. Fix rate by application on 3rd scan
App fix rate
Some/All fixed flaws 44%
No net-fixed flaws 56%
13. Room to Play
As scans move to sandboxes,
the prevalence of SQLi in
policy scans on first scan goes
down.
Prevalence for all first scans
remains the same.
14. Fix rates in context
Study published by O’Reilly and SIG.
Survey questions about use of code quality tools.
~32% of respondents reported fixing 80%+ of issues found
using code quality tools.
Only 11% of respondents reported fixing less than 20% of the
issues.
https://www.sig.eu/insight/improving-code-quality/
15. Does anything affect fix rates?
2016 State of Software Security did report on two factors
that appear to influence fix rates.
1. Remediation Coaching
2. eLearning subscriptions
https://www.veracode.com/sites/default/files/Resources/Reports/state-of-
software-security-volume-7-veracode-report.pdf
16. It's like we're helping.
Developers fix flaws when they’re found.
When developers have access to consequence
free scans, learning tools and help, fix rates get
even better.
Security working in conjunction with
development can allow both teams to succeed.
Hi,I’m Sarah
I work at Veracode, if you’re not familiar with what we do, we offer application security testing and services to business. Our main testing solutions are static binary analysis, and dynamic web application scanning.
We scan a lot. Doing hundreds of thousands of scans in the last year alone (get number).
I’ve done a couple different things at Veracode over the past five years, bouncing back and forth between services, operations, and engineering. Getting different perspectives on how people scan, and what we find.
This story starts about a year ago, the last time I changed jobs. I moved back to a customer facing role, helping developers understand and fix the flaws we find.
New job and last class of my program at Tufts.
I knew Ming “assigned” a research paper each semester, and you just needed to pick an interesting topic. I was starting to help people fix what they find, but didn’t have a lot of experience actually doing that myself. I thought about fixing a known vulnerable web app as a way to get into the mind of a developer having to fix a legacy application.
Ming’s review of my proposal asked if I could instead take a look at Veracode’s SOSS reports over the past couple years and see if there was anything interesting in the data over time. (I guess he wasn’t into an auto-ethnographic study of developers receiving security reports)
What is the SOSS?
?
Turns out there was something in there. I went looking for changes in fix rates, what I found was a lack of change in flaw introduction rates.
Quick detour.
I was still interested in fix rates. If I wasn’t going to better understand developers by fixing flaws myself, I wanted to see if I could get an understanding through the data.
In order to look at fix rates I needed to get an idea about what flaw introduction rates looked like.
The flaw prevalence number was really steady, I wasn’t sure how it was measured.
At first I assumed it was across all scans in a given year. My first hypothesis was that while flaw introduction rates were going down, flaw fix rates were non-existent. This would suck.
I was wrong. Flaw introduction rates were not going down.
So this is why.
The prevalence metric is only looking at first scan. It does not look at subsequent scans, and contains no fix data. At all.
If it was flat, then developers weren’t reducing flaw introduction rates over time.
We spend a lot of time thinking about prevention, and there are organizations that are very proactive in working with their developers on security, was it really flat?
Off by one errors.Etc
I expected all of what has happened in the last five years would have affected the rate of flaw introduction, and it hasn’t. IT’s the same.
150,000 scans
SQLi prevalence on first scan is the same as it was in 2013.
Kind of huge.
Avg Flaw count
44.10363
SD
456.2882
Median
5
Orgs are kept anonymous in data.
Pulled first 50 scan entries, removed duplicates, removed those orgs with less than 100 applications.
Trend line data?
Mean?
Clustering around 30%, which is what we would expect if previous graph is representative of the greater population and not just an artifact of something.
Sample description (three scans or more of similar analysis size)*
*similar defined as being defined as either a 90% similar set, or those values within one SD of mean.
Individual flaw fix rate 48%
Of applications in sample, 44% had a decrease in SQLi flaw count after 3 scans.
When including customer mitigations, 51% had a decrease in SQLi flaw count after 3 scans
“Consequence Free”
What is the Sandbox feature?
What does it allow devs and security to do?
This looks like it is affecting policy scan SQLi prevalence, why?
Room to experiment, to scan whatever, to learn. To do better.
What is my actual 80% fix rate by third scan number?
23% all fix
34% of the sample looked at had 80% or more of their SQLi flaws fixed by the third scan of similar size.
That lines up really nice.
That 11% compares to more than a 50% slice of the sample that had less than 20% fixed, no change, or introduced new flaws.
This will change.
Totally recent survey by ESG and Veracode 53% of respondents said that they felt that security and dev were working collaboratively in their org.
“Have some Sympathy”
Or some other rolling stones pun.
None of this stuff changes the rate at which we make mistakes, however giving access to these tools and resources to developers makes all the difference in the world.
Take aways, automated tools to find their own stuff, they will use them and fix things!If you give them coaching, they will fix them faster cause they know what they’re doing.Neither of these things stops mistakes from being written in the first place.