1. From Shaman to Scientist:
A Use Case in Data Driven Security
2. Nice To Meet You
About Me
CoFounder HoneyApps
Former CISO Orbitz
Contributing Author
Beautiful Security
CSO Magazine/Online Writer
HoneyApps
Vulnerability Management as a Service
16 Hot Startups - eWeek
3 Startups to Watch - Information Week
4. Stage 2: Where are all of my vulnerabilities?
“Back in my Yahoo days I performed hundreds of web
application vulnerability assessments. To streamline the
workload, I created an assessment methodology consisting
of a few thousand security tests averaging 40 hours to
complete per website. Yahoo had over 600 websites
enterprise-wide. To assess the security of every website
would have taken over 11 years to complete and the other
challenge was these websites would change all the time
which decayed the value of my reports.”
Jeremiah Grossman
Founder, WhiteHat Security
5. Stage 3: Scan & Dump
“thanks for the 1000 page report,
now what?!”
6. Why This Occurs
Lack of Communication
Lack of Data
Lack of Coordination
Silos, Silos, Everywhere
7. Stage 4: A New Beginning
Or......
Using What You Got!
8. Vulnerability Management: A Case Study
Building the Warehouse
Structured Data Load
WebApp Vulnerability
Type: XSS
Severity
Threat
Subtype: (persistent,reflected,etc)
Asset URL/URI
Confirmed?
Dates Found/Opened
Dates Closed
Description
Attack Parameters
9. Vulnerability Management: A Case Study
Building the Warehouse
Structured Data Load
WebApp Vulnerability Asset:URL
Type: XSS Platform / Code
Severity Web Server Version
Threat Application Server Version
Subtype: (persistent,reflected,etc) Database Version
Asset URL/URI
Confirmed?
Dates Found/Opened
Dates Closed
Description
Attack Parameters
10. Vulnerability Management: A Case Study
Building the Warehouse
Structured Data Load
WebApp Vulnerability Asset:URL
Type: XSS Platform / Code
Severity Web Server Version
Threat Application Server Version
Subtype: (persistent,reflected,etc) Database Version
Asset URL/URI
Confirmed? Asset:Host
Dates Found/Opened Host Operating System
Dates Closed Other Applications/Versions
Description IP Addresses
Attack Parameters Mac Address
Open Services/Ports
11. Vulnerability Management: A Case Study
Unstructured Data Load
WebApp Vulnerability
Asset:URL
Type: XSS Platform / Code
Severity Web Server Version
Threat Application Server Version
Database Version
Subtype: (persistent,reflected,etc)
Asset URL/URI
Asset:Host
Confirmed?
Dates Found/Opened
Host Operating System
Dates Closed Applications/Versions
Other
Description Addresses
IP
Attack Parameters
Mac Address
Open Services/Ports
12. Vulnerability Management: A Case Study
Unstructured Data Load
Meta Data
WebApp Vulnerability
Asset:URL
Type: XSS Platform / Code
Severity Web Server Version Business Unit
VERIS data
Threat Application Server Version Internal IP Address
Database Version
Subtype: (persistent,reflected,etc) Geographic Location External IP Address
Asset URL/URI
Asset:Host
Confirmed?
Development Team Network Location
Dates Found/Opened
Host Operating System Ops Team Site Name
Dates Closed Applications/Versions
Other Compliance Regulation
Description Addresses
IP Security Policy
Asset Group
Attack Parameters
Mac Address
Open Services/Ports
13. Vulnerability Management: A Case Study
Loosely Structured Data Load
Meta Data
WebApp Vulnerability
Asset:URL
Type: XSS Platform / Code
Severity Web Server Version
ThreatUnit Application Server Version
Business
VERIS data
Database Version
Subtype: (persistent,reflected,etc)
Internal IP Address
Asset URL/URI
Geographic Location
Confirmed? Asset:Host
External IP Address
DB HTTP
Development Team Network Location
Dates Found/Opened
Host Operating System
Dates Closed Applications/Versions
Ops Team Other Site Name
Description Addresses
IP
Compliance Regulation
Attack Parameters
Mac Address
Security Policy
Asset Group
Open Services/Ports
14. Vulnerability Management: A Case Study
Loosely Structured Data Load
Meta Data
WebApp Vulnerability
Asset:URL Apply Internal Threat Data
Type: XSS Platform / Code
Severity Web Server Version
ThreatUnit Application Server Version
Business
VERIS data Firewall Application
Database Version
Subtype: (persistent,reflected,etc)
Internal IP Address
Asset URL/URI
Geographic Location
Confirmed? Asset:Host
External IP Address
DB HTTP
Development Team Network Location
Dates Found/Opened
Host Operating System
Dates Closed Applications/Versions
Ops Team Other
Description Addresses
IP
Compliance Regulation
Site Name
IDS/IPS WAF
Attack Parameters
Mac Address
Security Policy
Asset Group
Open Services/Ports
15. Vulnerability Management: A Case Study
Mixed Data Set
Meta Data
WebApp Vulnerability
Asset:URL
Type: XSS Platform / Code
Severity Web Server Version
Apply Internal Threat
Threat Unit Application Server Version
Business
Internal IP Address
VERIS data
Database Version
Subtype: (persistent,reflected,etc)
Asset URL/URI
Geographic Location External IP Address
Firew
Asset:HostApplicati
Confirmed? Team
Development Network Location
Dates Found/Opened
Host Operating System
Dates Team Other Applications/Versions
Ops
Closed Site Name
IDS/ IPCompliance Regulation
Description Addresses
WA
Asset Group
Attack Parameters
Security Mac Address
Policy
Open Services/Ports
16. Vulnerability Management: A Case Study
Mixed Data Set
Apply External Threat Data
Meta Data
WebApp Vulnerability
Asset:URL
Type: XSS Platform / Code
Severity Web Server Version
Apply Internal Threat
Threat Unit Application Server Version
Business
Internal IP Address
VERIS data
Database Version
Subtype: (persistent,reflected,etc)
Asset URL/URI
Geographic Location External IP Address
Firew
Asset:HostApplicati
Confirmed? Team
Development Network Location
Dates Found/Opened
Host Operating System
Dates Team Other Applications/Versions
Ops
Closed Site Name
IDS/ IPCompliance Regulation
Description Addresses
WA
Asset Group
Attack Parameters
Security Mac Address
Policy
Open Services/Ports
17. Vulnerability Management: A Case Study
Mixed Data Set
Apply External Threat Data
Meta Data
WebApp Vulnerability
Asset:URL Example Data Sources
Type: XSS Platform / Code
Severity Web Server Version
Apply Internal Threat
Threat Unit Application Server Version
Business
Internal IP Address
VERIS data ❖DataLossDB
Database Version
Subtype: (persistent,reflected,etc) ❖Verizon DBIR
Asset URL/URI
Geographic Location External IP Address
Firew
Asset:HostApplicati
Confirmed? Team
❖WHID
Development
Dates Found/Opened
Host Operating System
Network Location
❖Trustwave Global Security Report
Dates Team Other Applications/Versions
Ops
Closed Site Name ❖FS-ISAC
IDS/ IPCompliance Regulation
Description Addresses ❖SANS ISC
WA
Asset Group
Attack Parameters
Security Mac Address
❖Veracode State of S/W Security
Policy
Open Services/Ports
❖ExploitDB
18. Vulnerability Management: A Case Study
Unstructured Data Load
WebApp Vulnerability
Asset:URL
Meta Data
Type: XSS Platform / Code
Severity Web Server Version
Threat Application Server Version
Apply Internal Threat
Database Version
Subtype: (persistent,reflected,etc)
Business Unit
Internal IP Address
VERIS data
Asset URL/URI
Asset:Host
Geographic Location External IP Address
Confirmed?
Firew Applicati
Dates Found/Opened Network Location
Host Operating System
Development Team
Dates Closed Applications/Versions
Ops TeamOther Site Name
Description Compliance Regulation
IP Addresses
IDS/
Attack Parameters
Mac Address Asset Group
WA
Security Policy
Open Services/Ports
19. Vulnerability Management: A Case Study
Unstructured Data Load
WebApp Vulnerability
Asset:URL
Meta Data
Type: XSS Platform / Code
Severity Web Server Version Remediation Statistics
Threat Application Server Version
Apply Internal Threat
Database Version
Subtype: (persistent,reflected,etc)
Business Unit
Internal IP Address
VERIS data
Asset URL/URI Internal Bug Tracking Reports
Asset:Host
Geographic Location External IP Address
Confirmed?
Firew Applicati
Dates Found/Opened Network Location
Host Operating System
Development Team
Denim Group Remediation Study
Dates Closed Applications/Versions
Ops TeamOther Site Name
Description Compliance Regulation
IP Addresses
IDS/
Attack Parameters
Mac Address Asset Group
Build and Development Process
WA
Security Policy
Open Services/Ports
21. Low Hanging Fruit
vulns >10% of external breaches
>10% of our malicious traffic
in scope for $regulation
sort
22. Secure Code Training Metric
all vulns by application*
vulns opened before 12/31/10**
vulns opened after 2/28/11**
*sort by vulnerability class
**secure code training rolled out 1/1/11 - 2/28/11
28. Got MSSP?
The Alex Hutton Formula
My(vuln posture * other threat activity) / (other vuln posture * other threat activity)
29. Got MSSP?
The Alex Hutton Formula
My(vuln posture * other threat activity) / (other vuln posture * other threat activity)
OR
When Will Our Luck Run Out?
36. Resources Referenced
Verizon DBIR http://www.verizonbusiness.com/dbir/WASC Web App Security Stats
http://projects.webappsec.org/w/page/
VERIS Framework https://www2.icsalabs.com/veris/ 13246989/Web-Application-Security-Statistics
Denim Group - Real Cost of S/W FS-ISAC http://www.fsisac.com/
Remediation
WHID http://projects.webappsec.org/w/page/
http://www.slideshare.net/denimgroup/real-cost-of-
13246995/Web-Hacking-Incident-Database/
software-remediation
SANS Internet Storm Center
DataLoss DB http://datalossdb.org/ http://isc.sans.org/
TrustWave Global Security Report XForce http://xforce.iss.net/
https://www.trustwave.com/GSR
Veracode SOSS http://www.veracode.com/
images/pdf/soss/veracode-state-of-software-
ExploitDB
security-report-volume2.pdf
http://www.exploit-db.com/
37. Q&A
follow us
the blog
http://blog.honeyapps.com/
twitter
@ebellis And one more thing....
@risk_io We’re Hiring! https://www.risk.io/jobs
Editor's Notes
From Shaman to Scientist - A Use Case in Data Driven Security\n
\n
An increasingly rare species, dominant in the 90’s and around the turn of the century.\n
\n
Enter the age of the automated scanners! \n
We NEED more New School! There is data out there to be had but a Lack of communication & coordination create this perceived\n“lack of data”\n
the first step towards the New School of Information Security. Baby steps towards a quant approach. Using less secrecy & religion and more \nopenness and information sharing. In order to take the first steps, we have to get our own house in order.\n
Vulnerability management is an easy use case for taking these first steps. Let’s start walking through these steps.\n
Vulnerability management is an easy use case for taking these first steps. Let’s start walking through these steps.\n
\n
\n
\n
\n
Time to Fix by team, by class, by severity, by biz unit,etc,etc\nSDLC - Build Schedule - testing process - etc,etc\nTech Remediation Stats from Denim Report - factor in bug tracking reports & build/dev process\n
convert this into a visual funnel to produce same queries \n\nhd moores law should be shown as a node map highlighted.\n
Insert Alex Hutton formula\n4:14:40 PM Alex Hutton: once the data hits catwalk we decision and the threat is funneled off....\n4:15:46 PM Ed Bellis: haha its a new risk language\n4:16:05 PM Ed Bellis: you guys pull a lot of public sources of data?\n4:16:16 PM Alex Hutton: not yet\n4:16:21 PM Alex Hutton: some but not a ton\n4:16:24 PM Ed Bellis: also any insight on whether VERIS taking off?\n4:16:41 PM Alex Hutton: there's a broader vision (you can fit) that would make a lot of sense\n4:17:02 PM Alex Hutton: but that would take cooperation from various sources which is going to be hard, and why VERIS is a bit stalled\n4:17:17 PM Alex Hutton: there are lots of folks using VERIS, but not sharing data\n4:17:31 PM Ed Bellis:\n4:17:51 PM Ed Bellis: i'd love to tap into that\n4:19:34 PM Alex Hutton: well, the better vision is for you to feed data along with threat (and incident data) into decision making on a daily or even hourly basis\n4:19:50 PM Alex Hutton: In a sense, you hold not only the orgs perimeter, but a broad sample of perimeters\n4:20:04 PM Alex Hutton: and their\n4:20:12 PM Alex Hutton: vuln posture\n4:20:38 PM Alex Hutton: So\n4:21:07 PM Ed Bellis: right we want to move into a position where our data becomes the primary asset to help make better security decisions\n4:21:16 PM Ed Bellis: *not a verb*\n4:21:49 PM Alex Hutton: My(vuln posture * other threat activity) / (other vuln posture * other threat activity) = a much better metric than just vuln data\n4:22:24 PM Ed Bellis: meaning am i safer than my neighbor?\n4:22:25 PM Alex Hutton: in fact, a very interesting likelihood data point to do some degree of probabilistic analysis on that I don't have the time to really explore yet, but is very, very interesting\n4:23:10 PM Alex Hutton: that may be a byproduct, more about establishing a rate at which I can expect my luck to run out.\n4:23:23 PM Ed Bellis: i see\n4:23:35 PM Alex Hutton: so you're the CSO of Orbitz, you know you have a vuln. to specific threat activity.  So far, you're lucky\n4:24:02 PM Alex Hutton: but what about establishing rate of threat activity vs. how pervasive that same vuln is around other orgs?\n4:24:31 PM Ed Bellis: where do we get the rate of threat activity?\n4:24:42 PM Ed Bellis: breach db's?\n4:24:51 PM Alex Hutton: High Threat Activity + Low pervasiveness in the aggregate population = Luck running out.\n4:25:06 PM Alex Hutton: nope.  that's part of your exit strategy\n4:25:19 PM Ed Bellis: bought by verizon ?\n4:25:33 PM Alex Hutton: Some MSSP who is running or aggregating WAF data, Firewall/IDS/IPS data\n4:25:58 PM Alex Hutton: you as "Yet another managed service" well, that's a scenario I believe in, yes.\n4:26:20 PM Alex Hutton: But you as "Next generation in managed services by data aggregation" is going to be very, very compelling, I think\n4:26:46 PM Alex Hutton: what you have to understand (and I think you do) is how to leverage your data into "decision" (the verb)\n
\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
talk about infosec vs fraud\n
Insert Alex Hutton formula\n4:14:40 PM Alex Hutton: once the data hits catwalk we decision and the threat is funneled off....\n4:15:46 PM Ed Bellis: haha its a new risk language\n4:16:05 PM Ed Bellis: you guys pull a lot of public sources of data?\n4:16:16 PM Alex Hutton: not yet\n4:16:21 PM Alex Hutton: some but not a ton\n4:16:24 PM Ed Bellis: also any insight on whether VERIS taking off?\n4:16:41 PM Alex Hutton: there's a broader vision (you can fit) that would make a lot of sense\n4:17:02 PM Alex Hutton: but that would take cooperation from various sources which is going to be hard, and why VERIS is a bit stalled\n4:17:17 PM Alex Hutton: there are lots of folks using VERIS, but not sharing data\n4:17:31 PM Ed Bellis:\n4:17:51 PM Ed Bellis: i'd love to tap into that\n4:19:34 PM Alex Hutton: well, the better vision is for you to feed data along with threat (and incident data) into decision making on a daily or even hourly basis\n4:19:50 PM Alex Hutton: In a sense, you hold not only the orgs perimeter, but a broad sample of perimeters\n4:20:04 PM Alex Hutton: and their\n4:20:12 PM Alex Hutton: vuln posture\n4:20:38 PM Alex Hutton: So\n4:21:07 PM Ed Bellis: right we want to move into a position where our data becomes the primary asset to help make better security decisions\n4:21:16 PM Ed Bellis: *not a verb*\n4:21:49 PM Alex Hutton: My(vuln posture * other threat activity) / (other vuln posture * other threat activity) = a much better metric than just vuln data\n4:22:24 PM Ed Bellis: meaning am i safer than my neighbor?\n4:22:25 PM Alex Hutton: in fact, a very interesting likelihood data point to do some degree of probabilistic analysis on that I don't have the time to really explore yet, but is very, very interesting\n4:23:10 PM Alex Hutton: that may be a byproduct, more about establishing a rate at which I can expect my luck to run out.\n4:23:23 PM Ed Bellis: i see\n4:23:35 PM Alex Hutton: so you're the CSO of Orbitz, you know you have a vuln. to specific threat activity.  So far, you're lucky\n4:24:02 PM Alex Hutton: but what about establishing rate of threat activity vs. how pervasive that same vuln is around other orgs?\n4:24:31 PM Ed Bellis: where do we get the rate of threat activity?\n4:24:42 PM Ed Bellis: breach db's?\n4:24:51 PM Alex Hutton: High Threat Activity + Low pervasiveness in the aggregate population = Luck running out.\n4:25:06 PM Alex Hutton: nope.  that's part of your exit strategy\n4:25:19 PM Ed Bellis: bought by verizon ?\n4:25:33 PM Alex Hutton: Some MSSP who is running or aggregating WAF data, Firewall/IDS/IPS data\n4:25:58 PM Alex Hutton: you as "Yet another managed service" well, that's a scenario I believe in, yes.\n4:26:20 PM Alex Hutton: But you as "Next generation in managed services by data aggregation" is going to be very, very compelling, I think\n4:26:46 PM Alex Hutton: what you have to understand (and I think you do) is how to leverage your data into "decision" (the verb)\n
Insert Alex Hutton formula\n4:14:40 PM Alex Hutton: once the data hits catwalk we decision and the threat is funneled off....\n4:15:46 PM Ed Bellis: haha its a new risk language\n4:16:05 PM Ed Bellis: you guys pull a lot of public sources of data?\n4:16:16 PM Alex Hutton: not yet\n4:16:21 PM Alex Hutton: some but not a ton\n4:16:24 PM Ed Bellis: also any insight on whether VERIS taking off?\n4:16:41 PM Alex Hutton: there's a broader vision (you can fit) that would make a lot of sense\n4:17:02 PM Alex Hutton: but that would take cooperation from various sources which is going to be hard, and why VERIS is a bit stalled\n4:17:17 PM Alex Hutton: there are lots of folks using VERIS, but not sharing data\n4:17:31 PM Ed Bellis:\n4:17:51 PM Ed Bellis: i'd love to tap into that\n4:19:34 PM Alex Hutton: well, the better vision is for you to feed data along with threat (and incident data) into decision making on a daily or even hourly basis\n4:19:50 PM Alex Hutton: In a sense, you hold not only the orgs perimeter, but a broad sample of perimeters\n4:20:04 PM Alex Hutton: and their\n4:20:12 PM Alex Hutton: vuln posture\n4:20:38 PM Alex Hutton: So\n4:21:07 PM Ed Bellis: right we want to move into a position where our data becomes the primary asset to help make better security decisions\n4:21:16 PM Ed Bellis: *not a verb*\n4:21:49 PM Alex Hutton: My(vuln posture * other threat activity) / (other vuln posture * other threat activity) = a much better metric than just vuln data\n4:22:24 PM Ed Bellis: meaning am i safer than my neighbor?\n4:22:25 PM Alex Hutton: in fact, a very interesting likelihood data point to do some degree of probabilistic analysis on that I don't have the time to really explore yet, but is very, very interesting\n4:23:10 PM Alex Hutton: that may be a byproduct, more about establishing a rate at which I can expect my luck to run out.\n4:23:23 PM Ed Bellis: i see\n4:23:35 PM Alex Hutton: so you're the CSO of Orbitz, you know you have a vuln. to specific threat activity.  So far, you're lucky\n4:24:02 PM Alex Hutton: but what about establishing rate of threat activity vs. how pervasive that same vuln is around other orgs?\n4:24:31 PM Ed Bellis: where do we get the rate of threat activity?\n4:24:42 PM Ed Bellis: breach db's?\n4:24:51 PM Alex Hutton: High Threat Activity + Low pervasiveness in the aggregate population = Luck running out.\n4:25:06 PM Alex Hutton: nope.  that's part of your exit strategy\n4:25:19 PM Ed Bellis: bought by verizon ?\n4:25:33 PM Alex Hutton: Some MSSP who is running or aggregating WAF data, Firewall/IDS/IPS data\n4:25:58 PM Alex Hutton: you as "Yet another managed service" well, that's a scenario I believe in, yes.\n4:26:20 PM Alex Hutton: But you as "Next generation in managed services by data aggregation" is going to be very, very compelling, I think\n4:26:46 PM Alex Hutton: what you have to understand (and I think you do) is how to leverage your data into "decision" (the verb)\n
Insert Alex Hutton formula\n4:14:40 PM Alex Hutton: once the data hits catwalk we decision and the threat is funneled off....\n4:15:46 PM Ed Bellis: haha its a new risk language\n4:16:05 PM Ed Bellis: you guys pull a lot of public sources of data?\n4:16:16 PM Alex Hutton: not yet\n4:16:21 PM Alex Hutton: some but not a ton\n4:16:24 PM Ed Bellis: also any insight on whether VERIS taking off?\n4:16:41 PM Alex Hutton: there's a broader vision (you can fit) that would make a lot of sense\n4:17:02 PM Alex Hutton: but that would take cooperation from various sources which is going to be hard, and why VERIS is a bit stalled\n4:17:17 PM Alex Hutton: there are lots of folks using VERIS, but not sharing data\n4:17:31 PM Ed Bellis:\n4:17:51 PM Ed Bellis: i'd love to tap into that\n4:19:34 PM Alex Hutton: well, the better vision is for you to feed data along with threat (and incident data) into decision making on a daily or even hourly basis\n4:19:50 PM Alex Hutton: In a sense, you hold not only the orgs perimeter, but a broad sample of perimeters\n4:20:04 PM Alex Hutton: and their\n4:20:12 PM Alex Hutton: vuln posture\n4:20:38 PM Alex Hutton: So\n4:21:07 PM Ed Bellis: right we want to move into a position where our data becomes the primary asset to help make better security decisions\n4:21:16 PM Ed Bellis: *not a verb*\n4:21:49 PM Alex Hutton: My(vuln posture * other threat activity) / (other vuln posture * other threat activity) = a much better metric than just vuln data\n4:22:24 PM Ed Bellis: meaning am i safer than my neighbor?\n4:22:25 PM Alex Hutton: in fact, a very interesting likelihood data point to do some degree of probabilistic analysis on that I don't have the time to really explore yet, but is very, very interesting\n4:23:10 PM Alex Hutton: that may be a byproduct, more about establishing a rate at which I can expect my luck to run out.\n4:23:23 PM Ed Bellis: i see\n4:23:35 PM Alex Hutton: so you're the CSO of Orbitz, you know you have a vuln. to specific threat activity.  So far, you're lucky\n4:24:02 PM Alex Hutton: but what about establishing rate of threat activity vs. how pervasive that same vuln is around other orgs?\n4:24:31 PM Ed Bellis: where do we get the rate of threat activity?\n4:24:42 PM Ed Bellis: breach db's?\n4:24:51 PM Alex Hutton: High Threat Activity + Low pervasiveness in the aggregate population = Luck running out.\n4:25:06 PM Alex Hutton: nope.  that's part of your exit strategy\n4:25:19 PM Ed Bellis: bought by verizon ?\n4:25:33 PM Alex Hutton: Some MSSP who is running or aggregating WAF data, Firewall/IDS/IPS data\n4:25:58 PM Alex Hutton: you as "Yet another managed service" well, that's a scenario I believe in, yes.\n4:26:20 PM Alex Hutton: But you as "Next generation in managed services by data aggregation" is going to be very, very compelling, I think\n4:26:46 PM Alex Hutton: what you have to understand (and I think you do) is how to leverage your data into "decision" (the verb)\n
