Security as Code:
A Cloud Automation Use Case
Jason Rohwedder, Risk I/O DevOps Geek
Cloud Security World May 21, 2015
Ed Bellis, Risk I/O Cofounder & CTO
About Jason Rohwedder & Ed Bellis
Ed Bellis: Risk I/O Cofounder & CTO | @ebellis
★ Cofounder of Risk I/O
★ Former CISO of Orbitz, VP InfoSec of Bank of America
★ Recognized Security Expert & Evangelist: Black Hat, OWASP, Gartner, SaaScon &
SecTor
★ Contributing Author: Beautiful Security by O’Reilly Media; Writer/Blogger: CSO
Magazine, CSO Online, InfoSec Island
Jason Rohwedder: Risk I/O Geek & DevOps Specialist | @jasonrohwedder
★Born in Germany, Raised in IL, Resides in TN
★Responsible for All Things DevOps
★Leader of an Urban Zoo
★Once sucker punched a co-worker “just to watch him cry”
Are You In The Right Room?
• Business Value of Automation
• Automation Top Concerns
• Applying DevOps Principles at Risk I/O via Automation
• Security Automation Use Case
• Q & A
Business Value
or
“why are we doing this?”
High Performers Are More Agile
more frequent
deployments
faster lead times
than their peers
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
High Performers Are More Agile
the change
success rate
faster mean time to
recover (MTTR)
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
High Performers Win in the Marketplace
more likely to exceed
profitability, market
share & productivity
goals
higher market
capitalization
growth over 3
years
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
To Those Who Came Before
Us
We Salute You
10 Deploys per Day
Dev & Ops Cooperation at Flickr
John Allspaw & Paul Hammond
Velocity 2009
Source: John Allspaw (@allspaw) and Paul Hammond (@ph)
Justin Collins, Neil Matatall & Alex Smolen from
Twitter
• Decouple feature releases from code deployments
• Deploy features in a disabled state, using feature flags
• Require all developers to check code into trunk daily
• Integrate security testing at each code check
• The result is code always being in a secure, deployable state
• Practice deploying smaller changes, which dramatically
reduces risk and improves MTTR
Source: http://www.facebook.com/note.php?note_id=14218138919
Deploy Smaller Changes, More
Frequently
Inject Failures Often
The Netflix Tech Blog
Five Lessons We’ve Learned Using AWS
We’ve sometimes referred to the Netflix software architecture in AWS as our Rambo Architecture. Each
system has to be able to succeed, no matter what, even all on its own. We’re designing each distributed
system to expect and tolerate failure from other systems on which it depends.
One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s
job is to randomly kill instances & services within our architecture. If we aren’t constantly testing our
ability to succeed despite failure, then it isn’t likely to work when it matters most - in the event of an
unexpected outage.
You Don’t Choose Chaos Monkey…Chaos
Monkey Chooses You
With a Little Help from our Friends (at Netflix)
Break Things Before Production
• Enforce consistency in code, environments and
configurations across the environments
• Add your ASSERTs to find misconfigurations, enforce
https, etc.
• Add static code analysis to automated continuous
integration and testing process
• (Note: don’t forget to harden those CI boxes!)
Automation At Risk I/O
Risk I/O By the Numbers
Small & Frequent Commits
• Average between 75 & 125
commits commits to Master/week
• Simplicity is your friend
Risk I/O By the Numbers
Always Be Deploying (EVERY day, MANY times per day)
Security Automation at Risk I/O
Chef All the Things!
Test All the Things! (including security)
Static + Dynamic Throughout
Continuous Integration via CircleCI
Open-Sourced Cookbooks
ModSecurity Nessus Nmap SSH
iptables encrypted volumes Duo 2FA openVPN
ChatOps = Slack + graphite + logstash + sensu + pagerduty
A DevOps Use Case
Continuous Security Testing
• Brakeman Static Analysis
• Dynamic Application Scanning
• Paid Penetration Testing Service
• Open Bug Bounty Program
• Continuous Infrastructure Scanning
All Results Loaded into Risk I/O via API
DevOps as a Compliance Enabler
Automation as Evidence & Doc
Cookbooks
Leveraging the ELK Stack
Elasticsearch
Logstash
Kibana
Github + Code Climate + Risk I/O
Compliance Automation Extra Credit: https://telekomlabs.github.io/
@Eellis
This Ain’t No Dogfooding Exercise
The Risk I/O API
EndPoints:
Vulnerabilities
Assets
Tagging (metadata)
Patches
Connectors (integration & scanner control)
Threat Processing & Risk Meters
Experiment: 626
Build a host vulnerability scanner
…that doesn’t scan
Why? What’s Wrong with FooScan?
Can Be Slow
Can Be Intrusive
Requiring Scanning during Maintenance Windows :(
A Lot Can Happen In A Week
Depends On Timely Vulnerability Definition Updates
What About Ours?
1. Built On Automation
2. Made of Composable Ingredients
3. It Should be Non-Intrusive
4. It Should be Close to Real-Time
5. It Should Support Multiple Sources for Definitions
6. It Should Only Take ~3.14 Days to Create
(have I mentioned it’s NOT a scanner)
Our Base Ingredients
Automation
(Chef, Ansible, Docker + stuff, cron + SSH)
Reliable Software + Version Info
Vulnerability Definitions
NVD/CVE feed with CPE included
“cpe:/a:google:chrome:8.0.552.215" is vulnerable to something
Eggs, Milk, Flour and Salt
These are not security related
Gather host: software+version
Automation Framework + Scripting
we used chef + tattle
XYZ Monitoring Services
opsmatic, logstash, splunk…
System Packager
rpm, dkpkg, pacman…
Dockerfiles
What is tattle?
Ridiculously simple way to store and regurgitate data
tattle update software myapp --version 1.2.3 --installer ubuntu
tattle report software
> {“myapp”:{“version":"1.2.3","installer":"ubuntu"}}
# on github soon
Vulnerability Definitions
NVD CVE Feed with CPE data
https://nvd.nist.gov/download.cfm
<entry id="CVE-2011-0001">
...<vuln:cve-id>CVE-2011-0001</vuln:cve-id>
...<vuln:vulnerable-software-list>
<vuln:product>cpe:/a:zaal:tgt:1.0.1</vuln:product>
<vuln:product>cpe:/a:zaal:tgt:1.0.0</vuln:product>
...<vuln:summary>Double free vulnerability in ...
Mix It TogetherParse Some XML
$ cat nvdcve-2.0-2014.xml | ruby -e 'require "ox"; p
Ox.parse(ARGF.read).nvd.locate("entry").select {|x|
x.locate("*/vuln:product/*").member? "cpe:/a:oracle:mysql:5.5.26"
}.map(&:id)’
["CVE-2014-0384", "CVE-2014-0386", "CVE-AWW-YEAH",...]
What was that?
• We have mysql 5.5.26 installed
• Searching for it’s most specific CPE locator
• We found that there are a bunch of Vulnerabilities for it from
2014
What We Did
Extend Our Risk I/O API to Allow Software+Version (we already had the data
anyway…)
curl -H "Content-type: application/json" 
https://api.riskio.us/assets/63/software -X PUT -d "{
‘asset’: {
‘software’: $(tattle report software)
}
}"
On Upload of Software Data
• Update Assets & Create Associated Vulnerabilities
• Close Vulnerabilities for Software that’s been Updated
Cake!
Cake!
Icing on the Cake
Also Known As Next Steps…
• Integrate Existing 0 Day Feeds
• Extend upgrade/uninstall tracking to build a timeline
• Alert when new Vulnerabilities match Assets
(The Real-Time Thing)
The Cake Is A Lie
Reliability of CPEs for all data sources
needs further study
Vendor/Product/Version mismatches
dpkg says “zlib1g”, you say “zlib”
Packager whitelists / Packager specific CPEs
“Ubuntu fixed this with USN-1337-1”
Resources
• http://puppetlabs.com/2013-state-of-devops-infographic
• https://api.risk.io
• https://telekomlabs.github.io/
• https://github.com/jro/automated_security
• https://github.com/joeyschoblaska/brakeman-risk-io
• https://github.com/jro/tattle - coming soon!
We Are Hiring!
risk.io/jobs
@ebellis @jasonrohwedder

Security as Code

  • 1.
    Security as Code: ACloud Automation Use Case Jason Rohwedder, Risk I/O DevOps Geek Cloud Security World May 21, 2015 Ed Bellis, Risk I/O Cofounder & CTO
  • 2.
    About Jason Rohwedder& Ed Bellis Ed Bellis: Risk I/O Cofounder & CTO | @ebellis ★ Cofounder of Risk I/O ★ Former CISO of Orbitz, VP InfoSec of Bank of America ★ Recognized Security Expert & Evangelist: Black Hat, OWASP, Gartner, SaaScon & SecTor ★ Contributing Author: Beautiful Security by O’Reilly Media; Writer/Blogger: CSO Magazine, CSO Online, InfoSec Island Jason Rohwedder: Risk I/O Geek & DevOps Specialist | @jasonrohwedder ★Born in Germany, Raised in IL, Resides in TN ★Responsible for All Things DevOps ★Leader of an Urban Zoo ★Once sucker punched a co-worker “just to watch him cry”
  • 3.
    Are You InThe Right Room? • Business Value of Automation • Automation Top Concerns • Applying DevOps Principles at Risk I/O via Automation • Security Automation Use Case • Q & A
  • 4.
  • 5.
    High Performers AreMore Agile more frequent deployments faster lead times than their peers Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
  • 6.
    High Performers AreMore Agile the change success rate faster mean time to recover (MTTR) Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
  • 7.
    High Performers Winin the Marketplace more likely to exceed profitability, market share & productivity goals higher market capitalization growth over 3 years Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
  • 8.
    To Those WhoCame Before Us We Salute You
  • 9.
    10 Deploys perDay Dev & Ops Cooperation at Flickr John Allspaw & Paul Hammond Velocity 2009 Source: John Allspaw (@allspaw) and Paul Hammond (@ph)
  • 10.
    Justin Collins, NeilMatatall & Alex Smolen from Twitter
  • 11.
    • Decouple featurereleases from code deployments • Deploy features in a disabled state, using feature flags • Require all developers to check code into trunk daily • Integrate security testing at each code check • The result is code always being in a secure, deployable state • Practice deploying smaller changes, which dramatically reduces risk and improves MTTR Source: http://www.facebook.com/note.php?note_id=14218138919 Deploy Smaller Changes, More Frequently
  • 12.
    Inject Failures Often TheNetflix Tech Blog Five Lessons We’ve Learned Using AWS We’ve sometimes referred to the Netflix software architecture in AWS as our Rambo Architecture. Each system has to be able to succeed, no matter what, even all on its own. We’re designing each distributed system to expect and tolerate failure from other systems on which it depends. One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s job is to randomly kill instances & services within our architecture. If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most - in the event of an unexpected outage.
  • 13.
    You Don’t ChooseChaos Monkey…Chaos Monkey Chooses You
  • 14.
    With a LittleHelp from our Friends (at Netflix)
  • 15.
    Break Things BeforeProduction • Enforce consistency in code, environments and configurations across the environments • Add your ASSERTs to find misconfigurations, enforce https, etc. • Add static code analysis to automated continuous integration and testing process • (Note: don’t forget to harden those CI boxes!)
  • 16.
  • 17.
    Risk I/O Bythe Numbers Small & Frequent Commits • Average between 75 & 125 commits commits to Master/week • Simplicity is your friend
  • 18.
    Risk I/O Bythe Numbers Always Be Deploying (EVERY day, MANY times per day)
  • 19.
    Security Automation atRisk I/O Chef All the Things! Test All the Things! (including security) Static + Dynamic Throughout Continuous Integration via CircleCI Open-Sourced Cookbooks ModSecurity Nessus Nmap SSH iptables encrypted volumes Duo 2FA openVPN ChatOps = Slack + graphite + logstash + sensu + pagerduty
  • 20.
    A DevOps UseCase Continuous Security Testing • Brakeman Static Analysis • Dynamic Application Scanning • Paid Penetration Testing Service • Open Bug Bounty Program • Continuous Infrastructure Scanning All Results Loaded into Risk I/O via API
  • 21.
    DevOps as aCompliance Enabler Automation as Evidence & Doc Cookbooks Leveraging the ELK Stack Elasticsearch Logstash Kibana Github + Code Climate + Risk I/O Compliance Automation Extra Credit: https://telekomlabs.github.io/ @Eellis
  • 22.
    This Ain’t NoDogfooding Exercise The Risk I/O API EndPoints: Vulnerabilities Assets Tagging (metadata) Patches Connectors (integration & scanner control) Threat Processing & Risk Meters
  • 23.
    Experiment: 626 Build ahost vulnerability scanner …that doesn’t scan
  • 25.
    Why? What’s Wrongwith FooScan? Can Be Slow Can Be Intrusive Requiring Scanning during Maintenance Windows :( A Lot Can Happen In A Week Depends On Timely Vulnerability Definition Updates
  • 26.
    What About Ours? 1.Built On Automation 2. Made of Composable Ingredients 3. It Should be Non-Intrusive 4. It Should be Close to Real-Time 5. It Should Support Multiple Sources for Definitions 6. It Should Only Take ~3.14 Days to Create (have I mentioned it’s NOT a scanner)
  • 27.
    Our Base Ingredients Automation (Chef,Ansible, Docker + stuff, cron + SSH) Reliable Software + Version Info Vulnerability Definitions NVD/CVE feed with CPE included “cpe:/a:google:chrome:8.0.552.215" is vulnerable to something Eggs, Milk, Flour and Salt These are not security related
  • 28.
    Gather host: software+version AutomationFramework + Scripting we used chef + tattle XYZ Monitoring Services opsmatic, logstash, splunk… System Packager rpm, dkpkg, pacman… Dockerfiles
  • 29.
    What is tattle? Ridiculouslysimple way to store and regurgitate data tattle update software myapp --version 1.2.3 --installer ubuntu tattle report software > {“myapp”:{“version":"1.2.3","installer":"ubuntu"}} # on github soon
  • 30.
    Vulnerability Definitions NVD CVEFeed with CPE data https://nvd.nist.gov/download.cfm <entry id="CVE-2011-0001"> ...<vuln:cve-id>CVE-2011-0001</vuln:cve-id> ...<vuln:vulnerable-software-list> <vuln:product>cpe:/a:zaal:tgt:1.0.1</vuln:product> <vuln:product>cpe:/a:zaal:tgt:1.0.0</vuln:product> ...<vuln:summary>Double free vulnerability in ...
  • 31.
    Mix It TogetherParseSome XML $ cat nvdcve-2.0-2014.xml | ruby -e 'require "ox"; p Ox.parse(ARGF.read).nvd.locate("entry").select {|x| x.locate("*/vuln:product/*").member? "cpe:/a:oracle:mysql:5.5.26" }.map(&:id)’ ["CVE-2014-0384", "CVE-2014-0386", "CVE-AWW-YEAH",...] What was that? • We have mysql 5.5.26 installed • Searching for it’s most specific CPE locator • We found that there are a bunch of Vulnerabilities for it from 2014
  • 32.
    What We Did ExtendOur Risk I/O API to Allow Software+Version (we already had the data anyway…) curl -H "Content-type: application/json" https://api.riskio.us/assets/63/software -X PUT -d "{ ‘asset’: { ‘software’: $(tattle report software) } }" On Upload of Software Data • Update Assets & Create Associated Vulnerabilities • Close Vulnerabilities for Software that’s been Updated
  • 33.
  • 34.
  • 35.
    Icing on theCake Also Known As Next Steps… • Integrate Existing 0 Day Feeds • Extend upgrade/uninstall tracking to build a timeline • Alert when new Vulnerabilities match Assets (The Real-Time Thing)
  • 36.
    The Cake IsA Lie Reliability of CPEs for all data sources needs further study Vendor/Product/Version mismatches dpkg says “zlib1g”, you say “zlib” Packager whitelists / Packager specific CPEs “Ubuntu fixed this with USN-1337-1”
  • 37.
    Resources • http://puppetlabs.com/2013-state-of-devops-infographic • https://api.risk.io •https://telekomlabs.github.io/ • https://github.com/jro/automated_security • https://github.com/joeyschoblaska/brakeman-risk-io • https://github.com/jro/tattle - coming soon!
  • 38.