Building a Secure DevOps Pipeline
(for your AppSec Program)
Aaron Weaver
Matt Tesauro
AppSec USA 2017
Building a Secure
DevOps Pipeline
(for your appSec program)
Matt Tesauro
Aaron Weaver
AppSec USA 2017
Hello!
I am Aaron Weaver
All around good guy and craftsman of wood and coffee
aaron@sec.training / @weavera
Hello!
I am Matt Tesauro
I think AppSec needs to change and
I’m going to tell you how I see it changing
matt.tesauro@owasp.org / @matt_tesauro
AppSec
Evolution
✣ Radically changed travel in the US
✣ Travel time across the US
Pre-train: 6 months + $1,000
Post-train: 1 week + $150
✣ Towns that had a stopping prospered
Those that didn’t, faded away
The Iron Horse Straddles
America
Trains == Change
✣ Changed the landscape for better or
worse
The US ‘got smaller’ - travel was in reach
Expanded markets, more customers
‘Cost’ of going west went way down
Trains <==> DevOps
✣ Changed the landscape for better or
worse
DevOps changed IT for better or worse
The US ‘got smaller’ - travel was in reach
Batch / change size got smaller (CI/CD)
Expanded markets, more customers
Increased agility, more customers
’Cost’ of going west went way down
Cost of experiments goes way down
When will we see this?
DevOps AppSec
Genis: Pipeline
Species: AppSec
A new evolution of AppSec
AppSec Pipelines
Using CI/CD as inspiration,
figure out your AppSec workflow
Key Features of
AppSec Pipelines
✣ Designed for iterative improvement
✣ Provides a reusable path for AppSec activities
✣ Provides a consistent process for both the team
and our constituency
✣ One way flow with well-defined states
✣ Relies heavily on automation
✣ Grow in functionality organically over time
✣ Gracefully interconnects with the development
process
What we want is
ala carte
- Just with a limited
number of choices
Pipeline
Gen 1 Pipelines
Look at your team's purpose
and those processes
which aid it
Spending time optimizing
anything other than
the critical resource is an
illusion.
W. Edwards Deming
AppSec Personnel
✣ They are the critical resource
- optimize their work
Automate the things that don’t require a human brain
Drive up consistency
Increase tracking of work status
Increase flow through the system
Increase visibility and metrics
Reduce any dev team friction with application security
Gen 1: Inside out focused AppSec
Then,
once your house
is in order...
Gen 2 Pipelines
Look outside your team's
purpose and those
processes which aid it
DevOps Pipeline AppSec Pipeline
Integrate with Dev & Ops teams
Weaponizing Jenkins
✣ Zero false positives
Anaphylactic shock
✣ Health Checks vs Scanning
Run these all the time
✣ Home of specific issue tests
Find a vuln, write a test
✣ Cadence for longer running tests
These NEVER break the build
Every X builds or every Y days
OWASP & AppSec Pipelines
OWASP Defect Dojo
✣ Single source of truth for findings
✣ AppSec Programs, QA, Pen Testers
Custom report generation
Metrics and Dashboards
App & Infrastructure findings supported
✣ New-ish OWASP Project
Code base is 3+ years - started at Rackspace
✣ Community and contributor friendly
Bugs triaged, verified and fixed quickly
11 contributors from multiple companies
✣ Github: 262 stars, 106 forks, 199 watchers
Species: AppSec
Family: Automation
Evolving AppSec faster
What went hand in
hand with the
transcontinental
railroad expansion?
The Telegraph
Telegraph | Automation
✣ Sped up signaling and communication
Enhanced benefits of easier travel
Followed the existing tracks
✣ Linked cities using a standard protocol
Morse code
Telegraph | Automation
✣ Sped up routine tasks
Enhances benefits of existing AppSec Pipeline
Follows the same path aka consistent
✣ Links software using a standard protocol
HTTP / REST
A call to action...
AppSec
Chat Ops
Making chat the way
you do security
FYI: You’re being attacked
FYI: You’re being blocked
Advice for Devs - 24x7
Static Analysis Integration
Recurring static analysis in about 10 minutes!
Scaling with
Docker
Containers
docker run -it --name kali-pipeline
kali-pipeline /bin/bash
/usr/local/bin/run.sh
'nikto localhost -h localhost -T 58'
results.txt
Docker Security
Tool Launch
(python, Go)
ZAP
Nikto
Return ZAP IP
Run Scan, Push
Results to S3
Benefits
✣ Effectively Scales
✣ Build security tools once,
run anywhere
✣ Ease of deployment
Pull in or scale out,
your choice
Pull in Docker containers
to your build server
ZAP
Nikto
Scale out to Docker Swarm
ZAP
Nikto
Jenkins Pipeline
Pipeline as Code
Conduct your own
AppSec Pipeline Experiment
Come down out of the
traditional AppSec trees
Pick a language
✣ To do AppSec well, you need to know
something about coding
✣ Don’t care what language, pick one
and stark hacking away
✣ Most Pipeline code is glue code
- l33t algorithms need not apply
Case Studies
AppSec Pipeline - Company #1
✣ Security Findings
Turn each into a self-contained test
✣ Add those tests to Jenkins
Run hourly or at least daily
Turn green when they are fixed
✣ Tied alerts / Chat ops to those tests
Let them tell you when they are fixed
✣ Developer knows release X fixed finding Y
Bonus points for connecting Jenkins test passing to
closing Jira bug
✣ 2 FTEs assessed 35 Apps in year 1
2014
✣ 44 assessments
~5x increase
2015
✣ ~200 assessments
Changes from 2014 to 2015:
- Created the AppSec Pipeline - initial launch in March 2015
- AppSec team numbers dropped
- lost a couple of key people approx 3.5 FTEs
- Two of the AppSec team members went meta for most of 2015
Company #2
2015
✣ ~200 assessments
~2x increase
2016
✣ 414 assessments
Changes from 2015 to 2015:
- Lost 2 key FTE engineers
- AppSec team numbers dropped
- not every vacant FTE position was filled
Company #2
2014
✣ 44 assessments
9.4x
increase
2016
✣ 414 assessments
Things to remember
- Year 1 may go slow - you need to build a solid foundation
- Get your house in order, THEN reach out to other teams
- Divide tests into
- Quick, low false-positive - these go into CI/CD
- Longer, less accurate tests
Company #2
@weavera@weavera
@weavera@weavera
“I am a nice shark, not a mindless
eating machine. If I am to change this
image, I must first change myself. Fish
are friends, not food.”
-Bruce, Chum and Anchor
@weavera@weavera
“I am a nice security professional,
not a mindless vulnerability spewing
machine. If I am to change this image,
I must first change myself.
Developers are friends, not fools.”
-Bruce, Aaron and Matt
@weavera@weavera
I’m with Bruce
@BruceSecDevOps
#BruceSecDevOpsTM
Thanks!
Any questions?
Aaron Weaver
@weavera
aaron@sec.training
/in/aweaver
github.com/aaronweaver
Matt Tesauro
@matt_tesauro
matt.tesauro@owasp.org
/in/matttesauro
github.com/mtesauro
References
Presentation template:
http://www.slidescarnival.com/dolabella-free-presentation-template/840
Black and White train image (original)
https://www.youtube.com/watch?v=-80sFvilSXs
Train/Transport facts from:
https://gtgtechnologygroup.com/transcontinental-railroad/
https://www.thoughtco.com/effect-of-railroads-on-the-united-states-104724
http://www.american-rails.com/transcontinental.html
Meeting of the railroads image (original) public domain image at
https://www.thoughtco.com/effect-of-railroads-on-the-united-states-104724
Model Train cars (original)
https://www.pinterest.com/njaredmartin99/trains/
What color is your parachute (original)
http://earthconservant.com/100-ways-earthfit-day-37-color-parachute/
Telegraph poles by a railroad picture (original)
https://www.pinterest.com/pin/412360909600598272/
Telegraph key
https://openclipart.org/detail/188507/telegraph-key
Instructions for use
EDIT IN GOOGLE SLIDES
Click on the button under the presentation
preview that says "Use as Google Slides
Theme".
You will get a copy of this document on
your Google Drive and will be able to edit,
add or delete slides.
You have to be signed in to your Google
account.
EDIT IN POWERPOINT®
Click on the button under the presentation
preview that says "Download as PowerPoint
template". You will get a .pptx file that you can
edit in PowerPoint.
Remember to download and install the fonts
used in this presentation (you’ll find the links to
the font files needed in the Presentation design
slide)
More info on how to use this template at
www.slidescarnival.com/help-use-presentation-template
This template is free to use under Creative Commons Attribution license. You can keep the Credits slide or
mention SlidesCarnival and other resources used in a slide footer.
Credits
Special thanks to all the people who made and
released these awesome resources for free:
✣ Presentation template by SlidesCarnival
✣ Photographs by Unsplash
✣ Paper texture by GraphicBurguer

Building a Secure DevOps Pipeline - for your AppSec Program

  • 1.
    Building a SecureDevOps Pipeline (for your AppSec Program) Aaron Weaver Matt Tesauro AppSec USA 2017
  • 2.
    Building a Secure DevOpsPipeline (for your appSec program) Matt Tesauro Aaron Weaver AppSec USA 2017
  • 3.
    Hello! I am AaronWeaver All around good guy and craftsman of wood and coffee aaron@sec.training / @weavera
  • 4.
    Hello! I am MattTesauro I think AppSec needs to change and I’m going to tell you how I see it changing matt.tesauro@owasp.org / @matt_tesauro
  • 5.
  • 7.
    ✣ Radically changedtravel in the US ✣ Travel time across the US Pre-train: 6 months + $1,000 Post-train: 1 week + $150 ✣ Towns that had a stopping prospered Those that didn’t, faded away The Iron Horse Straddles America
  • 8.
    Trains == Change ✣Changed the landscape for better or worse The US ‘got smaller’ - travel was in reach Expanded markets, more customers ‘Cost’ of going west went way down
  • 9.
    Trains <==> DevOps ✣Changed the landscape for better or worse DevOps changed IT for better or worse The US ‘got smaller’ - travel was in reach Batch / change size got smaller (CI/CD) Expanded markets, more customers Increased agility, more customers ’Cost’ of going west went way down Cost of experiments goes way down
  • 10.
    When will wesee this? DevOps AppSec
  • 11.
    Genis: Pipeline Species: AppSec Anew evolution of AppSec
  • 12.
    AppSec Pipelines Using CI/CDas inspiration, figure out your AppSec workflow
  • 13.
    Key Features of AppSecPipelines ✣ Designed for iterative improvement ✣ Provides a reusable path for AppSec activities ✣ Provides a consistent process for both the team and our constituency ✣ One way flow with well-defined states ✣ Relies heavily on automation ✣ Grow in functionality organically over time ✣ Gracefully interconnects with the development process
  • 15.
    What we wantis ala carte - Just with a limited number of choices
  • 16.
  • 17.
    Gen 1 Pipelines Lookat your team's purpose and those processes which aid it
  • 18.
    Spending time optimizing anythingother than the critical resource is an illusion. W. Edwards Deming
  • 19.
    AppSec Personnel ✣ Theyare the critical resource - optimize their work Automate the things that don’t require a human brain Drive up consistency Increase tracking of work status Increase flow through the system Increase visibility and metrics Reduce any dev team friction with application security
  • 20.
    Gen 1: Insideout focused AppSec
  • 21.
  • 22.
    Gen 2 Pipelines Lookoutside your team's purpose and those processes which aid it
  • 23.
    DevOps Pipeline AppSecPipeline Integrate with Dev & Ops teams
  • 24.
    Weaponizing Jenkins ✣ Zerofalse positives Anaphylactic shock ✣ Health Checks vs Scanning Run these all the time ✣ Home of specific issue tests Find a vuln, write a test ✣ Cadence for longer running tests These NEVER break the build Every X builds or every Y days
  • 25.
    OWASP & AppSecPipelines
  • 26.
    OWASP Defect Dojo ✣Single source of truth for findings ✣ AppSec Programs, QA, Pen Testers Custom report generation Metrics and Dashboards App & Infrastructure findings supported ✣ New-ish OWASP Project Code base is 3+ years - started at Rackspace ✣ Community and contributor friendly Bugs triaged, verified and fixed quickly 11 contributors from multiple companies ✣ Github: 262 stars, 106 forks, 199 watchers
  • 27.
  • 28.
    What went handin hand with the transcontinental railroad expansion?
  • 29.
  • 30.
    Telegraph | Automation ✣Sped up signaling and communication Enhanced benefits of easier travel Followed the existing tracks ✣ Linked cities using a standard protocol Morse code
  • 31.
    Telegraph | Automation ✣Sped up routine tasks Enhances benefits of existing AppSec Pipeline Follows the same path aka consistent ✣ Links software using a standard protocol HTTP / REST
  • 32.
    A call toaction...
  • 33.
    AppSec Chat Ops Making chatthe way you do security
  • 34.
  • 35.
  • 36.
  • 37.
    Static Analysis Integration Recurringstatic analysis in about 10 minutes!
  • 38.
  • 39.
    docker run -it--name kali-pipeline kali-pipeline /bin/bash /usr/local/bin/run.sh 'nikto localhost -h localhost -T 58' results.txt
  • 40.
    Docker Security Tool Launch (python,Go) ZAP Nikto Return ZAP IP Run Scan, Push Results to S3
  • 41.
    Benefits ✣ Effectively Scales ✣Build security tools once, run anywhere ✣ Ease of deployment
  • 42.
    Pull in orscale out, your choice Pull in Docker containers to your build server ZAP Nikto Scale out to Docker Swarm ZAP Nikto
  • 43.
  • 44.
  • 45.
    Conduct your own AppSecPipeline Experiment Come down out of the traditional AppSec trees
  • 46.
    Pick a language ✣To do AppSec well, you need to know something about coding ✣ Don’t care what language, pick one and stark hacking away ✣ Most Pipeline code is glue code - l33t algorithms need not apply
  • 47.
  • 48.
    AppSec Pipeline -Company #1 ✣ Security Findings Turn each into a self-contained test ✣ Add those tests to Jenkins Run hourly or at least daily Turn green when they are fixed ✣ Tied alerts / Chat ops to those tests Let them tell you when they are fixed ✣ Developer knows release X fixed finding Y Bonus points for connecting Jenkins test passing to closing Jira bug ✣ 2 FTEs assessed 35 Apps in year 1
  • 49.
    2014 ✣ 44 assessments ~5xincrease 2015 ✣ ~200 assessments Changes from 2014 to 2015: - Created the AppSec Pipeline - initial launch in March 2015 - AppSec team numbers dropped - lost a couple of key people approx 3.5 FTEs - Two of the AppSec team members went meta for most of 2015 Company #2
  • 50.
    2015 ✣ ~200 assessments ~2xincrease 2016 ✣ 414 assessments Changes from 2015 to 2015: - Lost 2 key FTE engineers - AppSec team numbers dropped - not every vacant FTE position was filled Company #2
  • 51.
    2014 ✣ 44 assessments 9.4x increase 2016 ✣414 assessments Things to remember - Year 1 may go slow - you need to build a solid foundation - Get your house in order, THEN reach out to other teams - Divide tests into - Quick, low false-positive - these go into CI/CD - Longer, less accurate tests Company #2
  • 52.
  • 53.
    @weavera@weavera “I am anice shark, not a mindless eating machine. If I am to change this image, I must first change myself. Fish are friends, not food.” -Bruce, Chum and Anchor
  • 54.
    @weavera@weavera “I am anice security professional, not a mindless vulnerability spewing machine. If I am to change this image, I must first change myself. Developers are friends, not fools.” -Bruce, Aaron and Matt
  • 55.
  • 56.
    Thanks! Any questions? Aaron Weaver @weavera aaron@sec.training /in/aweaver github.com/aaronweaver MattTesauro @matt_tesauro matt.tesauro@owasp.org /in/matttesauro github.com/mtesauro
  • 57.
    References Presentation template: http://www.slidescarnival.com/dolabella-free-presentation-template/840 Black andWhite train image (original) https://www.youtube.com/watch?v=-80sFvilSXs Train/Transport facts from: https://gtgtechnologygroup.com/transcontinental-railroad/ https://www.thoughtco.com/effect-of-railroads-on-the-united-states-104724 http://www.american-rails.com/transcontinental.html Meeting of the railroads image (original) public domain image at https://www.thoughtco.com/effect-of-railroads-on-the-united-states-104724 Model Train cars (original) https://www.pinterest.com/njaredmartin99/trains/ What color is your parachute (original) http://earthconservant.com/100-ways-earthfit-day-37-color-parachute/
  • 58.
    Telegraph poles bya railroad picture (original) https://www.pinterest.com/pin/412360909600598272/ Telegraph key https://openclipart.org/detail/188507/telegraph-key
  • 59.
    Instructions for use EDITIN GOOGLE SLIDES Click on the button under the presentation preview that says "Use as Google Slides Theme". You will get a copy of this document on your Google Drive and will be able to edit, add or delete slides. You have to be signed in to your Google account. EDIT IN POWERPOINT® Click on the button under the presentation preview that says "Download as PowerPoint template". You will get a .pptx file that you can edit in PowerPoint. Remember to download and install the fonts used in this presentation (you’ll find the links to the font files needed in the Presentation design slide) More info on how to use this template at www.slidescarnival.com/help-use-presentation-template This template is free to use under Creative Commons Attribution license. You can keep the Credits slide or mention SlidesCarnival and other resources used in a slide footer.
  • 60.
    Credits Special thanks toall the people who made and released these awesome resources for free: ✣ Presentation template by SlidesCarnival ✣ Photographs by Unsplash ✣ Paper texture by GraphicBurguer