AppSec Fast and Slow: Your DevSecOps CI/CD Pipeline Isn’t an SSA Program

© 2020 Denim Group – All Rights Reserved
Building a world where technology is trusted.
Dan Cornell | CTO
AppSec Fast And Slow
Your DevSecOps CI/CD Pipeline
Isn’t an SSA Program
August 18, 2020

1
Advisory
Services
Assessment
Services
Remediation
Services
Vulnerability Resolution
Platform
Building a world where technology is trusted
How we can help:
Denim Group is solely focused on helping build
resilient software that will withstand attacks.
• Since 2001, helping secure software
• Development background
• Tools + services model

Agenda
• Cool Kids: Moving FAST
• SSA Programs
• Fast and Slow
• OWASP SAMM Walkthrough
• Conclusions
• Questions
2

Cool Kids: Moving FAST

Security in the DevOps Pipeline
Organizations like Etsy and Netflix are doing amazing
things to secure application via their DevOps pipelines

All About the Pipeline
• Security checks in the pipeline
• Application
• Infrastructure
• Cloud
• Automation is king
5

But What Doesn’t Fit Into a Pipeline?
• Dangers of DevSecOps fundamentalism
• The Pipeline Isn’t the Program
6

SSA Programs

What is Your “Why?”
• Simon Sinek TED Talk
• (If you have seen this
before, rolling your
eyes at this point is
acceptable)
• Why -> How -> What
https://www.youtube.com
/watch?v=qp0HIF3SfI4

What is an SSA Program
• SSA = Software Security Assurance
• Set of practices and activities used to reliably create,
maintain, and deploy secure software
• “We do an annual app pen test for PCI” is not an SSA
program
• Or at least probably not a very effective one
• “Here are the security checks we figured out how to stuff into
our CI/CD pipeline” is also not an SSA program
• Danger: Don’t let the pipeline become your program
• “Shifting left” isn’t bad – it just isn’t everything
9

SSA Program References
• OWASP SAMM
• BSIMM
10

OWASP SAMM
• Originally OpenSAMM from Pravir Chandra
• OWASP’s evolution/fork
• Five Business Functions
• Three Security Practices for each
• Two Streams for each
https://owaspsamm.org/
11

OWASP SAMM
12

BSIMM
• Originally from Cigital (now Synopsys)
• Based on data collection from participating
organizations
• Four domains
• Three Practices for each
• Total of 119 Activities
https://www.bsimm.com/
13

BSIMM
14

OWASP SAMM Walkthrough
• We will use OWASP SAMM for the
purposes of this webinar
• More prescriptive
• Less vendor-centric
• If you are using BSIMM it is pretty trivial to
translate
15

If You Are Just Starting Out
• Assessing your program using either tool is less-than-ideal
• Better:
• Define your scope/mandate
• Do some testing
• Run some vulnerabilities through resolution
• Proceed from there
https://www.denimgroup.com/contact-us/
16

Fast and Slow

Thinking Fast and Slow
18
• Written by Daniel Kahneman
• System 1 (Fast): Instinctive,
emotional
• System 2 (Slow): Deliberative,
logical
• (For AppSec purposes, use
configuration/customization to minimize the
“emotional”)
https://www.amazon.com/Thinking-Fast-Slow-Daniel-
Kahneman/dp/0141033576/ref=asc_df_0141033576/

An Aside: What Horrible Names!
• System 1 and
System 2 ???
• Almost as bad as
Type I and Type II
Errors
19
https://www.simplypsychology.org/type_I_and_type_II_errors.html

Another Aside: The Undoing Project
• Michael Lewis book on the
research of and the
collaboration between Daniel
Kahneman and Amos
Tversky
https://www.amazon.com/Undoing-Project-Friendship-
Changed-Minds/dp/0393354776/ref=sr_1_2
20

Fast and Slow
In a culture like
DevSecOps that is so
focused on FAST, what
is still critical, but has to
go SLOW?
21

What Do We Mean By FAST?
Blog post: Power,
Responsibility, and
Security’s Role in
the DevOps
Pipeline
https://www.denimgroup.com/resources/blog/2019/02/powe
r-responsibility-and-securitys-role-in-the-devops-pipeline/
22

To Be DevSecOps FAST
1. Available quickly
2. High-value
3. Low (NO) false positives (no Type I errors)
• Limited time budget
• Developers have to care
• Don’t waste developers’ time
23

OWASP SAMM Walkthrough

Governance
• Strategy and Metrics
• Policy and Compliance
• Education and Guidance
25

Strategy and Metrics
• You can’t automate strategy
• SLOW
• You can use CI/CD to feed your metrics
• Kinda FAST
• Metrics in general: very automatable
26

Blog Posts on ThreadFix and Metrics
• Value of secure coding training for your organization
• https://threadfix.it/resources/applied-threadfix-fire-bullets-then-cannonballs-
appsec-edition/
• xAST scanner rollouts
• https://threadfix.it/resources/applied-threadfix-fire-bullets-then-cannonballs-part-2/
• Optimizing training investments
• https://threadfix.it/resources/applied-threadfix-getting-the-most-out-of-your-training-investment/
• Automating vulnerability exception reporting
• https://threadfix.it/resources/applied-threadfix-automated-vulnerability-exception-reporting/
27

Policy and Compliance
• You can’t automate the creation of your policies
• SLOW
• You can use CI/CD to automate some policy
checks
• CI/CD pass/fail
• Be careful of limitations – this is a helper, not
definitive
• Kinda FAST
28

CI/CD Policy Configuration
• Testing
• Synchronous
• Asynchronous
• Decision
• Reporting
29
Blog Post: Effective Application
Security Testing in DevOps Pipelines
http://www.denimgroup.com/blog/2016/12/effective-application-security-testing-in-devops-pipelines/
https://www.denimgroup.com/resources/effective-application-security-for-devops/

Automated Compliance for DoD
• Continuous Authority
to Operate (ATO) with
ThreadFix – Bringing
Commercial Insights
to the DoD
https://threadfix.it/resources/continuous-
authority-to-operate-with-threadfix/

Education and Guidance
• Instructor-led training: SLOW
• eLearning
• Monolithic: SLOW
• Targeted: Not FAST, but increasingly interesting
• Security Champions
• Common responsibility is to configure security
testing in CI/CD environments and tune scanning
• They make things FASTer
31

Security Champions
Webinar: Security
Champions: Pushing
Security Expertise to
the Edges of Your
Organization
https://www.denimgroup.com/resources/webinar/security-champions-
pushing-security-expertise-to-the-edges-of-your-organization/
32

Design
• Threat Assessment
• Security Requirements
• Security Architecture
33

Threat Assessment
• Determining your general application threat profiles
can’t be automated
• SLOW
• Threat Modeling also requires a lot of manual work
• Some new interesting automation, but nothing in CI/CD
pipelines
• Some vendors providing tooling support
• Can allow for manual incremental changes – not CI/CD, but fits
better into Agile environments
• SLOW
34

Security Requirements
• Determining your requirements is largely
manual
• Some tooling support is available
• SLOW
• Validating if they are met is largely manual,
but we will look at this later during the
Verification/Requirements-Driven Testing
activity
35

Secure Architecture
• Determining your architectural security
requirements is largely manual
• SLOW
• Validating if they are met is largely
manual, but we will look at this later during
the Verification/Architecture Assessment
activity
36

Implementation
• Secure Build
• Secure Deployment
• Defect Management
37

Secure Build
• This is really the crux of what we are discussing today
• FAST
• How can you integrate security into the build process?
• SAST/DAST/IAST
• SCA
• OWASP Dependency Check https://owasp.org/www-project-dependency-check/
• If you are even considering this you have to have a repeatable build
process
• Otherwise please log off this webinar and pick up a Jenkins for Dummies book. You
can pick this back up later.
• Software Bill of Materials (SBOM)
• OWASP Dependency Track https://dependencytrack.org/
38

Architectural Bill of Materials
Webinar: The As, Bs, and
Four Cs of Testing Cloud-
Native Applications
https://www.denimgroup.com/resources/webinar/the-as-bs-
and-four-cs-of-testing-cloud-native-applications/
39

Secure Deployment
• An extension of Secure Build
• Organizations tend to be a little less mature
• FAST
• Technologies like Puppet, Chef, Terraform
40

Defect Management
• Subsets of this can be FAST
• But you have to tune scanners or you will run into problems
• High-value, no false positives
• ThreadFix allows for automated defect creation
• In practice, it takes a while to get to this level
• Limited coverage: only works for vulnerabilities you can find
with automation in CI/CD pipelines
• We will talk more about these testing limitations in the
Verification discussions
41

Bundling Strategies
• Turning vulnerabilities into defects
• 1:1 approach?
• More time spent administering defects than fixing
issues
• Bundling
• By vulnerability type
• By severity (more mature applications)
• Other approaches
42

Metrics and Feedback Stream
• Scanner / developer provide separation of
duties
• Scanners find vulns, developers say
they fixed them, scanners confirm they
did
• Obviously only applies to
vulnerabilities identified by automation
• ThreadFix tracking mean-time-to-
remediation (MTTR)
• Good metric for Agile/DevOps teams –
how fast can you fix?
• (Better than defects per KLoC)
• Benchmark against data from
Veracode/WhiteHat
43

Verification
• Architecture Assessment
• Requirements-driven Testing
• Security Testing
44

Architecture Assessment
• This largely has to be done manually
• SLOW
• Some architectural policies may be
checked automatically
• Cloud configuration
45

ThreadFix and Irius Risk
• Pull data from ThreadFix
into Irius Risk threat
models
https://support.iriusrisk.com/hc/en-us/articles/360021522391-
Importing-Test-Results-from-External-Tools
46

ScoutSuite
• Check configuration of
cloud environments
• Checks for:
• Open S3 buckets
• IAM configuration
https://github.com/nccgroup/ScoutSuite
47

Requirements-Driven Testing
• Control verification: largely a manual process
• SLOW
• Misuse/abuse testing:
• Fuzzing can be automated, but runtimes can extend
beyond the time budget for FAST
• Abuse case and business logic testing is manual
• DoS testing does not fit in most general pipelines
• Mostly SLOW
• Some automation and integration possible
48

ThreadFix and SD Elements
Webinar: ThreadFix and SD Elements: Unifying
Security Requirements and Vulnerability
Management for Applications
https://threadfix.it/resources/threadfix-and-sd-elements-unifying-security-
requirements-and-vulnerability-management/
49

Security Testing
• THIS is really what the discussion comes
down to
• How sufficient is the security testing you can
stuff into a CI/CD pipeline?
• OWASP SAMM has two streams:
• Scalable baseline
• Deep understanding
50

OWASP and Testing
• OWASP has traditionally had a cultural focus on
the strengths (and weaknesses) of automated
testing tools
• Consultants vs scanner vendors
• Testing Guide
• https://owasp.org/www-project-web-security-testing-guide/
• ASVS
• https://owasp.org/www-project-application-security-verification-standard/
51

Scalable Baseline Stream
• Three levels of maturity
1. Use an automated tool
2. Employ application-specific automation (tuning)
3. Integrate into the build process
• This webinar presupposes the top level of
maturity
• You did remember to tune your scanner before
you put it in the build process, right?
52

Deep Understanding Stream
• This is all manual
• Manual test high-risk components
• Perform penetration testing
• Integrate testing into the development process
• Tooling can help
• Focus efforts on diffs / new or altered functionality
53

Testing in CI/CD Pipelines
54

SAST in CI/CD
• Mostly open source linting tools
• Need for speed
• Commercial-grade tools are less prevalent
• Run SAST on diffs?
• Cross-method/class data and control flow takes time
• Cut down the rules
• Shorten run times
• Limit false positives
55

DAST in CI/CD
• Concerns about run times
• Approaches for targeted DAST
• Focus on changes in the app
56

Targeting DAST Testing
Webinar: Monitoring
Application Attack
Surface and Integrating
Security into DevOps
Pipelines
https://threadfix.it/resources/monitorin
g-application-attack-surface-and-
integrating-security-into-devops-
pipelines/
57

IAST in CI/CD
• Great!
• Typically relies on generated traffic
• Use DAST testing to generate traffic
• Use integration tests to generate traffic
58

SCA in CI/CD
• Great!
• Look at run time tradeoffs vs. velocity of
new components and new vulnerabilities
59

Operations
• Incident Management
• Environmental Management
• Operational Management
60

Incident Management
• Not in a pipeline
• Use automation for detection where
possible
• Some automation frameworks available for
response
61

Application Logging for Security
Video: Top Strategies to
Capture Security
Intelligence for
Applications
https://www.denimgroup.com/resources/article/top-strategies-to-capture-
security-intelligence-for-applications-includes-educational-video/
62

Environment Management
• Servers should be cattle, not pets
• Configuration Handling stream:
• Hopefully you have this sorted given the work you have done for
Secure Deployment
• Chef, Puppet, Terraform
• ScoutSuite
• Patching and Updating stream:
• Detection: FAST
• Actual patching: SLOW
63

Operational Management
• Data Protection stream: SLOW
• Oh, wait, your DLP solution will sort this out
for you
• Decommissioning: SLOW
64

Conclusions

What Goes in a Pipeline?
• Linting SAST
• DAST if you can target it
• IAST if you can generate meaningful traffic
• SCA if you want
66

What Likely Has to be Done Outside?
• Full, commercial-grade SAST
• Full DAST
• Manual code review
• Penetration testing
• Threat modeling
67

What Has to be Done Outside?
• Most everything else
• Strategy
• Policy
• Training
• Architecture
• Security requirements
68

Shifting Left is Awesome…
• But it is only one aspect of a far more complicated
landscape
• For testing: think coverage
• Classes of vulnerabilities
• Detection approaches
• Quality of approaches
• For everything else:
• Thing programmatically
69

Questions

@denimgroup
www.denimgroup.com

AppSec Fast and Slow: Your DevSecOps CI/CD Pipeline Isn’t an SSA Program

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AppSec Fast and Slow: Your DevSecOps CI/CD Pipeline Isn’t an SSA Program

Similar to AppSec Fast and Slow: Your DevSecOps CI/CD Pipeline Isn’t an SSA Program (20)

More from Denim Group

More from Denim Group (17)

Recently uploaded

Recently uploaded (20)

AppSec Fast and Slow: Your DevSecOps CI/CD Pipeline Isn’t an SSA Program