LONDON 18-19 OCT 2018
Is your supply chain your Achille's heel ?
COLIN DOMONEY
LONDON 18-19 OCT 2018
About the Presenter
@colindomoney
• Built lots of hardware and software
• Done AppSec at scale in large enterprise
• Worked in vendor-land in AppSec
• Currently a transformation consultant
• Veteran DevSecCon presenter
• Interested in all things new and shiny
LONDON 18-19 OCT 2018
Thank You’s and Acknowledgements
@controlplaneio
@lukeb0nd @sublimino
Santiago Torres @ In-Toto
@torresariass
veracity
/vəˈrasɪti/
noun
conformity to facts; accuracy.
provenance
/ˈprɒv(ə)nəns/
noun
the place of origin or earliest known history of something.
LONDON 18-19 OCT 2018
How Do Other Industries
Manage Their Supply Chains
LONDON 18-19 OCT 2018
Big Pharmaceuticals Understand Supply Chains Surely ?
LONDON 18-19 OCT 2018
Then I Remembered This …
LONDON 18-19 OCT 2018
Consumer Electronics Understand Supply Chains Surely ?
LONDON 18-19 OCT 2018
And Then This Happened !
“Having a well-done, nation-state-
level hardware implant surface
would be like witnessing a unicorn
jumping over a rainbow”
Joe Grand
LONDON 18-19 OCT 2018
Software Supply Chain
Failure Modes
LONDON 18-19 OCT 2018
Vulnerable 3rd Party Components
LONDON 18-19 OCT 2018
Typosquatting
LONDON 18-19 OCT 2018
The CCleaner Malware Attack
• Malware distributed via official download site
• Affected 2.7 million users
• Initial entry point via a compromised developer
account
• Three stage deployment compromising
intermediate build machines
LONDON 18-19 OCT 2018
Trust, but Verify
• Ensure developers aren’t ‘optimising’ your
security testing out of the pipeline
• Validate what is scanned is what is deployed
• Validate that what you test is representative of
the actual application
• Hunt for shadow build infrastructure
• Get early warnings for new development
LONDON 18-19 OCT 2018
Build Pipelines : Then and
Now, and Beyond
LONDON 18-19 OCT 2018
Before DevOps …
LONDON 18-19 OCT 2018
DevSecOps … SecDevOps …
CAB
LONDON 18-19 OCT 2018
Making It Go Faster – Just Remove all Security Measures
LONDON 18-19 OCT 2018
Software Supply Chain Basics
LONDON 18-19 OCT 2018
Prescribe a Policy for OSS Use
• Prescribe a policy for the use of OSS based on:
• Risk appetite
• Business criticality
• Time to market
• Organisational maturity
• Provide a recommended architecture of
commonly used and pre-approved components
• Educate your security team in the use of OSS
components and risk determination
LONDON 18-19 OCT 2018
Control Your Repositories
• Use a caching binary repository server (such as
Nexus)
• Maintain a blacklist of known bad (and hence
banned) components
• Maintain a whitelist of known good (and hence
approved) components
• Quarantine unknown components until assessed
• In extremis disable access to public internet
repositories
LONDON 18-19 OCT 2018
Hardening your Build
Pipeline
LONDON 18-19 OCT 2018
Using Your Pipeline as a Bitcoin Miner
• Exploits CVE-2107-1000353 in Jenkins disclosed in
April 2017
• Deploys XMRig miner and a RAT
• Over $3 million mined thus far
https://www.csoonline.com/article/3256314/security/hackers-exploit-jenkins-
servers-make-3-million-by-mining-monero.html
LONDON 18-19 OCT 2018
Harden Your CI/CD Infrastructure
• Harden the hosts, ensure patching is rigorously applied
• Lock down your tools (Jenkins is wide open by default)
• Lock down and harden your config management tools
• Ensure that keys, credentials and secrets are protected
• Secure access to all repositories
• Review and audit your access controls to your pipeline
• Treat your pipeline as you would your production infrastructure
https://www.oreilly.com/ideas/9-tips-for-a-more-secure-continuous-delivery-pipeline
LONDON 18-19 OCT 2018
In-Depth with In-Toto
LONDON 18-19 OCT 2018
The Update Framework
TUF’s primary goals are:
• Framework that can be used to secure systems
• Minimise the impact of key compromises
• Be flexible and easy to integrate
Guards against the following attacks:
• Replay attacks of same file
• Compromised and vulnerable versions
• Key compromise in signing files
Implemented as Notary by Docker (originally)
http://www.eweek.com/security/cncf-brings-in-notary-the-update-framework-to-
boost-container-security
LONDON 18-19 OCT 2018
What Is In-Toto
Motivation:
“Although many frameworks ensuring security in the "last mile" (e.g., software
updaters) exist, they may be providing integrity and authentication to a product that is
already vulnerable; it is possible that, by the time the package makes it to a software
update repository, it has already been compromised.”
Goals:
“in-toto aims to provide integrity, authentication and auditability to the supply chain as
a whole. This means that all the steps within the supply chain are clearly laid out, that
the parties involved in carrying out a step are explicitly stated, and that each step
carried out meets the requirements specified by the actor responsible for this software
product.”
LONDON 18-19 OCT 2018
In-Toto Basic Terminology
Materials: the elements used (e.g., files) to perform a step in the supply chain.
Product: the result of carrying out a step. Products are recorded as part of link
metadata.
Link: metadata information gathered while performing a supply chain step or
inspection, signed by the functionary that performed the step or the client that
performed the inspection
Verification: the process by which data and metadata included in the final product is
used to ensure its correctness.
LONDON 18-19 OCT 2018
In-Toto Actors
Project Owner: Defines the layout of the software supply chain.
Functionary: Performs a step in the supply chain and provides a piece of link metadata
as a record that such a step was carried out.
Client: Performs verification on the final product by checking the provided layout and
link metadata.
LONDON 18-19 OCT 2018
In-Toto Layouts - Steps
• A recipe for taking materials and producing an
output product.
• Steps can be chained, and sub-layouts can be
specified.
LONDON 18-19 OCT 2018
In-Toto Layouts - Inspect
• Executes at the final stage of verification to verify
the resultant product matches that specified in the
layout.
• Takes an input list of expected materials and
expected products.
• Returns a go/no-go result.
LONDON 18-19 OCT 2018
In-Toto Links
• Record information about the execution
environment.
• Cryptographically signed by the functionary
carrying out the action.
LONDON 18-19 OCT 2018
And Finally : In-Toto In Action
A Passing Verification:
A Failing Verification: https://in-toto.github.io/
LONDON 18-19 OCT 2018
In-Toto in a Jenkins Server
stage('Build') {
agent {
docker {
#image name here
}
}
steps {
withCredentials([#any credentials here]) {
in_toto_wrap(['stepName': 'Build',
'keyPath': "${WORKER_KEY}",
'transport': "redis://${REDIS_ENDPOINT}:6379"]){
#your actual step here
}
}
}
}
}
LONDON 18-19 OCT 2018
Getting It Right By Design :
Cloud Native and Containers
LONDON 18-19 OCT 2018
Point Solutions Are Not Enough
LONDON 18-19 OCT 2018
What Can You Trust ?
• Git ensures integrity but not identity
• Anyone can pretend to commit as
someone else !
• Most people assume Git is a trusted
source
• Signing and verification are easy
• Enterprise key management not so
much !
https://mikegerwitz.com/papers/git-horror-story
https://medium.com/@pjbgf/spoofing-git-commits-7bef357d72f0
LONDON 18-19 OCT 2018
Security-hardened Container Supply Chain
Base Image Code Build Application Image Deploy
Controlled base
images
Hash based
addressing
Static analysis
Dependency analysis
Hermetic
Reproducible
Rootless
Vulnerability
scanning
Configuration
scanning
Admission control
Runtime
configurations
Docker Hub TUF
Notary
Grafeas
In-Toto
Clair
Aqua Microscanner
Kubernetes
Kritis
LONDON 18-19 OCT 2018
Securing Builds with Metadata
• Pipeline metadata is rich and varied
• Initiating users and/or events
• Installed dependencies and their versions
• Veracity test data (unit/integration/acceptance tests)
• Security test data
• Data can be used for:
• Recording i.e. audit
• Report/enforcing i.e. policy
LONDON 18-19 OCT 2018
Storing Metadata with Google Grafeas
• Google’s open-source project to audit and govern the
software supply chain
• Stores metadata about artefacts and their
vulnerabilities
• Twistlock, Aqua, JFrog Xray, BlackDuck can send
metadata to Grafeas
• Possible to query that metadata to gate builds and
deployments
LONDON 18-19 OCT 2018
Grafeas in Action
LONDON 18-19 OCT 2018
The Art of the Possible
LONDON 18-19 OCT 2018
In-Toto in a Container SDLC
LONDON 18-19 OCT 2018
DevSecOps … with In-Toto
CAB
LONDON 18-19 OCT 2018
Avoid the Horror
• Practice basic hygiene
• Trust with caution
• Trust but verify
• Understand your abuse cases
• Embrace new ways of working
• Backport the best of new technology
LONDON 18-19 OCT 2018
[Last slide for thank you
message, links, etc]
@colindomoney

DevSecCon London 2018: Is your supply chain your achille's heel

  • 1.
    LONDON 18-19 OCT2018 Is your supply chain your Achille's heel ? COLIN DOMONEY
  • 2.
    LONDON 18-19 OCT2018 About the Presenter @colindomoney • Built lots of hardware and software • Done AppSec at scale in large enterprise • Worked in vendor-land in AppSec • Currently a transformation consultant • Veteran DevSecCon presenter • Interested in all things new and shiny
  • 3.
    LONDON 18-19 OCT2018 Thank You’s and Acknowledgements @controlplaneio @lukeb0nd @sublimino Santiago Torres @ In-Toto @torresariass
  • 5.
  • 6.
    provenance /ˈprɒv(ə)nəns/ noun the place oforigin or earliest known history of something.
  • 7.
    LONDON 18-19 OCT2018 How Do Other Industries Manage Their Supply Chains
  • 8.
    LONDON 18-19 OCT2018 Big Pharmaceuticals Understand Supply Chains Surely ?
  • 9.
    LONDON 18-19 OCT2018 Then I Remembered This …
  • 10.
    LONDON 18-19 OCT2018 Consumer Electronics Understand Supply Chains Surely ?
  • 11.
    LONDON 18-19 OCT2018 And Then This Happened ! “Having a well-done, nation-state- level hardware implant surface would be like witnessing a unicorn jumping over a rainbow” Joe Grand
  • 13.
    LONDON 18-19 OCT2018 Software Supply Chain Failure Modes
  • 14.
    LONDON 18-19 OCT2018 Vulnerable 3rd Party Components
  • 15.
    LONDON 18-19 OCT2018 Typosquatting
  • 16.
    LONDON 18-19 OCT2018 The CCleaner Malware Attack • Malware distributed via official download site • Affected 2.7 million users • Initial entry point via a compromised developer account • Three stage deployment compromising intermediate build machines
  • 17.
    LONDON 18-19 OCT2018 Trust, but Verify • Ensure developers aren’t ‘optimising’ your security testing out of the pipeline • Validate what is scanned is what is deployed • Validate that what you test is representative of the actual application • Hunt for shadow build infrastructure • Get early warnings for new development
  • 18.
    LONDON 18-19 OCT2018 Build Pipelines : Then and Now, and Beyond
  • 19.
    LONDON 18-19 OCT2018 Before DevOps …
  • 20.
    LONDON 18-19 OCT2018 DevSecOps … SecDevOps … CAB
  • 21.
    LONDON 18-19 OCT2018 Making It Go Faster – Just Remove all Security Measures
  • 23.
    LONDON 18-19 OCT2018 Software Supply Chain Basics
  • 24.
    LONDON 18-19 OCT2018 Prescribe a Policy for OSS Use • Prescribe a policy for the use of OSS based on: • Risk appetite • Business criticality • Time to market • Organisational maturity • Provide a recommended architecture of commonly used and pre-approved components • Educate your security team in the use of OSS components and risk determination
  • 25.
    LONDON 18-19 OCT2018 Control Your Repositories • Use a caching binary repository server (such as Nexus) • Maintain a blacklist of known bad (and hence banned) components • Maintain a whitelist of known good (and hence approved) components • Quarantine unknown components until assessed • In extremis disable access to public internet repositories
  • 26.
    LONDON 18-19 OCT2018 Hardening your Build Pipeline
  • 27.
    LONDON 18-19 OCT2018 Using Your Pipeline as a Bitcoin Miner • Exploits CVE-2107-1000353 in Jenkins disclosed in April 2017 • Deploys XMRig miner and a RAT • Over $3 million mined thus far https://www.csoonline.com/article/3256314/security/hackers-exploit-jenkins- servers-make-3-million-by-mining-monero.html
  • 28.
    LONDON 18-19 OCT2018 Harden Your CI/CD Infrastructure • Harden the hosts, ensure patching is rigorously applied • Lock down your tools (Jenkins is wide open by default) • Lock down and harden your config management tools • Ensure that keys, credentials and secrets are protected • Secure access to all repositories • Review and audit your access controls to your pipeline • Treat your pipeline as you would your production infrastructure https://www.oreilly.com/ideas/9-tips-for-a-more-secure-continuous-delivery-pipeline
  • 29.
    LONDON 18-19 OCT2018 In-Depth with In-Toto
  • 30.
    LONDON 18-19 OCT2018 The Update Framework TUF’s primary goals are: • Framework that can be used to secure systems • Minimise the impact of key compromises • Be flexible and easy to integrate Guards against the following attacks: • Replay attacks of same file • Compromised and vulnerable versions • Key compromise in signing files Implemented as Notary by Docker (originally) http://www.eweek.com/security/cncf-brings-in-notary-the-update-framework-to- boost-container-security
  • 31.
    LONDON 18-19 OCT2018 What Is In-Toto Motivation: “Although many frameworks ensuring security in the "last mile" (e.g., software updaters) exist, they may be providing integrity and authentication to a product that is already vulnerable; it is possible that, by the time the package makes it to a software update repository, it has already been compromised.” Goals: “in-toto aims to provide integrity, authentication and auditability to the supply chain as a whole. This means that all the steps within the supply chain are clearly laid out, that the parties involved in carrying out a step are explicitly stated, and that each step carried out meets the requirements specified by the actor responsible for this software product.”
  • 32.
    LONDON 18-19 OCT2018 In-Toto Basic Terminology Materials: the elements used (e.g., files) to perform a step in the supply chain. Product: the result of carrying out a step. Products are recorded as part of link metadata. Link: metadata information gathered while performing a supply chain step or inspection, signed by the functionary that performed the step or the client that performed the inspection Verification: the process by which data and metadata included in the final product is used to ensure its correctness.
  • 33.
    LONDON 18-19 OCT2018 In-Toto Actors Project Owner: Defines the layout of the software supply chain. Functionary: Performs a step in the supply chain and provides a piece of link metadata as a record that such a step was carried out. Client: Performs verification on the final product by checking the provided layout and link metadata.
  • 34.
    LONDON 18-19 OCT2018 In-Toto Layouts - Steps • A recipe for taking materials and producing an output product. • Steps can be chained, and sub-layouts can be specified.
  • 35.
    LONDON 18-19 OCT2018 In-Toto Layouts - Inspect • Executes at the final stage of verification to verify the resultant product matches that specified in the layout. • Takes an input list of expected materials and expected products. • Returns a go/no-go result.
  • 36.
    LONDON 18-19 OCT2018 In-Toto Links • Record information about the execution environment. • Cryptographically signed by the functionary carrying out the action.
  • 37.
    LONDON 18-19 OCT2018 And Finally : In-Toto In Action A Passing Verification: A Failing Verification: https://in-toto.github.io/
  • 38.
    LONDON 18-19 OCT2018 In-Toto in a Jenkins Server stage('Build') { agent { docker { #image name here } } steps { withCredentials([#any credentials here]) { in_toto_wrap(['stepName': 'Build', 'keyPath': "${WORKER_KEY}", 'transport': "redis://${REDIS_ENDPOINT}:6379"]){ #your actual step here } } } } }
  • 39.
    LONDON 18-19 OCT2018 Getting It Right By Design : Cloud Native and Containers
  • 40.
    LONDON 18-19 OCT2018 Point Solutions Are Not Enough
  • 41.
    LONDON 18-19 OCT2018 What Can You Trust ? • Git ensures integrity but not identity • Anyone can pretend to commit as someone else ! • Most people assume Git is a trusted source • Signing and verification are easy • Enterprise key management not so much ! https://mikegerwitz.com/papers/git-horror-story https://medium.com/@pjbgf/spoofing-git-commits-7bef357d72f0
  • 42.
    LONDON 18-19 OCT2018 Security-hardened Container Supply Chain Base Image Code Build Application Image Deploy Controlled base images Hash based addressing Static analysis Dependency analysis Hermetic Reproducible Rootless Vulnerability scanning Configuration scanning Admission control Runtime configurations Docker Hub TUF Notary Grafeas In-Toto Clair Aqua Microscanner Kubernetes Kritis
  • 43.
    LONDON 18-19 OCT2018 Securing Builds with Metadata • Pipeline metadata is rich and varied • Initiating users and/or events • Installed dependencies and their versions • Veracity test data (unit/integration/acceptance tests) • Security test data • Data can be used for: • Recording i.e. audit • Report/enforcing i.e. policy
  • 44.
    LONDON 18-19 OCT2018 Storing Metadata with Google Grafeas • Google’s open-source project to audit and govern the software supply chain • Stores metadata about artefacts and their vulnerabilities • Twistlock, Aqua, JFrog Xray, BlackDuck can send metadata to Grafeas • Possible to query that metadata to gate builds and deployments
  • 45.
    LONDON 18-19 OCT2018 Grafeas in Action
  • 46.
    LONDON 18-19 OCT2018 The Art of the Possible
  • 47.
    LONDON 18-19 OCT2018 In-Toto in a Container SDLC
  • 48.
    LONDON 18-19 OCT2018 DevSecOps … with In-Toto CAB
  • 49.
    LONDON 18-19 OCT2018 Avoid the Horror • Practice basic hygiene • Trust with caution • Trust but verify • Understand your abuse cases • Embrace new ways of working • Backport the best of new technology
  • 50.
    LONDON 18-19 OCT2018 [Last slide for thank you message, links, etc] @colindomoney