COLIN DOMONEY The advent of DevOps and large scale automation of software construction and delivery has elevated the software supply chain – and its underpinning delivery pipeline – to mission critical status in any modern enterprise. The increased velocity of modern pipelines and the removal of manual checks and balances has meant that modern pipelines are potential single points of failure in the delivery of secure software. Automotive and consumer electronics industries have long understood the need for both provenance (understanding the origin of materials) and veracity (ensuring the integrity of their manufacturing processes) in their supply chains; this presentation will address threats to software supply chains and practical approaches to reducing the fragility of your supply chain. Several examples of software supply chain failures will be presented and deconstructed to understand the typical failure modes. At the most elementary level many pipelines are poorly constructed with low levels of repeatability and poor test coverage, in other organisations there is a lack of governance over the supply chain allowing careless or willingly negligent actors to subvert or bypass controls or testing within the pipeline. There is also no standard mechanism to ensure a ‘chain of custody’ within a pipeline due to a lack common interchange format between tools, or a standard manner to represent the steps within a pipeline build process. This presentation will cover approaches (using ‘people and process’) in enforcing governance within a supply chain by describing best practices used in large-scale AppSec programmes. Several emerging technology initiatives will be presented: Google’s Grafeas is a means to ensure vulnerability information is represented in a uniform manner across all steps of a pipeline process, while In-Toto is a project to formally enforce the integrity of a pipeline process. A reference secure pipeline will be presented demonstrating both tools working in symphony, along with standard open source and commercial AppSec tools. Finally the pipeline itself may become the Achille’s Heel in an organisation – many pipelines are not sufficiently hardened and are themselves open to attack by use of vulnerable components and their extensible nature, often along with very wide open permissions. Guidance will be given on hardening of typical pipelines, and a fully secured ephemeral Jenkins pipeline will be demonstrated. Benefits of this Session: The attendee will gain an increased awareness of the pivotal importance of the software supply chain, and gain an understanding of some common failure modes and weaknesses. Most importantly the attendee will come away with practical guidance on enforcing higher levels of governance on their supply chain without reducing delivery velocity, as well as how to harden the pipeline infrastructure itself.

1. LONDON 18-19 OCT 2018
Is your supply chain your Achille's heel ?
COLIN DOMONEY

2. LONDON 18-19 OCT 2018
About the Presenter
@colindomoney
• Built lots of hardware and software
• Done AppSec at scale in large enterprise
• Worked in vendor-land in AppSec
• Currently a transformation consultant
• Veteran DevSecCon presenter
• Interested in all things new and shiny

3. LONDON 18-19 OCT 2018
Thank You’s and Acknowledgements
@controlplaneio
@lukeb0nd @sublimino
Santiago Torres @ In-Toto
@torresariass

5. veracity
/vəˈrasɪti/
noun
conformity to facts; accuracy.

6. provenance
/ˈprɒv(ə)nəns/
noun
the place of origin or earliest known history of something.

7. LONDON 18-19 OCT 2018
How Do Other Industries
Manage Their Supply Chains

8. LONDON 18-19 OCT 2018
Big Pharmaceuticals Understand Supply Chains Surely ?

9. LONDON 18-19 OCT 2018
Then I Remembered This …

10. LONDON 18-19 OCT 2018
Consumer Electronics Understand Supply Chains Surely ?

11. LONDON 18-19 OCT 2018
And Then This Happened !
“Having a well-done, nation-state-
level hardware implant surface
would be like witnessing a unicorn
jumping over a rainbow”
Joe Grand

13. LONDON 18-19 OCT 2018
Software Supply Chain
Failure Modes

14. LONDON 18-19 OCT 2018
Vulnerable 3rd Party Components

15. LONDON 18-19 OCT 2018
Typosquatting

16. LONDON 18-19 OCT 2018
The CCleaner Malware Attack
• Malware distributed via official download site
• Affected 2.7 million users
• Initial entry point via a compromised developer
account
• Three stage deployment compromising
intermediate build machines

17. LONDON 18-19 OCT 2018
Trust, but Verify
• Ensure developers aren’t ‘optimising’ your
security testing out of the pipeline
• Validate what is scanned is what is deployed
• Validate that what you test is representative of
the actual application
• Hunt for shadow build infrastructure
• Get early warnings for new development

18. LONDON 18-19 OCT 2018
Build Pipelines : Then and
Now, and Beyond

19. LONDON 18-19 OCT 2018
Before DevOps …

20. LONDON 18-19 OCT 2018
DevSecOps … SecDevOps …
CAB

21. LONDON 18-19 OCT 2018
Making It Go Faster – Just Remove all Security Measures

23. LONDON 18-19 OCT 2018
Software Supply Chain Basics

24. LONDON 18-19 OCT 2018
Prescribe a Policy for OSS Use
• Prescribe a policy for the use of OSS based on:
• Risk appetite
• Business criticality
• Time to market
• Organisational maturity
• Provide a recommended architecture of
commonly used and pre-approved components
• Educate your security team in the use of OSS
components and risk determination

25. LONDON 18-19 OCT 2018
Control Your Repositories
• Use a caching binary repository server (such as
Nexus)
• Maintain a blacklist of known bad (and hence
banned) components
• Maintain a whitelist of known good (and hence
approved) components
• Quarantine unknown components until assessed
• In extremis disable access to public internet
repositories

26. LONDON 18-19 OCT 2018
Hardening your Build
Pipeline

27. LONDON 18-19 OCT 2018
Using Your Pipeline as a Bitcoin Miner
• Exploits CVE-2107-1000353 in Jenkins disclosed in
April 2017
• Deploys XMRig miner and a RAT
• Over $3 million mined thus far
https://www.csoonline.com/article/3256314/security/hackers-exploit-jenkins-
servers-make-3-million-by-mining-monero.html

28. LONDON 18-19 OCT 2018
Harden Your CI/CD Infrastructure
• Harden the hosts, ensure patching is rigorously applied
• Lock down your tools (Jenkins is wide open by default)
• Lock down and harden your config management tools
• Ensure that keys, credentials and secrets are protected
• Secure access to all repositories
• Review and audit your access controls to your pipeline
• Treat your pipeline as you would your production infrastructure
https://www.oreilly.com/ideas/9-tips-for-a-more-secure-continuous-delivery-pipeline

29. LONDON 18-19 OCT 2018
In-Depth with In-Toto

30. LONDON 18-19 OCT 2018
The Update Framework
TUF’s primary goals are:
• Framework that can be used to secure systems
• Minimise the impact of key compromises
• Be flexible and easy to integrate
Guards against the following attacks:
• Replay attacks of same file
• Compromised and vulnerable versions
• Key compromise in signing files
Implemented as Notary by Docker (originally)
http://www.eweek.com/security/cncf-brings-in-notary-the-update-framework-to-
boost-container-security

31. LONDON 18-19 OCT 2018
What Is In-Toto
Motivation:
“Although many frameworks ensuring security in the "last mile" (e.g., software
updaters) exist, they may be providing integrity and authentication to a product that is
already vulnerable; it is possible that, by the time the package makes it to a software
update repository, it has already been compromised.”
Goals:
“in-toto aims to provide integrity, authentication and auditability to the supply chain as
a whole. This means that all the steps within the supply chain are clearly laid out, that
the parties involved in carrying out a step are explicitly stated, and that each step
carried out meets the requirements specified by the actor responsible for this software
product.”

32. LONDON 18-19 OCT 2018
In-Toto Basic Terminology
Materials: the elements used (e.g., files) to perform a step in the supply chain.
Product: the result of carrying out a step. Products are recorded as part of link
metadata.
Link: metadata information gathered while performing a supply chain step or
inspection, signed by the functionary that performed the step or the client that
performed the inspection
Verification: the process by which data and metadata included in the final product is
used to ensure its correctness.

33. LONDON 18-19 OCT 2018
In-Toto Actors
Project Owner: Defines the layout of the software supply chain.
Functionary: Performs a step in the supply chain and provides a piece of link metadata
as a record that such a step was carried out.
Client: Performs verification on the final product by checking the provided layout and
link metadata.

34. LONDON 18-19 OCT 2018
In-Toto Layouts - Steps
• A recipe for taking materials and producing an
output product.
• Steps can be chained, and sub-layouts can be
specified.

35. LONDON 18-19 OCT 2018
In-Toto Layouts - Inspect
• Executes at the final stage of verification to verify
the resultant product matches that specified in the
layout.
• Takes an input list of expected materials and
expected products.
• Returns a go/no-go result.

36. LONDON 18-19 OCT 2018
In-Toto Links
• Record information about the execution
environment.
• Cryptographically signed by the functionary
carrying out the action.

37. LONDON 18-19 OCT 2018
And Finally : In-Toto In Action
A Passing Verification:
A Failing Verification: https://in-toto.github.io/

38. LONDON 18-19 OCT 2018
In-Toto in a Jenkins Server
stage('Build') {
agent {
docker {
#image name here
}
}
steps {
withCredentials([#any credentials here]) {
in_toto_wrap(['stepName': 'Build',
'keyPath': "${WORKER_KEY}",
'transport': "redis://${REDIS_ENDPOINT}:6379"]){
#your actual step here
}
}
}
}
}

39. LONDON 18-19 OCT 2018
Getting It Right By Design :
Cloud Native and Containers

40. LONDON 18-19 OCT 2018
Point Solutions Are Not Enough

41. LONDON 18-19 OCT 2018
What Can You Trust ?
• Git ensures integrity but not identity
• Anyone can pretend to commit as
someone else !
• Most people assume Git is a trusted
source
• Signing and verification are easy
• Enterprise key management not so
much !
https://mikegerwitz.com/papers/git-horror-story
https://medium.com/@pjbgf/spoofing-git-commits-7bef357d72f0

42. LONDON 18-19 OCT 2018
Security-hardened Container Supply Chain
Base Image Code Build Application Image Deploy
Controlled base
images
Hash based
addressing
Static analysis
Dependency analysis
Hermetic
Reproducible
Rootless
Vulnerability
scanning
Configuration
scanning
Admission control
Runtime
configurations
Docker Hub TUF
Notary
Grafeas
In-Toto
Clair
Aqua Microscanner
Kubernetes
Kritis

43. LONDON 18-19 OCT 2018
Securing Builds with Metadata
• Pipeline metadata is rich and varied
• Initiating users and/or events
• Installed dependencies and their versions
• Veracity test data (unit/integration/acceptance tests)
• Security test data
• Data can be used for:
• Recording i.e. audit
• Report/enforcing i.e. policy

44. LONDON 18-19 OCT 2018
Storing Metadata with Google Grafeas
• Google’s open-source project to audit and govern the
software supply chain
• Stores metadata about artefacts and their
vulnerabilities
• Twistlock, Aqua, JFrog Xray, BlackDuck can send
metadata to Grafeas
• Possible to query that metadata to gate builds and
deployments

45. LONDON 18-19 OCT 2018
Grafeas in Action

46. LONDON 18-19 OCT 2018
The Art of the Possible

47. LONDON 18-19 OCT 2018
In-Toto in a Container SDLC

48. LONDON 18-19 OCT 2018
DevSecOps … with In-Toto
CAB

49. LONDON 18-19 OCT 2018
Avoid the Horror
• Practice basic hygiene
• Trust with caution
• Trust but verify
• Understand your abuse cases
• Embrace new ways of working
• Backport the best of new technology

50. LONDON 18-19 OCT 2018
[Last slide for thank you
message, links, etc]
@colindomoney