https://bit.ly/supply_chain_security_for_containers
chukmunnlee@nus.edu.sg
Supply Chain
Security for
Containerised
Workloads
https://bit.ly/supply_chain_security_for_containers
Objective
Introduce a set of tools for working with containers
(Integratable into your CI)
https://bit.ly/supply_chain_security_for_containers
Trend - Kubernetes Adoption
https://www.cncf.io/wp-content/uploads/2022/02/CNCF-AR_FINAL-edits-15.2.21.pdf
https://bit.ly/supply_chain_security_for_containers
Why are Containers Gaining Popularity?
Development Distribution Deployment
https://bit.ly/supply_chain_security_for_containers
Difference Between Virtual Machines and Containers
Hardware
Hypervisor
Applicat
ion
OS
Applicat
ion
OS
Applicat
ion
OS
Hardware
Linux
Applicat
ion
Applicat
ion
Applicat
ion
Virtual machine
virtualizes the hardware
Network namespace
virtualizes Linux
https://bit.ly/supply_chain_security_for_containers
What is in a Container?
Configurations Runtime
Code
Environment Dependencies
https://bit.ly/supply_chain_security_for_containers
What is Docker (and other container tools)?
A set of tools and engine for
- Packaging applications into a standard format, referred to as an ‘image’
- Deploying images as containers into Linux namespaces
- Manages the containers, networking, volumes, resources, etc
Used the following core Linux technologies
- Network namespace
- CGroups
- Union filesystem
A write once deploy anywhere
- Anywhere that Docker is installed or on anywhere with OCI compliant runtime installed
https://bit.ly/supply_chain_security_for_containers
Dockerfile FROM node:20
WORKDIR /app
COPY config.json
COPY main.js .
COPY public .
COPY package.json .
RUN npm ci
ENV PORT=3000
EXPOSE ${PORT}
ENTRYPOINT node main
Runtime
Environment
Dependencies
Code
Configurations
BUILD
RUN
https://bit.ly/supply_chain_security_for_containers
Docker Workflow
Base image
Dockerfile Your application’s
image
Containers
Container registry
Run
Build
Push
Developer Run
Pull
Dependencies
Application and
artifacts
https://bit.ly/supply_chain_security_for_containers
Software Supply Chain Security
Modern cloud native application relies and uses
- Platform
- Code generators
- Libraries and dependencies
Attackers insert malicious code into platform, compilers and code generators,
libraries, etc
- Use these as the attack vector rather than attacking your application directly
Containerised applications supply chain
- Libraries and dependencies
- Pulling unverified images or from dubious image registries
- Badly written Dockerfile
https://bit.ly/supply_chain_security_for_containers
https://www.redhat.com/en/resources/kubernetes-adoption-security-market-trends-overview
Your software
supply chain
https://bit.ly/supply_chain_security_for_containers
Security Challenges
Base image
Dockerfile
Containers
Container registry
Run
Build
Push
Developer Run
Pull
Badly written
Unverified base image
Dependencies
Application and
artifacts
Pulling a
compromised image
Malicious code
in containers
Your application’s
image
https://bit.ly/supply_chain_security_for_containers
Container Image Integrity
Scan code base,
dependences,
configurations for
exposed secrets
https://bit.ly/supply_chain_security_for_containers
Scanning Image and Filesystem
trivy - tool to scan vulnerabilities in filesystem, repository, image
- Scan secrets, config, vulnerability for CVE listings
- Produces report in JSON, table, SARIF
- Generate SBOM (later)
trivy filesystem --format=json --output=scan.json 
--severity=HIGH,CRITICAL <src path>
trivy image --scanners=vuln --exit-code=1 
-s HIGH,CRITICAL node:20
trivy filesystem -s CRITICAL --secret-config policy.yaml .
Scan report format
Severity Exit code, eg. using
in CI pipeline
Only scan for
vulnerabilities
Custom rules
https://bit.ly/supply_chain_security_for_containers
Infrastructure as Code
Servers Networking
Storage and
databases
Monitoring Security
Iac Script
Virtualized infrastructure
Provision
● Process of provisioning and
managing IT resources through
machine readable files
● Scripts describe the required setup,
configurations and dependencies
between resources
● IaC tools provision the resources
PaaS
https://bit.ly/supply_chain_security_for_containers
Container Image Integrity
Analysis
Policy
Ensure that code
and IaC conforms
to best practices
Scan code base,
dependences,
configurations for
exposed secrets
Image
Build
https://bit.ly/supply_chain_security_for_containers
Docker Image Tag
Tagging image
- Never - no tag when building image, default to :latest
- Good - give meaningful names to tags, eg v1.0.0
- Better - use the tag to associate the image to a commit eg a release branch
- git rev-parse --short HEAD
Using image
- Never - use the :latest tag
- Good - use a tag
- Best - use the image digest instead of tag
- The image’s hash
- docker image -f ‘{{.RepoDigests}}’ <image>
https://bit.ly/supply_chain_security_for_containers
About Image Tags
docker build -t fred/myapp:v1 .
docker build -t fred/myapp:v1 .
Overwritten the previous image
with the same tag
FROM fred/myapp:v1
Might be using the incorrect image
sha256:b24cde…
sha256:3fbc6…
Each image produce a
different hash after pushing to
container registry
FROM fred/myapp@sha256:3fbc6…
No ambiguity
https://bit.ly/supply_chain_security_for_containers
Enforce Policy
conftest - evaluate Dockerfile against as set of rules
- https://www.conftest.dev/
- Scan results can be output as JUnit, Github
Rules are written with Rego, a Prolog like language
- https://www.openpolicyagent.org/docs/latest/#rego
- Free course available - https://academy.styra.com/courses/opa-rego
Open Policy Agent is general purpose tool to enforce policy
- https://www.openpolicyagent.org/docs/latest/
- Decouple policies from application
conftest test Dockerfile --output github --policy policy
https://bit.ly/supply_chain_security_for_containers
Example of Policy File
deny[msg] {
input[i].Cmd = “from”
val = input[i].Value
contains(val[_], “:latest”)
msg = sprintf(“Cannot use :latest tag %s”, val)
}
deny[msg] {
input[i].Cmd = “from”
val = input[i].Value
count(split(val, “:”)) < 2
msg = sprintf(“Please add an image tag to %s”, val)
}
More examples https://www.conftest.dev/examples
Policy
Prevent pulling images
with the ‘latest’ tag
https://bit.ly/supply_chain_security_for_containers
Container Image Integrity
Sign
Analysis
Image
Policy
Build
Image’s signature
Verify
Ensure that code
and IaC conforms
to best practices
Validate that the
image has not
been tempered
Scan code base,
dependences,
configurations for
exposed secrets
https://bit.ly/supply_chain_security_for_containers
Signing Images
cosign is tool for signing and verify container images and artifacts
- Generate public/private key pair for signing
- Signing and verify image signature
- Attaching and signing container artifacts like SBOM
Generate a key pair
- Produces cosign.key and cosign.pub
cosign generate-key-pair
https://bit.ly/supply_chain_security_for_containers
Signing and Verifying Images
cosign sign --key=path/to/cosign.key myimage@sha256:12345…
cosign verify --key=path/to/cosign.pub myimage@sha256:12345…
Use the image digest when you sign the
image. Then the signature is related to an
immutable image reference
Verify the image using the image digest
https://bit.ly/supply_chain_security_for_containers
Signing Image
cosign rekor
Record in
Sign
Produce
https://bit.ly/supply_chain_security_for_containers
Validating Image - 2
cosign triangulate myimage@sha256:12345…
crane manifest <cosign signature>
rekor-cli get --log-index <index>
Extract co-located cosign
signature from image
Get the signature’s manifest
Get the entry from Rekor
https://bit.ly/supply_chain_security_for_containers
Software Bill of Materials
List of inventories and dependencies that make up an
application
- Identifies the software components within larger pieces of
software and the licenses associated with their components
- In the context of containers, these will be the contents of the
image, its dependencies, licenses, etc
Purpose
- Track updates and known security vulnerabilities for each
component in the software project’s dependencies
- Auditing purposes as it ensures that only authorized
dependencies are included in a software project
Standards
- SPDX (Software Packet Data Exchange) - https://spdx.dev/
- CycloneDX - https://cyclonedx.org/
Image from https://spdx.github.io/spdx-spec/v2.2.2/composition-of-an-SPDX-document/
https://bit.ly/supply_chain_security_for_containers
Container Image Integrity
Sign
Analysis
Image SBOM
Policy
Build Produce
Sign
Image’s signature Attestation
Verify Verify
Ensure that code
and IaC conforms
to best practices
Validate that the
image has not
been tempered
Tracks dependencies
for security, licensing,
etc
Confirmation of
the provenance of
the image
Scan code base,
dependences,
configurations for
exposed secrets
https://bit.ly/supply_chain_security_for_containers
Image SBOM and Attestation
trivy image --format=spdx --output=sbom.spdx 
myimage@sha256:12345…
cosign attach sbom --type=spdx --sbom=sbom.spdx 
myimage@sha256:12345…
cosign attest --key=path/to/cosign.key 
--type=spdx --predicate=sbom.spdx 
myimage@sha256:12345…
cosign verify-attestation --key=path/to/cosign.pub 
--type=spdx myimage@sha256:12345…
Generate SBOM
in SPDX format
Attach SBOM to
the image
Create an attestation
by signing the SBOM
Verify the attestation
https://bit.ly/supply_chain_security_for_containers
Scan SBOM
cosign download sbom --output-file sbom.spdx 
myimage@sha256:12345…
trivy sbom --severity HIGH,CRITICAL sbom.spdx
Download SBOM for
the given image
Scan the SBOM for
vulnerabilities
https://bit.ly/supply_chain_security_for_containers
Tools
Sign
Analysis
Image SBOM
Policy
Build Produce
Sign
Image’s signature Attestation
Verify Verify
trivy
Scan for
vulnerabilities
in code
conftest
Enforce best
practices
cosign
Sign and verify
image and BOM
https://bit.ly/supply_chain_security_for_containers
My Medium Article - CI/CD Pipeline
Argo Workflow — A Pipeline to Build and Deploy Containers
https://bit.ly/supply_chain_security_for_containers

Supply Chain Security for Containerised Workloads - Lee Chuk Munn