Several recent high-profile security incidents were due to compromised software supply chains. Software Supply Chain is a collective term used to describe the stages of software lifecycle from source to deployment through CI/CD pipelines, and all the static and dynamic analyses in between. In the world of microservices and cloud computing, trust in your company’s supply chain is critical, as most of the tooling and dependencies are from open source and vendor projects. When the code hits production, it’s essential to have enough observability to detect and investigate the problem and get to the root cause and mitigation as quickly as possible. With software supply chain attacks, not only is the newly deployed code under suspicion, but also all the tooling used to produce it becomes a potential attack vector, so an efficient and effective way to verify the integrity of the supply chain is paramount. This talk will discuss what information needs to be collected to allow DevOps to inspect and verify the integrity of the supply chain, the challenges of having the right level of detail to reduce mean-time-to-detection and mean-time-to-understanding, some of the existing solutions and open problems in this space.

1. Software Supply Chains for DevOps
Aysylu Greenberg
July 20, 2021

2. Speaker introduction
● Sr Software Engineer @ Google
● Tech Lead of GCP Container Analysis
and Container Scanning team
● Maintainer of Grafeas and Kritis OSS
projects
● Track Host of the inaugural Software
Supply Chain track @ QCon SF 2019
● @aysylu22
Aysylu Greenberg

3. What do these 3 have in common?
EventStream

4. Proprietary + Confidential

5. Modern Challenges in Software Supply Chains

6. Modern Challenges in Software Supply Chains
https://security.googleblog.com/2021/06/introducing-slsa-end-to-end-framework.html

7. Proprietary + Confidential
Software Supply Chains = SSC

8. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

9. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

10. Software Supply Chains
Source code
Test
OSS &
proprietary libs
Build
Deploy to
Production
Deploy to
Staging
Incident
Monitor
workloads
Patch
Management

11. Software Supply Chains
Source code
Test
OSS &
proprietary libs
Build
Deploy to
Production
Deploy to
Staging
Monitor
workloads
CI pipeline
Incident
Patch
Management

12. Software Supply Chains
Source code
Test
OSS &
proprietary libs
Build
Deploy to
Production
Deploy to
Staging
Monitor
workloads
CD pipeline
Incident
Patch
Management

13. Software Supply Chains
Source code
Test
OSS &
proprietary libs
Build
Deploy to
Production
Deploy to
Staging
Monitor
workloads
Observability tools
Incident
Patch
Management

14. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

15. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

16. Source to Prod
CI/CD pipelines
Build &
Deploy
Build process
Automated test,
scan, analysis
Deploy
checks
Production

17. Incident
CI/CD pipelines
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
Build process

18. Incident: O11y to the Rescue
CI/CD pipelines
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
Build process

19. Incident: from Prod to Source
CI/CD pipelines
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
?
?
?
?
Build process

20. Incident: from Prod to Source
CI/CD pipelines
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
?
?
?
?
?
Build process

21. Incident: from Prod to Source
Universal Artifact Metadata
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
?
?
?
?
?
Build process

22. Incident: from Prod to Source
Universal Artifact Metadata
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
?
?
?
?
?
Build process

23. What level of
detail should we
collect?

24. Collect everything!
You never know when you’ll need that info and fine
granularity helps to avoid o11y gaps.
What level of
detail should we
collect?

25. Am I going to get
overwhelmed
with information?

26. The right signal to noise ratio is important.
Example: report all vulnerabilities on an image. False
positives?
Am I going to get
overwhelmed
with information?

28. The right signal to noise ratio is important.
Example: report all vulnerabilities on an image. False
positives?
Compliance (e.g. FedRAMP) requires detailed
information about software supply chain.
Am I going to get
overwhelmed
with information?

29. How to efficiently
query for these
details?

30. How to efficiently
query for these
details?
Need vertical and horizontal querying.

31. Horizontal query?
Across all artifacts with a specific property.
"Find all images that are built from a particular
Github commit that is known to have introduced a
security problem."

32. Vertical query?
Metadata across SDLC for a specific artifact.
"Find all source, build, test, vulnerabilities, and
deployments metadata for a container image."

33. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

34. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

35. Software Supply Chains Solutions
Universal Artifact Metadata
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
Build process

36. Software Supply Chains Solutions
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
Build process
Universal Artifact Metadata

37. github.com/grafeas/grafeas
grafeas.io
API to audit and govern your software supply chain
Artifact
metadata API:
Grafeas

38. Container Analysis API on GCP

39. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

40. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

41. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s",
*projectID),
Filter: `kind = "DEPLOYABLE"`,
})

42. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

43. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range
occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)

44. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

45. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if
occ.GetDeployment().GetUndeployTime() ==
nil {
// The workload is still running.
runningImgs[occ.GetResourceUri(
)] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]

46. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

47. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

48. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

49. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ :=
client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent:
fmt.Sprintf("projects/%s", *projectID),
Filter:
fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})

50. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
for _, img := range runningImgs {
buildResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: fmt.Sprintf(`kind="BUILD_DETAILS" AND
resourceUrl=%q`, img),
})
// assume each build generates a unique container
image URI
build := buildResp.GetOccurrences()[0]
if time.Now().Sub(build.GetCreateTime().AsTime()) >
*buildHorizon {
log.Infof("Workload running image %s is
exceeding build horizon policy of %q.", img, *buildHorizon)
}
}

51. Proprietary + Confidential
Proprietary + Confidential
Find all
workloads that
exceed build
horizon
occResp, _ := client.ListOccurrences(ctx,
&gpb.ListOccurrencesRequest{
Parent: fmt.Sprintf("projects/%s", *projectID),
Filter: `kind = "DEPLOYABLE"`,
})
for _, occ := range occResp.GetOccurrences() {
if occ.GetDeployment().GetUndeployTime() == nil {
// The workload is still running.
runningImgs[occ.GetResourceUri()] = true
}
}
…
// assume each build generates a unique container image
URI
build := buildResp.GetOccurrences()[0]
if
time.Now().Sub(build.GetCreateTime().
AsTime()) > *buildHorizon {
log.Infof("Workload running
image %s is exceeding build horizon
policy of %q.", img, *buildHorizon)
}
}

52. Software Supply Chains Solutions
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
��
Build process
Universal Artifact Metadata

53. Deploy-time Policy Enforcer for Kubernetes applications
Deploy Policy Checks:
Kritis
github.com/grafeas/kritis

54. github.com/grafeas/voucher
Voucher creates attestations for Binary Authorization
Deploy Policy Checks:
Kritis & Voucher
��
github.com/grafeas/kritis

55. Deploy Policy Checks:
Binary Authorization on GCP

56. Software Supply Chains Solutions
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
��
Build process
Universal Artifact Metadata

57. spdx.dev
Open standard for communicating software bill of material
information, including components, licenses, copyrights, and
security references
Software Bill of
Materials

58. Software Supply Chains Solutions
Automated test,
scan, analysis
Deploy
checks
Production
Build &
Deploy
��
Build process
Universal Artifact Metadata

59. in-toto.io
A framework to secure the integrity of software supply chains
Integrity of
Supply Chain

60. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

61. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

62. Proprietary + Confidential
Secure builds and supply chain
01
04
03
Some Open Problems (not a complete list)
05
Supply chain integrity
Data quality in vulnerability scanning
Security and integrity in OSS
02 Compliance

63. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps

64. Proprietary + Confidential
Takeaways
Software supply
chains = SDLC in
Cloud
01

65. Proprietary + Confidential
Takeaways
Software supply
chains = SDLC in
Cloud
01 02
SSC metadata
needed to help
DevOps

66. Proprietary + Confidential
Takeaways
Software supply
chains = SDLC in
Cloud
01 02 03
SSC metadata
needed to help
DevOps
Existing solutions
(Grafeas, SBOM, Kritis,
in-toto),
challenges & open
problems

67. Proprietary + Confidential
Today
Q & A
Software
Supply Chains
Open
problems
Existing
Solutions
SSC for
DevOps