Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure but we haven't nearly spend enough time to train our self to review, monitor and respond to outages. With the the introduction of CI/CD best practices into our day to day workflows we protect ourselves for introducing "bad" code into production and exposing flaws to our (end-)users. But what about influences from bad actors in- and out-side our projects. This talk will focus on the additional steps we can add to our Waypoint build pipelines to also protect ourselves to so called supply chain attacks while running our jobs in Nomad. We ll discuss scanning for vulnerabilities in incoming code, packages and images and signing the content artefacts we trust before exposing them to our users.

1. Running trusted payloads with
Nomad and Waypoint
Bram Vogelaar
@attachmentgenie

2. $ whoami
•Used to be a Molecular Biologist
•Then became a Dev
•Now an Ops
•Currently Cloud Engineer @ The Factory

3. The intern “solved” the problem with $product
by “fixing” the database connection
Monday Morning: Standup time!

4. Wait! What?

5. #nocontext

6. $problems++?

7. Access Security

8. The discipline of experimenting on a distributed
system in order to build confidence in
the system’s capability to withstand turbulent
conditions in production
CI/CD is in place to provide trust
Chaos Engineering Definition

9. Testing Pyramid

10. Shared Responsibility

11. Waypoint
•Modern workflow to release across platforms.
•Extendable by with (self-written) plugins
•Pipelines written in (H)ashiCorp (C)onfiguration (L)anguage
https://www.waypointproject.io/

12. Common nomenclature for pipeline steps
project = ”lorem-ipsum”
app ”lorem-ipsum” {
build {}
deploy {}
release {}
}

13. Reusable and repeatable scripts plugins
project = ”lorem-ipsum”
app ”lorem-ipsum” {
build {
use "docker" {
buildkit = true
}
}
deploy {
use "exec" {
command = ["docker", "run", "{{.Input.DockerImageFull}}"]
}
}
}

14. Supply Chain Attack Vectors
https://slsa.dev/

15. Don’t trust the internet!
Standing on the shoulders of giants, has pro’s and con’s
Please, Please, use your $language’s favourite package manager
• Npm
• Maven
• Composer
• Rubygems

16. Be Self Sufficient!
Build from local ensures continuity
• OS repositories
• Image repositories
• Code packages

17. Know thy self!
Create a Software Bill of Materials (SBOM)
• What is included in the first place?
• Unwanted Licenses
• Vulnerable Packages / Images

18. Garbage in, garbage out?
app ”lorem-ipsum” {
build {
hook {
when = "before"
command = ["./snyk_scan.sh", "build starting"]
}
hook {
when = "after"
command = ["./artifact_fingerprint.sh", "build finished"]
}
}
}

19. Don’t promote code, but promote artifacts

20. Nomad
•Open Source tool for dynamic workload scheduling
•Batch, containerized, and non-containerized applications.
•Has native Consul and Vault integrations.
•Has token based access setup.
•Jobs written in (H)ashiCorp (C)onfiguration (L)anguage
https://www.nomadproject.io/

21. Waypoint some Nomad
project = ”lorem-ipsum”
app ”lorem-ipsum” {
build {}
deploy {
use "nomad" {
datacenter = ”lab"
namespace = ”theory"
}
}
release {}
}

22. Day to Day Nomad
job "lorem-ipsum" {
group ”frontend" {
network {
port "http" { static = ”8080” }
}
task "server" {
driver = "docker"
config {
image = ”cicero/lorem-ipsum"
ports = ["http"]
}
}
}

23. project = ”lorem-ipsum”
app ”lorem-ipsum” {
build {}
deploy {
use "nomad-jobspec" {
jobspec = templatefile("${path.app}/app.nomad.tpl")
}
}
release {}
}
W
Waypoint some more Nomad

24. Vault
•Open Source tool to do secrets management
•Certificate management
•Secure, store and tightly control access to tokens,
passwords, certificates, encryption keys for protecting
secrets and other sensitive data using a UI, CLI, or HTTP API.
•Supports Password as a Service for tools like SSH and MySQL
https://www.vaultproject.io/

25. Don’t commit secrets!
project = ”lorem-ipsum”
variable "registry_password" {
default = dynamic("vault", {
path = "config/data/secret/registry"
key = "data/registry_password"
})
type = string
sensitive = true
description = "password for registry" // DO NOT COMMIT YOUR PASSWORD TO GIT
}
app ”lorem-ipsum” {

26. Trusted Content
https://goharbor.io/

27. Establish your identity
$ docker trust key generate bram *****
$ docker trust key load key.pem --name bram
$ docker trust signer add --key cert.pem bram registry.example.com/cicero/lorem-ipsum
**** Or have vault generate one! ****

28. Sign your name
$ export DOCKER_CONTENT_TRUST=1
$ docker trust sign registry.example.com/cicero/lorem-ipsum:1
$ docker push registry.example.com/cicero/lorem-ipsum:1
$ docker trust revoke registry.example.com/admin/demo:1

29. Validate before running
$ docker trust inspect --pretty registry.example.com/admin/demo:1
$ export DOCKER_CONTENT_TRUST=1
$ docker pull registry.example.com/user/image:1

30. Supply Chain Attack Vectors
https://slsa.dev/
Q.E.D?

31. Contact
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie

32. Questions ?
The Floor is yours…