Slides from the presentation "CI/CD Patterns for Kubernetes with Spinnaker" at Spinnaker Summit 2018, by Andrew Phillips and Lars Wander

1. CI/CD Patterns for
Kubernetes with Spinnaker
Andrew Phillips, Lars Wander
Spinnaker Summit, Oct 8, 2018

2. The bio slide
● Been on most sides of this space:
developer, infra builder, product owner,
evangelist and more
● Long-standing open-source contributor
● Author and regular conference and
meetup presenter
● Co-organizer of ContainerDays Boston &
NYC
● Leading OSS Spinnaker team
● Been working on Spinnaker team for about
3 years
● Mostly worked on Kubernetes integration
& Halyard

3. Agenda
1. What’s the challenge?
2. CI/CD practices & patterns today
3. What does Kubernetes add into the mix?
4. A CI/CD pipeline for Kubernetes with Spinnaker
5. Next steps for your scenario
6. Q&A

4. 1. The context
● Lots of organizations (incl. many new Spinnaker users) looking at Kubernetes
right now
● Trying to use this also as an opportunity to “clean up” sub-optimal software
delivery pipelines
● How to do this right?

5. 1. The challenge
● Lots of new runtime-specific concepts to deal with
● “Kubernetes-native” best practices still very much in flux
● Wrapping your head around “new” practices (e.g. GitOps) is hard
● Extrapolating from new practices in theory to actual, working implementations
is even harder
● Figuring out how all this new stuff relates to accepted industry practices and
tools (e.g. Spinnaker) is harder still

6. 1. The approach
● Recall good practices that we’d like to retain
● Incorporate refinements related to Kubernetes to create a straw man setup
● Develop a mental model to understand advantages and shortfalls of the setup
● Refine the setup based on tradeoffs to be made related to each user’s
particular situation
● Implement using appropriate tools

7. 2. Existing good practice
● Reproducible builds
● Store source and derived artifacts appropriately
● Minimize duplication, especially around environment config
● Keep the business process flexible and the env automation robust
● Support 4 related processes:
○ Application update
○ Environment (config) update
○ Environment spin-up/restore
○ Environment drift detection and remediation
● Specifically, support env-specific (e.g. log settings change) and cross-env
(e.g. new app version rollout) processes concisely

8. Deployment execution: business process vs. technical process
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

9. 2. The four related processes
● Application update
○ “I want to validate a new release candidate and promote it through envs to prod”
● Environment (config) update
○ “I want to change the attributes of a particular env only”
● Environment spin-up/restore
○ “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point
in time”
● Environment diff, drift detection and remediation
○ “I want to understand how the actual config of an env relates to the intended config, also
potentially across different environments”

10. 3. What’s new with Kubernetes
● “as-code” description of what an environment should look like
(“environments-as-code”)
● Actuation based on reconciliation engine built into runtime with continuous
enforcement
○ Interest in “pull-based”, async invocation via repo-watching
● Out-of-the-box support for some types of rollout via Deployment object, as
well as CRDs to define your own
○ But can also manipulate underlying objects directly
● GitOps ~= environments-as-code + async invocation + repo workflow for
business process

11. Deployment execution: adding in Kubernetes
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
“environments-as-code” repositories
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

12. 3. The straw man

13. team1-repo
env-repo
staging-branch prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config

14. App1 update

15. team1-repo
env-repo
staging-branch prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}

16. team1-repo
env-repo
staging-branch
app1.yaml
prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}
Step 1
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: staging
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app1
namespace: staging
...
---
apiVersion: v1
kind: ConfigMap
...

17. team1-repo
env-repo
staging-branch
app1.yaml
prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}
Step 2
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: production
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app1
namespace: production
...
---
apiVersion: v1
kind: ConfigMap
...

18. App2 update

19. team1-repo
env-repo
staging-branch
app1.yaml
prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}

20. team1-repo
env-repo
staging-branch
app1.yaml
app2.yaml
app2-env.yaml
prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 1app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app2
namespace: staging
...
spec:
replicas: 1
...
image:
gcr.io/image@sha256:abcdef
ports:
- name: http
containerPort: 90
---
...
---
apiVersion: v1
kind: ConfigMap
...
data:
config.yaml: |
'app.version': 0.0.4

21. team1-repo
env-repo
staging-branch
app1.yaml
app2.yaml
app2-env.yaml
prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 2
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app2
namespace: production
...
spec:
replicas: 1
...
image:
gcr.io/image@sha256:abcdef
ports:
- name: http
containerPort: 90
---
...
---
apiVersion: v1
kind: ConfigMap
...
data:
config.yaml: |
'app.version': 0.0.4
app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}

22. App2 prod config change

23. team1-repo
env-repo
staging-branch
app1.yaml
app2.yaml
app2-env.yaml
prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': INFOTRACE

24. team1-repo
env-repo
staging-branch
app1.yaml
app2.yaml
app2-env.yaml
prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 1
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': INFOTRACE
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': TRACE

25. 3. The straw man
● Source config in app repo, “compiled” (a.k.a. hydrated) config in env repo(s)
○ Cf. source code in app repo, compiled code in artifact repository
● One app repo per app/team, shared env repos for environments
● Use branches where possible to represent different environments
○ If greater separation via access control is needed, use different repos or consider other
storage
● Keep config in app repo unless it needs to be independent of the app lifecycle
or you really, really need to be able to change it for one env
○ Allows you to use templating/overriding to cut down on duplication
○ The fewer places to keep track of config, the better
● Source config doesn’t have to be “raw” YAML, can be more suitable
abstraction!
○ Cf. higher-level language source code vs. low-level assembly code

26. 3. The straw man
● Prefer explicit invocation over “repo-watching”
○ Avoids “root-level” process running inside cluster, and provides more flexibility for
multi-step rollouts
○ “Repo-watching” makes visualizing current status harder, and requires some sort of
feedback mechanism to distinguish successful from failed deployments in the repo
○ Hard to support phased/multi-step application
○ Easier to reproduce/simulate
● Commit after successful application, not before
○ Avoids having to distinguish attempted from successful deployments in the repo
○ Allows for richer pre-application validation than code diff in a PR (e.g. three-way diff
against actual environment)
○ Avoids commit rights to repo being equivalent to deploy rights to env (and your
automation will need commit rights to make pull requests, unless you used forked repos)
○ Harder if pull requests are used for the business process (as in vanilla GitOps) - requires
multiple branches or “on approve” deployment

27. 4. The mental model
build &
validate
deploy to
prod
deploy to
staging

28. 4. The mental model
build &
validate
deploy to
prod
deploy to
staging
● Often an over-simplification

29. 4. The mental model
● Understand how your app is updated across two dimensions:

30. 4. The mental model
● Understand how your app is updated across two dimensions:
● code change promoted through to
prod
● config change tested in all envs
● updating license for third-party
dependency
● adding debug logging to staging
● configuring a scaling policy based on
a new metric
● rotating database credentials for prod
● updating discovery service endpoint
dependent on app version independent from app version
dependenton
environment
independent
from
environment

31. 4. The mental model
1. Application update
a. “I want to validate a new release candidate and promote it through envs to prod”
2. Environment (config) update
a. “I want to change the attributes of a particular env only”
3. Environment spin-up/restore
a. “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point
in time”
4. Environment diff, drift detection and remediation
a. “I want to understand how the actual config of an env relates to the intended config, also
potentially across different environments”

32. Demo time!

33. 1. Application update

39. 2. Environment update

43. 3. Environment backup/restore (rollback)

46. 4. Environment diff

48. ● App repo is driving deployment, env repo is snapshotting cluster state
○ Think source code in github is to docker image in registry as template in app repo is to
manifest in env repo
● Env repo is just a checkpoint in time, cluster can evolve
○ Kubernetes applies changes to manifests
○ Strategies like exponential rollouts or traffic shifting apply changes over time
● Env repo is not a guaranteed healthy state
○ We can defer snapshotting until some health/success metric… but rollback has no silver bullet
● Deletion from either repo ≠ deletion from cluster
○ Challenge is: not trivial to know if cluster depends, or will depend on manifest not submitted to
repo
○ kubectl apply --prune attempts to solve this with a lot of (scary) caveats
4. The mental model

49. 5. Tuning for your scenario
● Appropriate level of abstraction? How much “raw” Kubernetes YAML
should our developers have access to?
● Where should the abstraction live? In templates? In CRDs? In the
automation tool?
● What to use for templating? Token replacement or overrides?
● When do you snapshot/publish to the env repo? On every change to the
cluster? Or when a desired end-state is reached? (think multi-step rollout)
● Access control: how many repos do you need? Are code repositories right
for your use case, or e.g. a blobstore better?
● Am I distributing or deploying? Helm is much better suited for distribution
than deployment
○ Its templating capability is often used as part of deployment flows, though

50. 5. Implement
● Choose storage implementations and partitioning strategies for your
environments-as-code
● Define the appropriate level of abstraction for your developers and choose
tools to support it
● Choose a flexible automation tool for your deployment business process
● Define an appropriate definition of deployment health/success to determine
when a deployment is “good”
● Decide which of the four processes - app update, env config update, env
restore and env drift detection - you want to support
● Build pipelines
● Done!

51. Try it out!
● Try this out (ping us for a sample) - anph@, lwander@
○ Codelab to try out the Spinnaker Kubernetes v2 provider
● Join #sig-kubernetes on slack
● Come to monthly Spinnaker Kubernetes SIG
● Suggest Spinnaker features to help with this