Uploaded byAndrew Phillips
Spinnaker Summit 2018: CI/CD Patterns for Kubernetes with Spinnaker
The document outlines CI/CD patterns for Kubernetes using Spinnaker, addressing current challenges and practices in software delivery. It emphasizes the integration of Kubernetes-native best practices, the importance of reproducible builds, and outlines key processes such as application updates and environment drift detection. The presentation also discusses practical implementations and mental models for optimizing deployment pipelines in Kubernetes environments.
More Related Content
Similar to Spinnaker Summit 2018: CI/CD Patterns for Kubernetes with Spinnaker
Spinnaker Summit 2018: CI/CD Patterns for Kubernetes with Spinnaker
- 1. CI/CD Patterns for Kuberneteswith Spinnaker Andrew Phillips, Lars Wander Spinnaker Summit, Oct 8, 2018
- 2. The bio slide ●Been on most sides of this space: developer, infra builder, product owner, evangelist and more ● Long-standing open-source contributor ● Author and regular conference and meetup presenter ● Co-organizer of ContainerDays Boston & NYC ● Leading OSS Spinnaker team ● Been working on Spinnaker team for about 3 years ● Mostly worked on Kubernetes integration & Halyard
- 3. Agenda 1. What’s thechallenge? 2. CI/CD practices & patterns today 3. What does Kubernetes add into the mix? 4. A CI/CD pipeline for Kubernetes with Spinnaker 5. Next steps for your scenario 6. Q&A
- 4. 1. The context ●Lots of organizations (incl. many new Spinnaker users) looking at Kubernetes right now ● Trying to use this also as an opportunity to “clean up” sub-optimal software delivery pipelines ● How to do this right?
- 5. 1. The challenge ●Lots of new runtime-specific concepts to deal with ● “Kubernetes-native” best practices still very much in flux ● Wrapping your head around “new” practices (e.g. GitOps) is hard ● Extrapolating from new practices in theory to actual, working implementations is even harder ● Figuring out how all this new stuff relates to accepted industry practices and tools (e.g. Spinnaker) is harder still
- 6. 1. The approach ●Recall good practices that we’d like to retain ● Incorporate refinements related to Kubernetes to create a straw man setup ● Develop a mental model to understand advantages and shortfalls of the setup ● Refine the setup based on tradeoffs to be made related to each user’s particular situation ● Implement using appropriate tools
- 7. 2. Existing goodpractice ● Reproducible builds ● Store source and derived artifacts appropriately ● Minimize duplication, especially around environment config ● Keep the business process flexible and the env automation robust ● Support 4 related processes: ○ Application update ○ Environment (config) update ○ Environment spin-up/restore ○ Environment drift detection and remediation ● Specifically, support env-specific (e.g. log settings change) and cross-env (e.g. new app version rollout) processes concisely
- 8. Deployment execution: businessprocess vs. technical process Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 9. 2. The fourrelated processes ● Application update ○ “I want to validate a new release candidate and promote it through envs to prod” ● Environment (config) update ○ “I want to change the attributes of a particular env only” ● Environment spin-up/restore ○ “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point in time” ● Environment diff, drift detection and remediation ○ “I want to understand how the actual config of an env relates to the intended config, also potentially across different environments”
- 10. 3. What’s newwith Kubernetes ● “as-code” description of what an environment should look like (“environments-as-code”) ● Actuation based on reconciliation engine built into runtime with continuous enforcement ○ Interest in “pull-based”, async invocation via repo-watching ● Out-of-the-box support for some types of rollout via Deployment object, as well as CRDs to define your own ○ But can also manipulate underlying objects directly ● GitOps ~= environments-as-code + async invocation + repo workflow for business process
- 11. Deployment execution: addingin Kubernetes Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment “environments-as-code” repositories Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 12. 3. The strawman
- 13.
- 14.
- 15. team1-repo env-repo staging-branch prod-branch master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind:Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }}
- 16. team1-repo env-repo staging-branch app1.yaml prod-branch master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name:app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }} Step 1 apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app: app1 env: staging --- apiVersion: apps/v1beta2 kind: Deployment metadata: name: app1 namespace: staging ... --- apiVersion: v1 kind: ConfigMap ...
- 17. team1-repo env-repo staging-branch app1.yaml prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name:app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }} Step 2 apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app: app1 env: production --- apiVersion: apps/v1beta2 kind: Deployment metadata: name: app1 namespace: production ... --- apiVersion: v1 kind: ConfigMap ...
- 18.
- 19. team1-repo env-repo staging-branch app1.yaml prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config app.version: 0.0.34 port: 8090 image:gcr.io/image@sha256:{{ container.image.digest }}
- 20. team1-repo env-repo staging-branch app1.yaml app2.yaml app2-env.yaml prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 1app.version: 0.0.34 port:8090 image: gcr.io/image@sha256:{{ container.image.digest }} apiVersion: apps/v1beta2 kind: Deployment metadata: name: app2 namespace: staging ... spec: replicas: 1 ... image: gcr.io/image@sha256:abcdef ports: - name: http containerPort: 90 --- ... --- apiVersion: v1 kind: ConfigMap ... data: config.yaml: | 'app.version': 0.0.4
- 21. team1-repo env-repo staging-branch app1.yaml app2.yaml app2-env.yaml prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 2 apiVersion: apps/v1beta2 kind:Deployment metadata: name: app2 namespace: production ... spec: replicas: 1 ... image: gcr.io/image@sha256:abcdef ports: - name: http containerPort: 90 --- ... --- apiVersion: v1 kind: ConfigMap ... data: config.yaml: | 'app.version': 0.0.4 app.version: 0.0.34 port: 8090 image: gcr.io/image@sha256:{{ container.image.digest }}
- 22. App2 prod configchange
- 23. team1-repo env-repo staging-branch app1.yaml app2.yaml app2-env.yaml prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config apiVersion: v1 kind: ConfigMap metadata: name:app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': INFOTRACE
- 24. team1-repo env-repo staging-branch app1.yaml app2.yaml app2-env.yaml prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 1 apiVersion: v1 kind:ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': INFOTRACE apiVersion: v1 kind: ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': TRACE
- 25. 3. The strawman ● Source config in app repo, “compiled” (a.k.a. hydrated) config in env repo(s) ○ Cf. source code in app repo, compiled code in artifact repository ● One app repo per app/team, shared env repos for environments ● Use branches where possible to represent different environments ○ If greater separation via access control is needed, use different repos or consider other storage ● Keep config in app repo unless it needs to be independent of the app lifecycle or you really, really need to be able to change it for one env ○ Allows you to use templating/overriding to cut down on duplication ○ The fewer places to keep track of config, the better ● Source config doesn’t have to be “raw” YAML, can be more suitable abstraction! ○ Cf. higher-level language source code vs. low-level assembly code
- 26. 3. The strawman ● Prefer explicit invocation over “repo-watching” ○ Avoids “root-level” process running inside cluster, and provides more flexibility for multi-step rollouts ○ “Repo-watching” makes visualizing current status harder, and requires some sort of feedback mechanism to distinguish successful from failed deployments in the repo ○ Hard to support phased/multi-step application ○ Easier to reproduce/simulate ● Commit after successful application, not before ○ Avoids having to distinguish attempted from successful deployments in the repo ○ Allows for richer pre-application validation than code diff in a PR (e.g. three-way diff against actual environment) ○ Avoids commit rights to repo being equivalent to deploy rights to env (and your automation will need commit rights to make pull requests, unless you used forked repos) ○ Harder if pull requests are used for the business process (as in vanilla GitOps) - requires multiple branches or “on approve” deployment
- 27. 4. The mentalmodel build & validate deploy to prod deploy to staging
- 28. 4. The mentalmodel build & validate deploy to prod deploy to staging ● Often an over-simplification
- 29. 4. The mentalmodel ● Understand how your app is updated across two dimensions:
- 30. 4. The mentalmodel ● Understand how your app is updated across two dimensions: ● code change promoted through to prod ● config change tested in all envs ● updating license for third-party dependency ● adding debug logging to staging ● configuring a scaling policy based on a new metric ● rotating database credentials for prod ● updating discovery service endpoint dependent on app version independent from app version dependenton environment independent from environment
- 31. 4. The mentalmodel 1. Application update a. “I want to validate a new release candidate and promote it through envs to prod” 2. Environment (config) update a. “I want to change the attributes of a particular env only” 3. Environment spin-up/restore a. “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point in time” 4. Environment diff, drift detection and remediation a. “I want to understand how the actual config of an env relates to the intended config, also potentially across different environments”
- 32.
- 33.
- 39.
- 43. 3. Environment backup/restore(rollback)
- 46.
- 48. ● App repois driving deployment, env repo is snapshotting cluster state ○ Think source code in github is to docker image in registry as template in app repo is to manifest in env repo ● Env repo is just a checkpoint in time, cluster can evolve ○ Kubernetes applies changes to manifests ○ Strategies like exponential rollouts or traffic shifting apply changes over time ● Env repo is not a guaranteed healthy state ○ We can defer snapshotting until some health/success metric… but rollback has no silver bullet ● Deletion from either repo ≠ deletion from cluster ○ Challenge is: not trivial to know if cluster depends, or will depend on manifest not submitted to repo ○ kubectl apply --prune attempts to solve this with a lot of (scary) caveats 4. The mental model
- 49. 5. Tuning foryour scenario ● Appropriate level of abstraction? How much “raw” Kubernetes YAML should our developers have access to? ● Where should the abstraction live? In templates? In CRDs? In the automation tool? ● What to use for templating? Token replacement or overrides? ● When do you snapshot/publish to the env repo? On every change to the cluster? Or when a desired end-state is reached? (think multi-step rollout) ● Access control: how many repos do you need? Are code repositories right for your use case, or e.g. a blobstore better? ● Am I distributing or deploying? Helm is much better suited for distribution than deployment ○ Its templating capability is often used as part of deployment flows, though
- 50. 5. Implement ● Choosestorage implementations and partitioning strategies for your environments-as-code ● Define the appropriate level of abstraction for your developers and choose tools to support it ● Choose a flexible automation tool for your deployment business process ● Define an appropriate definition of deployment health/success to determine when a deployment is “good” ● Decide which of the four processes - app update, env config update, env restore and env drift detection - you want to support ● Build pipelines ● Done!
- 51. Try it out! ●Try this out (ping us for a sample) - anph@, lwander@ ○ Codelab to try out the Spinnaker Kubernetes v2 provider ● Join #sig-kubernetes on slack ● Come to monthly Spinnaker Kubernetes SIG ● Suggest Spinnaker features to help with this