Uploaded byAndrew Phillips
Docker New York City: From GitOps to a scalable CI/CD Pattern for Kubernetes
The document outlines the challenges and opportunities organizations face when implementing CI/CD practices in Kubernetes, emphasizing the need for integrating GitOps and Kubernetes-native best practices. It provides a framework for creating a scalable CI/CD pipeline that incorporates application updates, environment updates, and drift detection, while advocating for a mental model that helps users navigate the complexities of Kubernetes configurations. Key considerations include the management of environment-specific configurations, the relationship between app repositories and environment repositories, and the importance of automation in achieving effective deployment processes.
More Related Content
Similar to Docker New York City: From GitOps to a scalable CI/CD Pattern for Kubernetes
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Docker New York City: From GitOps to a scalable CI/CD Pattern for Kubernetes
- 1. From GitOps toan scalable CI/CD pattern for Kubernetes Andrew Phillips Docker New York City; Jan 16, 2019
- 2. From GitOps toan scalable CI/CD pattern...for everything? Andrew Phillips Docker New York City; Jan 16, 2019
- 3. The bio slide ●Been on most sides of this space: developer, infra builder, product owner, evangelist and more ● Long-standing open-source contributor ● Author and regular conference and meetup presenter ● Co-organizer of ContainerDays Boston & NYC
- 4. Agenda 1. What’s thechallenge? 2. CI/CD practices & patterns today 3. What does Kubernetes add into the mix? 4. A CI/CD pipeline for Kubernetes 5. Next steps for your scenario 6. Q&A
- 5. The context ● Lotsof organizations looking at Kubernetes right now ● Trying to use this also as an opportunity to “clean up” sub-optimal software delivery pipelines ● How to do this right?
- 6. The challenge ● Lotsof new runtime-specific concepts to deal with ● “Kubernetes-native” best practices still very much in flux ● Wrapping your head around “new” practices (e.g. GitOps) is hard ● Extrapolating from new practices in theory to actual, working implementations is even harder ● Figuring out how all this new stuff relates to accepted industry practices and tools is harder still
- 7. Failure mode: CCAOA CargoCult All Over Again
- 8. The approach ● Recallgood practices that we’d like to retain ● Incorporate refinements related to Kubernetes to create a straw man setup ● Develop a mental model to understand advantages and shortfalls of the setup ● Refine the setup based on tradeoffs to be made related to each user’s particular situation ● Implement using appropriate tools
- 9. Existing good practice ●Reproducible builds ● Store source and derived artifacts appropriately ● Minimize duplication, especially around environment config ● Keep the business process flexible and the env automation robust ● Support 4 related processes: ○ Application update ○ Environment (config) update ○ Environment spin-up/restore ○ Environment drift detection and remediation ● Specifically, support env-specific (e.g. log settings change) and cross-env (e.g. new app version rollout) processes concisely
- 10. Deployment execution: businessprocess vs. technical process Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 11. What’s new withKubernetes ● Declarative == intent/desired state is front-of-mind ● “as-code” description of the intended state of an environment (“environments-as-code”) ● Purpose of automation is to modify that intent and apply it to the target runtime(s) ● Needed UX (generally poor currently!) is to see/understand current intent; review and approve changes to intent; and track status of intent (“being applied”, “successfully applied”, “failed”, “drifted” etc.)
- 12. Deployment execution: addingin Kubernetes Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment “environments-as-code” repositories Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 13. TL;DR Release pipeline Abusiness process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment “environments-as-code” repositories Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace Imperative pipeline across environments Declarative spec for each environment (with support for some imperative “cheating” where necessary)
- 14. What’s popular inthe Kubernetes community ● Actuation based on reconciliation engine built into runtime with continuous enforcement ○ Interest in “pull-based”, async invocation via repo-watching ● Out-of-the-box support for some types of rollout via Deployment object, as well as CRDs to define your own ○ But can also manipulate underlying objects directly ● GitOps ~ environments-as-code + async invocation + repo workflow for business process
- 15.
- 16. ● App repois driving deployment, env repo is snapshotting desired cluster state ○ Think source code in github is to container image in registry as template in app repo is to manifest in env repo ● Env repo is just a checkpoint in time, cluster can evolve ○ Kubernetes applies changes to manifests ○ Strategies like exponential rollouts or traffic shifting apply changes over time ● Env repo is not a guaranteed healthy state ○ We can defer snapshotting until some health/success metric… but rollback has no silver bullet ● Deletion from either repo ≠ deletion from cluster ○ Challenge is: not trivial to know if cluster depends, or will depend on manifest not submitted to repo ○ kubectl apply --prune attempts to solve this with a lot of (scary) caveats Mental model
- 17. One team, oneenv, app pipeline only
- 18.
- 19.
- 20. env-repo applied-staging-branch applied-prod-branch team1-repo master- branch service config deployment + overrides Valuesthat need to change across in variables: could be specified as patches, or variables, or...
- 21. Updating settings fora single env only
- 22. The mental model ●Understand how your app is updated across two dimensions: ● code change promoted through to prod ● common externalized config setting, e.g. localized title ● rotating database credentials for prod ● updating discovery service endpoint ● adding debug logging to staging ● configuring a scaling policy in prod based on a new metric dependent on app version independent of environment dependent on environment independent of app version
- 23. The mental model ●Understand how your app is updated across two dimensions: Application update Static environment config update (App-linked) environment config update dependent on app version independent of environment dependent on environment independent of app version
- 24. The mental model ●Understand how your app is updated across two dimensions: Application update Static environment config update (App-linked) environment config update dev responsibility ~ app repo platform responsibility ~ env repo
- 25. team1-repo env-repo applied-staging-branch applied-prod-branch master- branch service config + overrides staging- branch deployme nt prod- branch deployme nt Settings canonly be updated by running the full app pipeline, requiring an app rebuild. Advantage is no config duplication Settings in the env can be updated without rebuilding the app. Downside is more config duplication.
- 26.
- 27.
- 28. The four relatedprocesses ● Application update ○ “I want to validate a new release candidate and promote it through envs to prod” ● Environment (config) update ○ “I want to change the attributes of a particular env only” ● Environment spin-up/restore ○ “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point in time” ● Environment diff, drift detection and remediation ○ “I want to understand how the actual config of an env relates to the intended config, also potentially across different environments”
- 29.
- 30. team1-repo env-repo applied-staging-branch applied-prod-branch master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind:Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }}
- 31. team1-repo env-repo applied-staging-branch app1.yaml applied-prod-branch master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name:app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }} Step 1 apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app: app1 env: staging --- apiVersion: apps/v1beta2 kind: Deployment metadata: name: app1 namespace: staging ... --- apiVersion: v1 kind: ConfigMap ...
- 32. team1-repo env-repo applied-staging-branch app1.yaml applied-prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod-bra nch config apiVersion: v1 kind: Service metadata: name:app1-service ... spec: type: ClusterIP ports: - port: 808080 targetPort: http protocol: TCP name: http selector: app: app1 env: {{ environment }} Step 2 apiVersion: v1 kind: Service metadata: name: app1-service ... spec: type: ClusterIP ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app: app1 env: production --- apiVersion: apps/v1beta2 kind: Deployment metadata: name: app1 namespace: production ... --- apiVersion: v1 kind: ConfigMap ...
- 33.
- 34. team1-repo env-repo applied-staging-branch app1.yaml applied-prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config app.version: 0.0.34 port: 8090 image:gcr.io/image@sha256:{{ container.image.digest }}
- 35. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 1app.version: 0.0.34 port:8090 image: gcr.io/image@sha256:{{ container.image.digest }} apiVersion: apps/v1beta2 kind: Deployment metadata: name: app2 namespace: staging ... spec: replicas: 1 ... image: gcr.io/image@sha256:abcdef ports: - name: http containerPort: 90 --- ... --- apiVersion: v1 kind: ConfigMap ... data: config.yaml: | 'app.version': 0.0.4
- 36. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 2 app.version: 0.0.34 port:8090 image: gcr.io/image@sha256:{{ container.image.digest }} apiVersion: apps/v1beta2 kind: Deployment metadata: name: app2 namespace: production ... spec: replicas: 1 ... image: gcr.io/image@sha256:abcdef ports: - name: http containerPort: 90 --- ... --- apiVersion: v1 kind: ConfigMap ... data: config.yaml: | 'app.version': 0.0.4
- 37. App2 prod configchange
- 38. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config apiVersion: v1 kind: ConfigMap metadata: name:app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': INFOTRACE
- 39. team1-repo env-repo applied-staging-branch app1.yaml app2.yaml app2-env.yaml applied-prod-branch app1.yaml app2.yaml app2-env.yaml master- branch service config staging- branch deployme nt prod- branch deployme nt team2-repo master- branch custom-s ettings staging- branch config prod- branch config Step 1 apiVersion: v1 kind:ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': INFOTRACE apiVersion: v1 kind: ConfigMap metadata: name: app2-env-config namespace: production labels: app: app2 env: production data: config.yaml: | 'log.level': TRACE
- 40. The four relatedprocesses Demonstrated: ● Application update ● Environment (config) update Should be relatively easy to see how to do the following with the same data: ● Environment spin-up/restore ● Environment diff, drift detection and remediation
- 41. Straw man ina nutshell ● Source config in app repo, “compiled” (a.k.a. hydrated) config in env repo(s) ○ Cf. source code in app repo, compiled code in artifact repository ● One app repo per app/team, shared env repos for environments ● Use branches where possible to represent different environments ○ If greater separation via access control is needed, use different repos or consider other storage ● Keep config in app repo unless it needs to be independent of the app lifecycle or you really, really need to be able to change it for one env ○ Allows you to use templating/overriding to cut down on duplication ○ The fewer places to keep track of config, the better ● Source config doesn’t have to be “raw” YAML, can be more suitable abstraction! ○ Cf. higher-level language source code vs. low-level assembly code
- 42. A quick refresheron the overall picture... Release pipeline A business process, represented as a sequence, possibly very specific to a service Test environment Staging environment Technical components with interdependencies, defined “as-code”. To be automatically sequenced if possible Production environment “environments-as-code” repositories Deploy to Test Review Approve Deploy to Prod... App Endpoint Config App Endpoint Config’ Old app version Namespace
- 43. Implementation thoughts ● Preferexplicit invocation over “repo-watching” ○ Avoids “root-level” process running inside cluster, and provides more flexibility for multi-step rollouts ○ “Repo-watching” makes visualizing current status harder, and requires some sort of feedback mechanism to distinguish successful from failed deployments in the repo ○ Hard to support phased/multi-step application ○ Easier to reproduce/simulate
- 44. Implementation thoughts ● Committo master (also) after successful application, not (just) before ○ Avoids having to distinguish attempted from successful deployments on master ○ Allows for richer pre-application validation than code diff in a PR (e.g. three-way diff against actual environment) ○ Avoids commit rights to repo being equivalent to deploy rights to env (and your automation will need commit rights to make pull requests, unless you used forked repos) ○ Harder if pull requests are used for the business process (as in vanilla GitOps) - requires multiple branches or “on approve” deployment
- 45. Tuning for yourscenario ● Appropriate level of abstraction? How much “raw” Kubernetes YAML should our developers have access to? ● Where should the abstraction live? In templates? In CRDs? In the automation tool? ● What to use for templating? Token replacement or overrides? ● When do you snapshot/publish to the env repo? On every change to the cluster? Or when a desired end-state is reached? (think multi-step rollout) ● Access control: how many repos do you need? Are code repositories right for your use case, or e.g. a blobstore better? ● Am I distributing or deploying? Helm is much better suited for distribution than deployment ○ Its templating capability is often used as part of deployment flows, though
- 46. Implement ● Choose storageimplementations and partitioning strategies for your environments-as-code ● Define the appropriate level of abstraction for your developers and choose tools to support it ● Choose a flexible automation tool for your deployment business process ● Define an appropriate definition of deployment health/success to determine when a deployment is “good” ● Decide which of the four processes - app update, env config update, env restore and env drift detection - you want to support ● Build pipelines ● Done!
- 47.