1. GitOps
Modern best practices for high velocity
app dev using cloud native tools
Alexis Richardson & William Denniss
CEO, Weaveworks PM, Google Kubernetes Engine
TOC Chair, CNCF K8s Conformance WG Lead
@monadic @williamdenniss
December 2017

2. Meet Qordoba
● SF based team use machine learning to
create ”local” marketing UX for big brands
● Rapid iteration while obeying SOC2
compliance
● Google Cloud – Kubernetes Engine &
Container Builder
● Weave Cloud – single cont. delivery &
observability pipeline

4. Over 30 releases per day, up from 1-2 per week
1) Estimated time needed to fix prod software bugs ~60% less time
2) Estimated time to respond to customer requests ~43% less time
3) Uptime 99% → 100%
Impact

5. GitOps builds on DevOps with Git as a single source of truth for the whole system.
Over the few years at Weaveworks we learned that success came down to getting 3
things right:
1. Have a complete automated pipeline
2. Operating a fast paced business 24/7 requires monitoring and observability
baked into the beginning. Security is of critical importance
3. Everything has to be version controlled and stored in a single source of truth from
which you can recover
Introducing GitOps

6. Image credit:
Helen Beal,
Ranger4
Build on a decade of DevOps best practices

7. Add declarative infrastructure

8. • We use declarative infrastructure ie.
Kubernetes, Docker, Terraform, & more
• Our entire system including code, config,
monitoring rules, dashboards, is
described in GitHub with full audit trail
• We can roll our major or minor changes
as pull requests, and automatically
check for diffs if system diverges from
the desired “source of truth” in Git
How did Weaveworks rebuild our systems in 45 mins?

9. • Config is code
• Code must be version controlled
• Config must be version controlled too
GitOps follows the Logic of DevOps

10. GitOps follows the Logic of DevOps
• Config is code
• Code must be version controlled
• Config must be version controlled too
• What can be described can be automated
• Describe everything: code, config,
monitoring & policy; and then keep it in
version control

11. GitOps
• Git as a source of truth for desired state of whole system
• Compare desired with actual state to fire diff alerts
• Make ops changes by pull request

12. What this gets us
• Any developer can use GitHub
• Anyone can join team and ship a new
app or make changes easily
• All changes can be triggered, stored,
audited and validated in Git
And we didn’t have to do anything very
new or clever ☺

13. Three pillars of GitOps
Pipelines
Observability
Control

14. Three pillars of GitOps
Pipelines
Observability
Control
Joined Up Pipelines
Continuous Integration
Continuous Deployment
Release Automation
>>> Git as source of truth for desired
system state

15. First you must join up these components into a pipeline
Deployment
App Dev Build (CI) Containers
Execution
(CD + Release
Automation)

16. Service deployments are controlled using operator pattern
Git
Config
Kubernetes Cluster
Deployment
Service
Deploy
Operator

17. Service deployments are controlled using operator pattern
○ Config is code & everything is config (‘declarative infra’)
○ Code (& config!) must be version controlled
○ Anything that does not record changes in version
control is harmful

18. Extracting Declarative Config From Running Clusters
• What if you’re already running a cluster setup and don’t have the
resource configuration available?
• `kubectl get <object> -o yaml --export` can be used to
extract the existing configuration and bootstrap your configuration
repository
• Be sure to manually review the output, “--export” intentionally drops
information that it sees as transient, but it’s not perfect.

19. ❯ GitOps operational mindset, all
k8s applications stored in Git.
❯ Securely automate & share
secrets publicly
❯ Asymmetric (public key)
cryptography
❯ Encrypt data up to (and inside)
K8s cluster
Bitnami: Encrypt Kubernetes SecretsSealed
Secrets

20. Structure of a GitOps repository
● 1 repository per application/service
● Use a separate branch per environment (maps to a Kubernetes
namespace, or cluster)
● Push changes such as the image name, health checks, etc to
staging (or feature) branches first.
● Rolling out to production involves a merge. (use `git merge -s
ours branchname` to skip a set of staging-only changes).
● Use protected branches to enforce code review requirements.

21. Example GitOps Pipeline
Git
Code
Git
Config
Container
Registry
Build
Container
(CI)
Update image in staging config
1/ Code change
2/ Merge
Staging to
Prod
Config Updater
Kubernetes Cluster
Deployment
Service
Deploy
Operator

22. Three pillars of GitOps
Pipelines
Observability
Control
Observability
Monitoring
Logging
Tracing & Visualization
>>> Holistic view of the real system state
‘right now’

23. Implication: each service PR has to be observed & measured

24. Service level view, from UX to health to deep diagnostics
• End user happiness is all
• Integrate GitOps CD pipeline with
tools to observe results of PRs
• Developers have to correlate UX
to operational concepts like
monitoring, tracing, logs
• Like doctors, we must be able to
validate health as well as
diagnose problems

25. In GitOps, every service should have a unified interactive dash
(eg. at Lyft, metrics + events + actions)

26. Three pillars of GitOps
Pipelines
Observability
Control
Control
1) Everything via Git
(updates, policy, security)
2) Orchestration
3) Diff & Sync
>>> Automation is Convergence

27. Every service has two sources of truth
Desired State
In Git
Observed State
In Real System
Same?

28. Control means CONVERGENCE
● Orchestration: Kubernetes & CD operators (Weave Flux)
● When orchestration fails use Diff & Sync tools
● When sync fails make changes via Git not kubectl

29. Examples

30. Kubernetes
Monitoring
Tracing
Logging…
Control &
Operations
Git Diff
GitOps Service Lifecycle

31. Kubernetes
Monitoring
Tracing
Logging…
Control &
Operations
Git Diff
GitOps Service Lifecycle
”release driven”
Observe
Orient
Decide
Act
Release

32. ONLY what can
be described
and observed
can be
automated and
controlled and
accelerated
Fundamental Theorem of GitOps

33. Use declarative configuration to define your application and
services.
All changes need to go through your git review process – noone
should be using kubectl directly
Use an operator in the cluster to drive the observed cluster state
to the desired state, as declared by your configuration in git.
The 3 core principles of GitOps

34. GitOps is for Developers
● If you can merge a pull
request on GitHub then you
are an app developer
● If you use Cloud Native
tools you can describe the
whole stack & app in Git
● GitOps is the logical
conclusion for operations
Image: GitHub

35. What’s new vs DevOps and “infra as code”?
How is this different from PaaS?
Does GitOps work with …
● ChatOps?
● Secrets?
● Helm (non-compositional templates)
● Istio (How to do Canary rollouts and “partial” updates?)
What else can be declarative? Policy & Security & Dashboards & …
Any Questions?