In your ever-changing Infrastructure, some changes are intentional while others are not. Drift is what happens whenever the real-world state of your infrastructure differs from the state defined in your configuration. This can happen for many reasons, sometimes it happens when adding or removing resources, other times when changing resource definitions upon resource termination or failure, and even when changes have been made manually or via other automation tools. While Terraform itself can detect drifts, in most cases, you will be informed about it too late: just before you are about to deploy new changes to your infrastructure. What’s interesting about Terraform though, is that you can apply changes in two separate and distinct steps of “Planning” and “Applying”. This means that you have full visibility of what Terraform is planning on doing beforehand, and if you are satisfied with the changes, you can choose to apply them. So how does this work? When something is changed intentionally, it will appear in the source code, and the Terraform plan will not do anything. However, if any part of the infrastructure has been changed manually, Terraform’s plan will identify this, and alert you to the change. In other words, if your IaC drifted from its expected state, then Terraform’s plan will, in fact, detect it. Applying this simple solution can empower DevOps and developer velocity, with the reassurance and context for unexpected changes in your IaC, in near real-time. This talk will showcase real-world examples, and practical ways to apply this in your production environments while doing so safely and at the pace of your engineering cycles.

1. Eran Bibi, Co-Founder and CPO, Firefly
Don’t Panic: Get Your K8s
Drifts Under Control

2. E6i
Co-Founder & CPO, Firefly
Former Head of DevOps & SaaS Platform, Aqua
A CLI Type of Guy

3. • Code reviews
• Immutable infrastructure
• Disaster recovery planning
• Shift left security scanning for the infrastructure
• Pre-deployment cost analysis
Why IaC?
kubeadm
Ansible
Terraform
Cloud Console
eksctl
kops
The Hard Way
Infrastructure-as-
code
The Challenge: Infrastructures Tend to

4. 9 out of 10
Large Scale Deployments
Have Configuration Drifts
50% of the Drifts
Were Unnoticed
Toil 60%
Security 20%
Reliability 20%
100% Impact

5. Infrastructure Drift
Drift is what happens whenever the real-world state of your
infrastructure differs from the defined one in your configuration.
The Configuration The Real-world State

6. And Now for Real
The Configuration
The Real-world State

7. K8s drift
The Configuration
The Real-world State

8. Data / Outcome
Drifts: Cause & Effect
Manual changes
(“emergency
fix”)
Environment
discrepancies
Automated apps
Infrastructure Drift:
• Security vulnerabilities
• Increased costs
• Excessive work
• Decreased efficiency
• Reliability risk
Misalignment between
IaC and Cloud APIs

9. K8s drifts: Double Trouble

10. 1st Approach:
Establish a Drift Detection
Strategy
Understand when drifts may become risky
Catch the drifts: implement drift detection automation per your stack/s
Classify and route each response to respective individual or team

11. Drift Detection
using
Terraform plan

12. 2nd Approach: GitOps

13. Takeaways
● Implement tools to detect drifts
○ Terraform Plan
○ driftctl
○ kubediff
● Consider GitOps methodology as a mitigation strategy
○ ArgoCD
○ Flux
○ JenkinsX
● Tools are not enough, you also need to change behavior

14. May your Infrastructure Never Drift
gofirefly.io
Thank You!