SlideShare a Scribd company logo
1 of 53
Download to read offline
Kris Buytaert

@krisbuytaert
Infrastructure as Code Patterns
StackConf 2022, Berlin
• I used to be a developer
• Then I became an Ops person
• Chief Trolling/Travel/Technical Officer @ Inuits.eu
• Chief Yak Shaver @ o11y.eu
• Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ...
• Everything is a Freaking DNS Problem
• Cofounder of all of the above
• DNS : devops needs sushi
• @krisbuytaert on twitter/github
Who am I ?
@krisbuytaert
Devops origins
@krisbuytaert
A global movement to improve the quality of software delivery leveraging Open
Source experiences , started in Gent in 2009
Devops definition
@krisbuytaert
We love running in circles
@krisbuytaert
• 1996 : Manual installation
• 2001 : Mondo Rescue (reproducable single instance)
• 2003 : System Imager , fast reproducable "identical" Infrastructure
• Image Sprawl
• "Overrides"
• 2005 : JeOS + Early IAC (CfEngine / Kickstart / FAI)
• 2010 : Desired State infrastructure as code , Puppet, Chef etc
• 2015 : Cloud provisioning , Terraform etc ..
Deploying an Infrastructure
@krisbuytaert
• 1 (Snowflake) server
• Multiple (snowflake) servers (cluster nodes)
• Virtualization
• Cloud
• Containers
20+ years of changes
@krisbuytaert
• Manual error
• Differences in nodes due to manual changes
• Not reproducable
• Humans
• Scale
Some of The problems
@krisbuytaert
Config Drift
@krisbuytaert
• Manually Crafted Snowflake
• Copied and modified
• Copied and modified
• Impossible to trace origin
• Impossible to reproduce
• Patching / Security nightmare
Image Sprawling & Golden Images vs JeOS
@krisbuytaert
FROM ubuntu

RUN apt-get update

RUN apt-get install -y git
RUN git clone git@github.com:prometheus/prometheus.git
RUN make install
What's wrong with this GW-Basic code ?
@krisbuytaert
Infrastructure as What ?
@krisbuytaert
• Treat configuration automation as Code
• Model your infrastructure
• Development best practices
• Model your infrastructure
• Pipeline your Code
• Test your Code
Infrastructure as Code
@krisbuytaert
• Reproducable Environments
• You never deploy 1x
• Local Test environments
• Development / User Acceptance / Testing / Shadow / Production / DR
• HA <- Keep cluster configuration in sync
• Prevent Configuration drift
• Cheap Disaster recovery
• ...
Infrastructure as Code Goals
@krisbuytaert
3 types of tools needed
• Provisioning
• Desired state
• Orchestration
2022 IaC Requirements
@krisbuytaert
• Create me an instance of application X
• Container instance
• VM instance
• Service X configuration via API

e.g Terraform , Pulumi
Provisioning
@krisbuytaert
• Ensure that this file present / service is running
• With these permissions
• Always / Verified
e.g Chef, Puppet, Salt
Desired state
@krisbuytaert
• Non frequent
• Trigger action X on resource Y based on characteristics A,B and or C
• First do X here then do Y there
• One off actions

e.g Ansible, Salt, Puppet Bolt
Orchestration
@krisbuytaert
• In computing, an idempotent operation is one that has no additional effect if it is called
more than once with the same input parameters.
• Not idempotent :
echo “root=disabled” >> /etc/someconfigfile
Idempodency
@krisbuytaert
Frequent enforcing desired state with reporting =
Automated population of a CMDB (Single source of truth) , with up to date data

Side Effect :
Having a single source from where to generate dynamic configuration

eg. reverse proxy , firewall rules, monitoring configuration, backup configuration
IaC & CMDB
@krisbuytaert
• Do you want a reproducible infrastructure ?
• Do you want to reduce manual error ?
• Do you want to achieve desired state ?
• Do you want to have automated configuration for monitoring, metrics, backups,
security etc ?
• Do you want to trivially spin up different identical environments (dev/uat/prod/dr)
• Do you want to be able to detect and prevent config drift ?
• Do you want to prevent image sprawl ?
Questions to ask :
@krisbuytaert
The Patterns
@krisbuytaert
• Code is not in source control
• Bolt / Ansible / Terraform is triggered from the admin's local machine
• Works for individual admin
• Doesn't Scale
• Prevents Collaboration
Pattern: It works from my homedir
@krisbuytaert
Improvement over $prev, but still doesn't Scale
Collaboration has been improved.
Pattern: It works from my workstation but its in git
@krisbuytaert
• Just the operating System
• Maybe some basic services
• Application is untouched
• Partial source of maybes
• Not suitable for derived configuration
• Kill your Silos
Pattern: We only automate the OS
@krisbuytaert
• Operating System
• Middle Ware
• Application
Pattern: We automate all the things
@krisbuytaert
• Package
• Config
• Service
Pattern: PCS
@krisbuytaert
• PCS
• Non Functional Requirements
• Monitoring
• Backups
• External dependencies
• Metrics
• ...
Pattern: PCS NFR
@krisbuytaert
• Package
• Config Files
• User Generated Data
Pattern: PCS UGC
@krisbuytaert
Pattern: Reproducable and Scalable Infrastructure
@krisbuytaert
• Ops use Chef but Devs use Java templating tool

or
• Terraform but Manual changes from UI.
Variants:
• Non centralized code base,

people run code from local forks undoing changes made by colleagues
• Templating tool is not idempotent (ruby sort ? / XML)
• Bonus points if the tool restarts a service on a change
Pattern: Multiple tools overwrite changes
@krisbuytaert
• Code is checked into a git repository
• The only way to apply is by central user
Pattern: Centralised apply , from source control
@krisbuytaert
• Tool is run once at deploy time then yolo
• manual changes afterwards are not identified or reversed
• State is not enforced, people are afraid to rerun playbooks
• Now we don't dare to update anymore
Orchestraition only mode, no desired state

AKA Yolo or Integrator mode
Pattern: We haven't run it for 6 months
@krisbuytaert
• Reporting says it's broken, we are not going to fix it
• Its been disabled for weeks, local modifications break it
• See Config Drift
Pattern: It's broken anyhow .
@krisbuytaert
• X < 120
• Periodically we run ensure the state of our Infrastructure
• We prevent config Drift
• We actively monitor broken runs
• We actively monitor change frequency
We have achieved desired state of our infra.
Pattern: We run it every X minutes
@krisbuytaert
• We setup everything, it worked for a while
• Then we broke service A ,
• Then something unrelated broke service B
• Automation got blamed
• We are not allowed to enable it again
• Team X doesn't trust automation,
• Often Team X = Security
• We manually make the changes now , but we have visibility on drift.
• We have semi desired state
Pattern: We run it in nooop now
@krisbuytaert
Pattern: We have a pipeline
@krisbuytaert
What is an Environment , a stack, a platform ?
A logically split of set of servers that together offer a service,
Typically existing in dev/uat/prod flavours
Environments
@krisbuytaert
• Limiting Blast radius
• Easy to spin up multiple variants
• Feature flagged code introduction
• And promotion of features
• Clear line of ownership (team bound)
• Same code , different config per environment
• Easier release management
Pattern : Using Environments
@krisbuytaert
• Huge blast radius
• Potential Scaling issues
• Difficult to "promote" changes
• Upgrade everthing or write host based if then case spaghetti
• Code duplication :

role_app_dev, role_app_uat, role_app_prod
Pattern : Not using environments
@krisbuytaert
• Typically with no environments
• Environment specific config is hardcoded
• See Config drift
Pattern: Config is hardcoded in the code
@krisbuytaert
• The same codebase is used
• Only the config data differs
• Multiple Environments possible
Pattern: Config Data defines config per environment
@krisbuytaert
• Code is run
• We will see the errors when they happen
• Code is not tested
Pattern: It we don't test
@krisbuytaert
• Every config run
• Collect information of the run,
• Store that information
• Be able to query a central source with live, correctly, machine generated data
• Reconfigure services based on changes in that data
• Full lifecycle
• Decomission instance => Automatically Clean out the data
Pattern: Single source of truth
@krisbuytaert
SSOT : e.g Puppet Exported Resources
@krisbuytaert
• Our Ansible playbook does a git commit to a config repo
• That repo gets checked out on the monitoring server
• Humans also modify this config
• Information is scattered over different tools
• you can't leverage the real power of automation , no dynamic reconfiguration of
resources
therefore configuration is often done manually , but through a tool
Pattern: No single source of truth
@krisbuytaert
We have one repo with all
• code
• passwords
• software binaries
Pattern: One repo for all the things
@krisbuytaert
"Blue printing"
Building a UI on top of a provisioning framework is typically an intermediate
step before people realize they either need to build in all AAA functionality and
buy a bloated tool.
Or realize git + Terraform solves these problems in a better way.
Pattern: Provisioning from a UI
@krisbuytaert
• We have written all code in house ?
• What do you mean there are modules on github to manage apache ?
• We spent about 3 months building our own
Pattern: we ignore the existing codebase group
@krisbuytaert
• Aka We do NOT understand Continuous Integration
• We have mapped our environments to git branches
• Obviously they are long running
• We are cherry picking changes from uat to prod
• We don't test those combinations
• We are in merge hell now
Puppet r10k is an antipattern.
So is running your gitops from multiple branches.
Pattern: We love Branching
@krisbuytaert
Puppet & Chef
Terraform & Pulumi
Pattern: We want a full language
@krisbuytaert
• All of these people say they are doing infrastructure as Code
• Some with 0 real benefit
• Find the patterns you like / dislike
• Implement or remove the ones you please
• My preferences might not fit your use case
Lessons Learned
@krisbuytaert
Kris Buytaert
@krisbuytaert
kris@inuits.eu
Essensteenweg 31
2930 Brasschaat
Belgium
Contact:
info@inuits.eu
+32-3-8082105

More Related Content

Similar to stackconf 2022: Infrastructure Automation (anti) patterns

Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable productJulian Simpson
 
Database as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesDatabase as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesObjectRocket
 
Evolving for Kubernetes
Evolving for KubernetesEvolving for Kubernetes
Evolving for KubernetesChris McEniry
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremKris Buytaert
 
Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)Chandrapal Badshah
 
Operating Consul as an Early Adopter
Operating Consul as an Early AdopterOperating Consul as an Early Adopter
Operating Consul as an Early AdopterNelson Elhage
 
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAutomation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAppDynamics
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGroup
 
DevOps: Automate all the things
DevOps: Automate all the thingsDevOps: Automate all the things
DevOps: Automate all the thingsMat Mannion
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesAmazon Web Services
 
OSMC 2021 | Pipeline your Dashboards as Code
OSMC 2021 | Pipeline your Dashboards as CodeOSMC 2021 | Pipeline your Dashboards as Code
OSMC 2021 | Pipeline your Dashboards as CodeNETWAYS
 
Pipeline all the Dashboards as Code
Pipeline all the Dashboards as CodePipeline all the Dashboards as Code
Pipeline all the Dashboards as CodeKris Buytaert
 
Git Going w/ Git
Git Going w/ GitGit Going w/ Git
Git Going w/ GitheyMP
 
Reactive Development: Commands, Actors and Events. Oh My!!
Reactive Development: Commands, Actors and Events.  Oh My!!Reactive Development: Commands, Actors and Events.  Oh My!!
Reactive Development: Commands, Actors and Events. Oh My!!David Hoerster
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comAviran Mordo
 
Immutable infrastructure isn’t the answer
Immutable infrastructure isn’t the answerImmutable infrastructure isn’t the answer
Immutable infrastructure isn’t the answerSam Bashton
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOpsEklove Mohan
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure FirstKris Buytaert
 

Similar to stackconf 2022: Infrastructure Automation (anti) patterns (20)

Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable product
 
Database as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on KubernetesDatabase as a Service (DBaaS) on Kubernetes
Database as a Service (DBaaS) on Kubernetes
 
Evolving for Kubernetes
Evolving for KubernetesEvolving for Kubernetes
Evolving for Kubernetes
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPrem
 
Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)Detecting secrets in code committed to gitlab (in real time)
Detecting secrets in code committed to gitlab (in real time)
 
Operating Consul as an Early Adopter
Operating Consul as an Early AdopterOperating Consul as an Early Adopter
Operating Consul as an Early Adopter
 
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAutomation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
 
Hosting Ruby Web Apps
Hosting Ruby Web AppsHosting Ruby Web Apps
Hosting Ruby Web Apps
 
DevOps: Automate all the things
DevOps: Automate all the thingsDevOps: Automate all the things
DevOps: Automate all the things
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
 
OSMC 2021 | Pipeline your Dashboards as Code
OSMC 2021 | Pipeline your Dashboards as CodeOSMC 2021 | Pipeline your Dashboards as Code
OSMC 2021 | Pipeline your Dashboards as Code
 
Pipeline all the Dashboards as Code
Pipeline all the Dashboards as CodePipeline all the Dashboards as Code
Pipeline all the Dashboards as Code
 
Git Going w/ Git
Git Going w/ GitGit Going w/ Git
Git Going w/ Git
 
Reactive Development: Commands, Actors and Events. Oh My!!
Reactive Development: Commands, Actors and Events.  Oh My!!Reactive Development: Commands, Actors and Events.  Oh My!!
Reactive Development: Commands, Actors and Events. Oh My!!
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
Immutable infrastructure isn’t the answer
Immutable infrastructure isn’t the answerImmutable infrastructure isn’t the answer
Immutable infrastructure isn’t the answer
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
 
Redundant devops
Redundant devopsRedundant devops
Redundant devops
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
 

Recently uploaded

Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfChristopherTHyatt
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2DianaGray10
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomCzechDreamin
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyUXDXConf
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationZilliz
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfFIDO Alliance
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureFemke de Vroome
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...CzechDreamin
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIES VE
 

Recently uploaded (20)

Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 

stackconf 2022: Infrastructure Automation (anti) patterns

  • 1. Kris Buytaert @krisbuytaert Infrastructure as Code Patterns StackConf 2022, Berlin
  • 2. • I used to be a developer • Then I became an Ops person • Chief Trolling/Travel/Technical Officer @ Inuits.eu • Chief Yak Shaver @ o11y.eu • Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ... • Everything is a Freaking DNS Problem • Cofounder of all of the above • DNS : devops needs sushi • @krisbuytaert on twitter/github Who am I ? @krisbuytaert
  • 4. A global movement to improve the quality of software delivery leveraging Open Source experiences , started in Gent in 2009 Devops definition @krisbuytaert
  • 5. We love running in circles @krisbuytaert
  • 6. • 1996 : Manual installation • 2001 : Mondo Rescue (reproducable single instance) • 2003 : System Imager , fast reproducable "identical" Infrastructure • Image Sprawl • "Overrides" • 2005 : JeOS + Early IAC (CfEngine / Kickstart / FAI) • 2010 : Desired State infrastructure as code , Puppet, Chef etc • 2015 : Cloud provisioning , Terraform etc .. Deploying an Infrastructure @krisbuytaert
  • 7. • 1 (Snowflake) server • Multiple (snowflake) servers (cluster nodes) • Virtualization • Cloud • Containers 20+ years of changes @krisbuytaert
  • 8. • Manual error • Differences in nodes due to manual changes • Not reproducable • Humans • Scale Some of The problems @krisbuytaert
  • 10. • Manually Crafted Snowflake • Copied and modified • Copied and modified • Impossible to trace origin • Impossible to reproduce • Patching / Security nightmare Image Sprawling & Golden Images vs JeOS @krisbuytaert
  • 11. FROM ubuntu RUN apt-get update RUN apt-get install -y git RUN git clone git@github.com:prometheus/prometheus.git RUN make install What's wrong with this GW-Basic code ? @krisbuytaert
  • 12. Infrastructure as What ? @krisbuytaert
  • 13. • Treat configuration automation as Code • Model your infrastructure • Development best practices • Model your infrastructure • Pipeline your Code • Test your Code Infrastructure as Code @krisbuytaert
  • 14. • Reproducable Environments • You never deploy 1x • Local Test environments • Development / User Acceptance / Testing / Shadow / Production / DR • HA <- Keep cluster configuration in sync • Prevent Configuration drift • Cheap Disaster recovery • ... Infrastructure as Code Goals @krisbuytaert
  • 15. 3 types of tools needed • Provisioning • Desired state • Orchestration 2022 IaC Requirements @krisbuytaert
  • 16. • Create me an instance of application X • Container instance • VM instance • Service X configuration via API e.g Terraform , Pulumi Provisioning @krisbuytaert
  • 17. • Ensure that this file present / service is running • With these permissions • Always / Verified e.g Chef, Puppet, Salt Desired state @krisbuytaert
  • 18. • Non frequent • Trigger action X on resource Y based on characteristics A,B and or C • First do X here then do Y there • One off actions e.g Ansible, Salt, Puppet Bolt Orchestration @krisbuytaert
  • 19. • In computing, an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. • Not idempotent : echo “root=disabled” >> /etc/someconfigfile Idempodency @krisbuytaert
  • 20. Frequent enforcing desired state with reporting = Automated population of a CMDB (Single source of truth) , with up to date data Side Effect : Having a single source from where to generate dynamic configuration eg. reverse proxy , firewall rules, monitoring configuration, backup configuration IaC & CMDB @krisbuytaert
  • 21. • Do you want a reproducible infrastructure ? • Do you want to reduce manual error ? • Do you want to achieve desired state ? • Do you want to have automated configuration for monitoring, metrics, backups, security etc ? • Do you want to trivially spin up different identical environments (dev/uat/prod/dr) • Do you want to be able to detect and prevent config drift ? • Do you want to prevent image sprawl ? Questions to ask : @krisbuytaert
  • 23. • Code is not in source control • Bolt / Ansible / Terraform is triggered from the admin's local machine • Works for individual admin • Doesn't Scale • Prevents Collaboration Pattern: It works from my homedir @krisbuytaert
  • 24. Improvement over $prev, but still doesn't Scale Collaboration has been improved. Pattern: It works from my workstation but its in git @krisbuytaert
  • 25. • Just the operating System • Maybe some basic services • Application is untouched • Partial source of maybes • Not suitable for derived configuration • Kill your Silos Pattern: We only automate the OS @krisbuytaert
  • 26. • Operating System • Middle Ware • Application Pattern: We automate all the things @krisbuytaert
  • 27. • Package • Config • Service Pattern: PCS @krisbuytaert
  • 28. • PCS • Non Functional Requirements • Monitoring • Backups • External dependencies • Metrics • ... Pattern: PCS NFR @krisbuytaert
  • 29. • Package • Config Files • User Generated Data Pattern: PCS UGC @krisbuytaert
  • 30. Pattern: Reproducable and Scalable Infrastructure @krisbuytaert
  • 31. • Ops use Chef but Devs use Java templating tool or • Terraform but Manual changes from UI. Variants: • Non centralized code base, people run code from local forks undoing changes made by colleagues • Templating tool is not idempotent (ruby sort ? / XML) • Bonus points if the tool restarts a service on a change Pattern: Multiple tools overwrite changes @krisbuytaert
  • 32. • Code is checked into a git repository • The only way to apply is by central user Pattern: Centralised apply , from source control @krisbuytaert
  • 33. • Tool is run once at deploy time then yolo • manual changes afterwards are not identified or reversed • State is not enforced, people are afraid to rerun playbooks • Now we don't dare to update anymore Orchestraition only mode, no desired state AKA Yolo or Integrator mode Pattern: We haven't run it for 6 months @krisbuytaert
  • 34. • Reporting says it's broken, we are not going to fix it • Its been disabled for weeks, local modifications break it • See Config Drift Pattern: It's broken anyhow . @krisbuytaert
  • 35. • X < 120 • Periodically we run ensure the state of our Infrastructure • We prevent config Drift • We actively monitor broken runs • We actively monitor change frequency We have achieved desired state of our infra. Pattern: We run it every X minutes @krisbuytaert
  • 36. • We setup everything, it worked for a while • Then we broke service A , • Then something unrelated broke service B • Automation got blamed • We are not allowed to enable it again • Team X doesn't trust automation, • Often Team X = Security • We manually make the changes now , but we have visibility on drift. • We have semi desired state Pattern: We run it in nooop now @krisbuytaert
  • 37. Pattern: We have a pipeline @krisbuytaert
  • 38. What is an Environment , a stack, a platform ? A logically split of set of servers that together offer a service, Typically existing in dev/uat/prod flavours Environments @krisbuytaert
  • 39. • Limiting Blast radius • Easy to spin up multiple variants • Feature flagged code introduction • And promotion of features • Clear line of ownership (team bound) • Same code , different config per environment • Easier release management Pattern : Using Environments @krisbuytaert
  • 40. • Huge blast radius • Potential Scaling issues • Difficult to "promote" changes • Upgrade everthing or write host based if then case spaghetti • Code duplication : role_app_dev, role_app_uat, role_app_prod Pattern : Not using environments @krisbuytaert
  • 41. • Typically with no environments • Environment specific config is hardcoded • See Config drift Pattern: Config is hardcoded in the code @krisbuytaert
  • 42. • The same codebase is used • Only the config data differs • Multiple Environments possible Pattern: Config Data defines config per environment @krisbuytaert
  • 43. • Code is run • We will see the errors when they happen • Code is not tested Pattern: It we don't test @krisbuytaert
  • 44. • Every config run • Collect information of the run, • Store that information • Be able to query a central source with live, correctly, machine generated data • Reconfigure services based on changes in that data • Full lifecycle • Decomission instance => Automatically Clean out the data Pattern: Single source of truth @krisbuytaert
  • 45. SSOT : e.g Puppet Exported Resources @krisbuytaert
  • 46. • Our Ansible playbook does a git commit to a config repo • That repo gets checked out on the monitoring server • Humans also modify this config • Information is scattered over different tools • you can't leverage the real power of automation , no dynamic reconfiguration of resources therefore configuration is often done manually , but through a tool Pattern: No single source of truth @krisbuytaert
  • 47. We have one repo with all • code • passwords • software binaries Pattern: One repo for all the things @krisbuytaert
  • 48. "Blue printing" Building a UI on top of a provisioning framework is typically an intermediate step before people realize they either need to build in all AAA functionality and buy a bloated tool. Or realize git + Terraform solves these problems in a better way. Pattern: Provisioning from a UI @krisbuytaert
  • 49. • We have written all code in house ? • What do you mean there are modules on github to manage apache ? • We spent about 3 months building our own Pattern: we ignore the existing codebase group @krisbuytaert
  • 50. • Aka We do NOT understand Continuous Integration • We have mapped our environments to git branches • Obviously they are long running • We are cherry picking changes from uat to prod • We don't test those combinations • We are in merge hell now Puppet r10k is an antipattern. So is running your gitops from multiple branches. Pattern: We love Branching @krisbuytaert
  • 51. Puppet & Chef Terraform & Pulumi Pattern: We want a full language @krisbuytaert
  • 52. • All of these people say they are doing infrastructure as Code • Some with 0 real benefit • Find the patterns you like / dislike • Implement or remove the ones you please • My preferences might not fit your use case Lessons Learned @krisbuytaert
  • 53. Kris Buytaert @krisbuytaert kris@inuits.eu Essensteenweg 31 2930 Brasschaat Belgium Contact: info@inuits.eu +32-3-8082105