SlideShare a Scribd company logo
Immutable Infrastructure as Code.
True story.
Vladlen Fedosov, Director of R&D @Namecheap, Inc
Vladlen Fedosov
Director of R&D @Namecheap
TL;DR:

• 10 years in the industry

• Went path from Junior to Architect 

• Amateur DevOps evangelist 

• AWS ninja

• Believe in self-organized, cross-functional teams
“Opening the door for
everyone to a free and
open Internet”
The beginning
Disclaimer: Here I’m talking mostly about my experience and part of the infrastructure that my team
was responsible for. So further statements may not apply to every department in the company. Timeline for the things
mentioned below was changes to simplify storytelling.
DevOps Fest 2020. immutable infrastructure as code. True story.
Project takeover from outsourcing company
Brave New World
For a small dev team with no infra support / capabilities
So what do we have now?
State of the project after takeover
● Half broken Chef cookbooks

● Sketchy CD pipelines

● Fault tolerance in place

● And… Everything went down after failure of the single (out of 3)
etcd node. We realized that we have fault tolerance only on paper
After some refactoring… We’ve got typical setup
Issues we noticed
- Multiple apps were sharing same OS, language versions, dependencies
- Horizontal scaling was hard

- Sometimes failing chef scripts on random instances mostly due to network
errors & configuration differences
Configuration Synchronisation problem, in short
martinfowler.com/bliki/ConfigurationSynchronization.html
Issues we noticed
- Manual sync of the AWS setup between environments

- Manually configured CI/CD

- Easy to break something and hard to repair or modify anything
Blue sky vision
● Immutable infrastructure
● Everything as code
○ Infrastructure as code

○ CI/CD as code
Blue sky vision
● Hard to break
● Easy to repair
● Easy to modify
Immutable infrastructure
+ extra takeaways
What is immutable infrastructure?
martinfowler.com/bliki/ImmutableServer.html
Why go Immutable?
● Forget about change flow, only “create” matters

● Defeat Configuration Drift

● Use much simpler tools

● Build highly available systems easier

● Fix issues faster
How to achieve this?
● Complexity → Docker (or AMI) images

● OS → Docker runtime only

● App/OS configuration methods:

○ K8s pod definitions (or similar)

○ “cloud-init”

● Terraform to define “datacenter config”
Main tools we use
● AMI images & Docker images

● Cloud-Init (98%)

● Hashicorp Packer (2%)

● Terraform, Terraform everywhere
Single fact you could memorise here
Immutable Infrastructure allows you to significantly simplify
management steps and consequently reduce number of
bugs your customers will face with. Work with images rather
than servers.
CI/CD as code
DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.
Everything as code: full list
● Everything as code, it supports:
○ job steps definition as code (via pipelines)

○ jobs creation as code (via job dsl)

○ system configuration as code (via groovy API, XML configs & CasC yml)

● Shared libraries, ability to share common steps between apps
P.S: Talk to me if know better alternative ;)
Other factors influenced the choice
● Has deployment dashboard so we can see the state of all the
environments
● Highly extensible
● Elastic EC2 instances as agents
Nowadays
● It went beyond one project and now almost every team at Namecheap uses it

● 300+ pipelines

● Around 38 projects

● We expect even more in the future
Nowadays: CI
@Library('namecheap/common') _
node('CommonCLarge') {
ciJavascript.servicePipeline {
productName = "ProductA"
serviceName = "Apps.Api"
}
}
Nowadays: CD
@Library('namecheap/common') _
properties([parameters([
string(name: 'image', description: 'Application container name'),
string(name: 'version', description: 'Application container tag'),
choice(choices: ['production', 'sandbox'], description: 'Environment', name: 'env')
])])
node() {
def authToken = "XXXX"
stage('Deploy') {
deployToOKD(params, authToken)
}
}
Lessons learned
● Hide all complexity inside. Provide as much logic as you can in a form of
Shared Libraries. 

● Documentation & examples are crucial for developers

● If possible - provide standardized pipelines invoked as Shared Library
function

● It takes about 1.5 months for 2 people to setup Jenkins properly for the first
time
Single fact you could memorise here
Always keep your CI/CD configuration, written as code,
near the app, in the same repo. It will give an understanding
to everyone in your company on how to build & deploy any
app.
Infrastructure as code: Terraform
How Terraform is different to Ansible/Chef/Puppet
Terraform is not a configuration management tool. It focuses on the higher-level
abstraction of the datacenter (or cloud provider), without sacrificing the ability to
use configuration management tools to do what they do best: bootstrapping
and initializing resources.
resource "aws_instance" "web" {
ami = "ami-dbc3b9aa"
instance_type = "t2.micro"
}
Imperative VS Declarative infrastructure code
Declarative
“Can I have a cup of coffee on my
desk at 9AM on Monday morning?”
Imperative
“Go to that machine, then get the
glass jar, then fill it with water, then
put it back in the machine” 

…you get the idea...
Why go Declarative?
Key challenges this approach solves for us:

● Dealing with “Configuration Drift” / State management

● Idempotency

● Dependency graph management, correct order of operations
Terraform CI
Try now:

https://www.runatlantis.io/ 

https://app.terraform.io/
Deploy with Terraform
What we had:
1. Write TF configs

2. Run TF to create infrastructure

3. Take TF outputs & enter them to
Jenkins

4. Deploy app itself with Jenkins
What we wanted to have:
1. Write TF configs

2. Deploy app
Deploy with Terraform
Deploy with Terraform
!"" vars <-- Environment specific variables
# !"" production.tfvars
# !"" staging.tfvars
!"" main.tf
!"" io.tf
!"" db.tf
!"" etc.tf
Single fact you could memorize here
Try to have as much declarative infrastructure configs as
you can, avoid imperative scripts at all cost.
Learnings & further improvements
Tests for infrastructure code
The more infrastructure code you have - the more bugs you see.
Chaos monkey
We wrote a Lambda that randomly reboots every instance once a day. This
simple tweak ensures that:

● Apps you launch can survive instance failure

● Updates to the cluster setup is easy as you’re sure that you won’t harm
anyone by killing outdated machines

● Problem resolution can be simpler sometimes. You can always reboot/kill any
instance that behaves abnormally as a first action

Apply this to control fleet too
No SSH keys distribution over instances
● If you’re using AWS - simply install SSM agent to your instances and disable
SSH daemon. You will be able to use SSH console to perform your
administrative actions.

● If you’re not in AWS - you can use Hashicorp Vault. It provides you with
SSH backend that allows central management & audit of the login identities.
Things that work for 3 teams - doesn’t work for 10
Issues
# of users
Key learnings here:
● Operational work grows exponentially the more teams you add 

● 1 new tool/approach for devs at a time

● Conduct educational courses for big new things like Docker, AWS, Terraform

● Gain trust within the team you’re challenging with a change first

● Documentation is paramount, start it as early as possible
Things that work for 3 teams - doesn’t work for 10
DevOps on Call & transparent SLAs
● We’ve established “on call” schedule 

● Agreed on SLAs & shared then among the teams

● Created chat room & Jira board

Result: significant reduction of the distraction level, better productivity, happier
teams
Even you’re doing good now, you can make it even better with this practice
Encourage feedback
● Ask for it proactively, show that it's important for you

● Public retrospectives

● Respond to the feedback
Summing up
What we’ve achieved
● Immutable infrastructure (done)
○ ECS with immutable data plane

○ Immutable EC2 instances for stateful instances

● Everything as code (done)
○ Infrastructure as code: Terraform, Cloud-init

○ CI/CD as code: Jenkins

Now it’s hard to break & easy to repair things as well as

easy to track changes.
Vlad Fedosov
Director of R&D
@Namecheap, Inc
vlad.fedosov@gmail.com
Or just scan it:
Evolving Terraform experience

More Related Content

What's hot

Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
Yaniv cohen
 
Infrastructure as Code for Network
Infrastructure as Code for NetworkInfrastructure as Code for Network
Infrastructure as Code for Network
Damien Garros
 
Puppet overview
Puppet overviewPuppet overview
Puppet overview
joshbeard
 
Continuous integration of_puppet_code
Continuous integration of_puppet_codeContinuous integration of_puppet_code
Continuous integration of_puppet_code
Devoteam Revolve
 
DevOps Summit 2016 - The immutable Journey
DevOps Summit 2016 - The immutable JourneyDevOps Summit 2016 - The immutable Journey
DevOps Summit 2016 - The immutable Journey
smalltown
 
PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...
PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...
PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...
Puppet
 
Devops For Drupal
Devops  For DrupalDevops  For Drupal
Devops For Drupal
Kris Buytaert
 
Is Python still production ready ? Ludovic Gasc
Is Python still production ready ? Ludovic GascIs Python still production ready ? Ludovic Gasc
Is Python still production ready ? Ludovic Gasc
Pôle Systematic Paris-Region
 
DevTernity - DevOps with smell
DevTernity - DevOps with smellDevTernity - DevOps with smell
DevTernity - DevOps with smell
Antons Kranga
 
Rule jenkins with configuration as code
Rule jenkins with configuration as codeRule jenkins with configuration as code
Rule jenkins with configuration as code
Christian Rasp
 
Immutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App DeploymentImmutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App Deployment
Axel Fontaine
 
CommandBox & ForgeBox Package Management
CommandBox & ForgeBox Package ManagementCommandBox & ForgeBox Package Management
CommandBox & ForgeBox Package Management
Ortus Solutions, Corp
 
Continuous Deployment with Cloud Foundry, Github and Travis CI
Continuous Deployment with Cloud Foundry, Github and Travis CIContinuous Deployment with Cloud Foundry, Github and Travis CI
Continuous Deployment with Cloud Foundry, Github and Travis CI
Platform CF
 
From VB Script to PowerShell
From VB Script to PowerShellFrom VB Script to PowerShell
From VB Script to PowerShell
Concentrated Technology
 
Drupal Deployment
Drupal DeploymentDrupal Deployment
Drupal Deployment
Jeff Eaton
 
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like ServersPuppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet
 
PS scripting and modularization
PS scripting and modularizationPS scripting and modularization
PS scripting and modularization
Concentrated Technology
 
ContainerCon - Test Driven Infrastructure
ContainerCon - Test Driven InfrastructureContainerCon - Test Driven Infrastructure
ContainerCon - Test Driven Infrastructure
Yury Tsarev
 
John Adams Puppet Camp 2010
John Adams Puppet Camp 2010John Adams Puppet Camp 2010
John Adams Puppet Camp 2010
Puppet
 
Implementing blue-green deployment with Atlassian Bamboo
Implementing blue-green deployment with Atlassian BambooImplementing blue-green deployment with Atlassian Bamboo
Implementing blue-green deployment with Atlassian Bamboo
Dave Clark
 

What's hot (20)

Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
 
Infrastructure as Code for Network
Infrastructure as Code for NetworkInfrastructure as Code for Network
Infrastructure as Code for Network
 
Puppet overview
Puppet overviewPuppet overview
Puppet overview
 
Continuous integration of_puppet_code
Continuous integration of_puppet_codeContinuous integration of_puppet_code
Continuous integration of_puppet_code
 
DevOps Summit 2016 - The immutable Journey
DevOps Summit 2016 - The immutable JourneyDevOps Summit 2016 - The immutable Journey
DevOps Summit 2016 - The immutable Journey
 
PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...
PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...
PuppetConf 2016: Keynote: Pulling the Strings to Containerize Your Life - Sco...
 
Devops For Drupal
Devops  For DrupalDevops  For Drupal
Devops For Drupal
 
Is Python still production ready ? Ludovic Gasc
Is Python still production ready ? Ludovic GascIs Python still production ready ? Ludovic Gasc
Is Python still production ready ? Ludovic Gasc
 
DevTernity - DevOps with smell
DevTernity - DevOps with smellDevTernity - DevOps with smell
DevTernity - DevOps with smell
 
Rule jenkins with configuration as code
Rule jenkins with configuration as codeRule jenkins with configuration as code
Rule jenkins with configuration as code
 
Immutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App DeploymentImmutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App Deployment
 
CommandBox & ForgeBox Package Management
CommandBox & ForgeBox Package ManagementCommandBox & ForgeBox Package Management
CommandBox & ForgeBox Package Management
 
Continuous Deployment with Cloud Foundry, Github and Travis CI
Continuous Deployment with Cloud Foundry, Github and Travis CIContinuous Deployment with Cloud Foundry, Github and Travis CI
Continuous Deployment with Cloud Foundry, Github and Travis CI
 
From VB Script to PowerShell
From VB Script to PowerShellFrom VB Script to PowerShell
From VB Script to PowerShell
 
Drupal Deployment
Drupal DeploymentDrupal Deployment
Drupal Deployment
 
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like ServersPuppet Camp Charlotte 2015: Manage Your Switches Like Servers
Puppet Camp Charlotte 2015: Manage Your Switches Like Servers
 
PS scripting and modularization
PS scripting and modularizationPS scripting and modularization
PS scripting and modularization
 
ContainerCon - Test Driven Infrastructure
ContainerCon - Test Driven InfrastructureContainerCon - Test Driven Infrastructure
ContainerCon - Test Driven Infrastructure
 
John Adams Puppet Camp 2010
John Adams Puppet Camp 2010John Adams Puppet Camp 2010
John Adams Puppet Camp 2010
 
Implementing blue-green deployment with Atlassian Bamboo
Implementing blue-green deployment with Atlassian BambooImplementing blue-green deployment with Atlassian Bamboo
Implementing blue-green deployment with Atlassian Bamboo
 

Similar to DevOps Fest 2020. immutable infrastructure as code. True story.

Successful DevOps implementation for small teams a true story
Successful DevOps implementation for small teams  a true storySuccessful DevOps implementation for small teams  a true story
Successful DevOps implementation for small teams a true story
Jakub Paweł Głazik
 
Docker in Production at the Aurora Team
Docker in Production at the Aurora TeamDocker in Production at the Aurora Team
Docker in Production at the Aurora Team
Haufe-Lexware GmbH & Co KG
 
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
Docker, Inc.
 
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Richard Bullington-McGuire
 
Meetup 2020 - Back to the Basics part 101 : IaC
Meetup 2020 - Back to the Basics part 101 : IaCMeetup 2020 - Back to the Basics part 101 : IaC
Meetup 2020 - Back to the Basics part 101 : IaC
DamienCarpy
 
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...
Haggai Philip Zagury
 
Old Is the New New
Old Is the New NewOld Is the New New
Old Is the New New
Kevlin Henney
 
Product! - The road to production deployment
Product! - The road to production deploymentProduct! - The road to production deployment
Product! - The road to production deployment
Filippo Zanella
 
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Lean IT Consulting
 
The "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/OpsThe "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/Ops
Erik Osterman
 
JUST EAT: Embracing DevOps
JUST EAT: Embracing DevOpsJUST EAT: Embracing DevOps
JUST EAT: Embracing DevOps
Peter Mounce
 
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Demi Ben-Ari
 
Scalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUs
Scalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUsScalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUs
Scalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUs
Indrajit Poddar
 
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeConfoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Steve Mercier
 
How Percolate uses CFEngine to Manage AWS Stateless Infrastructure
How Percolate uses CFEngine to Manage AWS Stateless InfrastructureHow Percolate uses CFEngine to Manage AWS Stateless Infrastructure
How Percolate uses CFEngine to Manage AWS Stateless Infrastructure
Percolate
 
Drupal 8 DevOps . Profile and SQL flows.
Drupal 8 DevOps . Profile and SQL flows.Drupal 8 DevOps . Profile and SQL flows.
Drupal 8 DevOps . Profile and SQL flows.
Andrii Podanenko
 
Normalizing x pages web development
Normalizing x pages web development Normalizing x pages web development
Normalizing x pages web development
Shean McManus
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
Stanislav Pogrebnyak
 
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
Ambassador Labs
 
Devops interview questions 1 www.bigclasses.com
Devops interview questions  1  www.bigclasses.comDevops interview questions  1  www.bigclasses.com
Devops interview questions 1 www.bigclasses.com
bigclasses.com
 

Similar to DevOps Fest 2020. immutable infrastructure as code. True story. (20)

Successful DevOps implementation for small teams a true story
Successful DevOps implementation for small teams  a true storySuccessful DevOps implementation for small teams  a true story
Successful DevOps implementation for small teams a true story
 
Docker in Production at the Aurora Team
Docker in Production at the Aurora TeamDocker in Production at the Aurora Team
Docker in Production at the Aurora Team
 
Using Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at SplunkUsing Docker EE to Scale Operational Intelligence at Splunk
Using Docker EE to Scale Operational Intelligence at Splunk
 
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitche...
 
Meetup 2020 - Back to the Basics part 101 : IaC
Meetup 2020 - Back to the Basics part 101 : IaCMeetup 2020 - Back to the Basics part 101 : IaC
Meetup 2020 - Back to the Basics part 101 : IaC
 
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...
 
Old Is the New New
Old Is the New NewOld Is the New New
Old Is the New New
 
Product! - The road to production deployment
Product! - The road to production deploymentProduct! - The road to production deployment
Product! - The road to production deployment
 
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
Continuos Integration and Delivery: from Zero to Hero with TeamCity, Docker a...
 
The "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/OpsThe "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/Ops
 
JUST EAT: Embracing DevOps
JUST EAT: Embracing DevOpsJUST EAT: Embracing DevOps
JUST EAT: Embracing DevOps
 
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
 
Scalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUs
Scalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUsScalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUs
Scalable TensorFlow Deep Learning as a Service with Docker, OpenPOWER, and GPUs
 
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeConfoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
 
How Percolate uses CFEngine to Manage AWS Stateless Infrastructure
How Percolate uses CFEngine to Manage AWS Stateless InfrastructureHow Percolate uses CFEngine to Manage AWS Stateless Infrastructure
How Percolate uses CFEngine to Manage AWS Stateless Infrastructure
 
Drupal 8 DevOps . Profile and SQL flows.
Drupal 8 DevOps . Profile and SQL flows.Drupal 8 DevOps . Profile and SQL flows.
Drupal 8 DevOps . Profile and SQL flows.
 
Normalizing x pages web development
Normalizing x pages web development Normalizing x pages web development
Normalizing x pages web development
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
 
Devops interview questions 1 www.bigclasses.com
Devops interview questions  1  www.bigclasses.comDevops interview questions  1  www.bigclasses.com
Devops interview questions 1 www.bigclasses.com
 

More from Vlad Fedosov

OdessaJs 2020 - How to build your first micro frontend in a matter of minutes
OdessaJs 2020 - How to build your first micro frontend in a matter of minutesOdessaJs 2020 - How to build your first micro frontend in a matter of minutes
OdessaJs 2020 - How to build your first micro frontend in a matter of minutes
Vlad Fedosov
 
Maximizing your professional value, from junior to leader
Maximizing your professional value, from junior to leaderMaximizing your professional value, from junior to leader
Maximizing your professional value, from junior to leader
Vlad Fedosov
 
XP Days 2019: First secret delivery for modern cloud-native applications
XP Days 2019: First secret delivery for modern cloud-native applicationsXP Days 2019: First secret delivery for modern cloud-native applications
XP Days 2019: First secret delivery for modern cloud-native applications
Vlad Fedosov
 
JSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendJSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontend
Vlad Fedosov
 
LvivCSS: Web Components as a foundation for Design System
LvivCSS: Web Components as a foundation for Design SystemLvivCSS: Web Components as a foundation for Design System
LvivCSS: Web Components as a foundation for Design System
Vlad Fedosov
 
KharkivJS: Flaws of the Web Components in 2019 and how to address them
KharkivJS: Flaws of the Web Components in 2019 and how to address themKharkivJS: Flaws of the Web Components in 2019 and how to address them
KharkivJS: Flaws of the Web Components in 2019 and how to address them
Vlad Fedosov
 

More from Vlad Fedosov (6)

OdessaJs 2020 - How to build your first micro frontend in a matter of minutes
OdessaJs 2020 - How to build your first micro frontend in a matter of minutesOdessaJs 2020 - How to build your first micro frontend in a matter of minutes
OdessaJs 2020 - How to build your first micro frontend in a matter of minutes
 
Maximizing your professional value, from junior to leader
Maximizing your professional value, from junior to leaderMaximizing your professional value, from junior to leader
Maximizing your professional value, from junior to leader
 
XP Days 2019: First secret delivery for modern cloud-native applications
XP Days 2019: First secret delivery for modern cloud-native applicationsXP Days 2019: First secret delivery for modern cloud-native applications
XP Days 2019: First secret delivery for modern cloud-native applications
 
JSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontendJSFest 2019: Technology agnostic microservices at SPA frontend
JSFest 2019: Technology agnostic microservices at SPA frontend
 
LvivCSS: Web Components as a foundation for Design System
LvivCSS: Web Components as a foundation for Design SystemLvivCSS: Web Components as a foundation for Design System
LvivCSS: Web Components as a foundation for Design System
 
KharkivJS: Flaws of the Web Components in 2019 and how to address them
KharkivJS: Flaws of the Web Components in 2019 and how to address themKharkivJS: Flaws of the Web Components in 2019 and how to address them
KharkivJS: Flaws of the Web Components in 2019 and how to address them
 

Recently uploaded

Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
Priyanka Aash
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 

Recently uploaded (20)

Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 

DevOps Fest 2020. immutable infrastructure as code. True story.

  • 1. Immutable Infrastructure as Code. True story. Vladlen Fedosov, Director of R&D @Namecheap, Inc
  • 2. Vladlen Fedosov Director of R&D @Namecheap TL;DR: • 10 years in the industry • Went path from Junior to Architect  • Amateur DevOps evangelist  • AWS ninja • Believe in self-organized, cross-functional teams
  • 3. “Opening the door for everyone to a free and open Internet”
  • 4. The beginning Disclaimer: Here I’m talking mostly about my experience and part of the infrastructure that my team was responsible for. So further statements may not apply to every department in the company. Timeline for the things mentioned below was changes to simplify storytelling.
  • 6. Project takeover from outsourcing company
  • 7. Brave New World For a small dev team with no infra support / capabilities
  • 8. So what do we have now?
  • 9. State of the project after takeover ● Half broken Chef cookbooks ● Sketchy CD pipelines ● Fault tolerance in place ● And… Everything went down after failure of the single (out of 3) etcd node. We realized that we have fault tolerance only on paper
  • 10. After some refactoring… We’ve got typical setup
  • 11. Issues we noticed - Multiple apps were sharing same OS, language versions, dependencies - Horizontal scaling was hard - Sometimes failing chef scripts on random instances mostly due to network errors & configuration differences
  • 12. Configuration Synchronisation problem, in short martinfowler.com/bliki/ConfigurationSynchronization.html
  • 13. Issues we noticed - Manual sync of the AWS setup between environments - Manually configured CI/CD - Easy to break something and hard to repair or modify anything
  • 14. Blue sky vision ● Immutable infrastructure ● Everything as code ○ Infrastructure as code ○ CI/CD as code
  • 15. Blue sky vision ● Hard to break ● Easy to repair ● Easy to modify
  • 17. What is immutable infrastructure? martinfowler.com/bliki/ImmutableServer.html
  • 18. Why go Immutable? ● Forget about change flow, only “create” matters ● Defeat Configuration Drift ● Use much simpler tools ● Build highly available systems easier ● Fix issues faster
  • 19. How to achieve this? ● Complexity → Docker (or AMI) images ● OS → Docker runtime only ● App/OS configuration methods: ○ K8s pod definitions (or similar) ○ “cloud-init” ● Terraform to define “datacenter config”
  • 20. Main tools we use ● AMI images & Docker images ● Cloud-Init (98%) ● Hashicorp Packer (2%) ● Terraform, Terraform everywhere
  • 21. Single fact you could memorise here Immutable Infrastructure allows you to significantly simplify management steps and consequently reduce number of bugs your customers will face with. Work with images rather than servers.
  • 25. Everything as code: full list ● Everything as code, it supports: ○ job steps definition as code (via pipelines) ○ jobs creation as code (via job dsl) ○ system configuration as code (via groovy API, XML configs & CasC yml) ● Shared libraries, ability to share common steps between apps P.S: Talk to me if know better alternative ;)
  • 26. Other factors influenced the choice ● Has deployment dashboard so we can see the state of all the environments ● Highly extensible ● Elastic EC2 instances as agents
  • 27. Nowadays ● It went beyond one project and now almost every team at Namecheap uses it ● 300+ pipelines ● Around 38 projects ● We expect even more in the future
  • 28. Nowadays: CI @Library('namecheap/common') _ node('CommonCLarge') { ciJavascript.servicePipeline { productName = "ProductA" serviceName = "Apps.Api" } }
  • 29. Nowadays: CD @Library('namecheap/common') _ properties([parameters([ string(name: 'image', description: 'Application container name'), string(name: 'version', description: 'Application container tag'), choice(choices: ['production', 'sandbox'], description: 'Environment', name: 'env') ])]) node() { def authToken = "XXXX" stage('Deploy') { deployToOKD(params, authToken) } }
  • 30. Lessons learned ● Hide all complexity inside. Provide as much logic as you can in a form of Shared Libraries. ● Documentation & examples are crucial for developers ● If possible - provide standardized pipelines invoked as Shared Library function ● It takes about 1.5 months for 2 people to setup Jenkins properly for the first time
  • 31. Single fact you could memorise here Always keep your CI/CD configuration, written as code, near the app, in the same repo. It will give an understanding to everyone in your company on how to build & deploy any app.
  • 33. How Terraform is different to Ansible/Chef/Puppet Terraform is not a configuration management tool. It focuses on the higher-level abstraction of the datacenter (or cloud provider), without sacrificing the ability to use configuration management tools to do what they do best: bootstrapping and initializing resources. resource "aws_instance" "web" { ami = "ami-dbc3b9aa" instance_type = "t2.micro" }
  • 34. Imperative VS Declarative infrastructure code Declarative “Can I have a cup of coffee on my desk at 9AM on Monday morning?” Imperative “Go to that machine, then get the glass jar, then fill it with water, then put it back in the machine” …you get the idea...
  • 35. Why go Declarative? Key challenges this approach solves for us: ● Dealing with “Configuration Drift” / State management ● Idempotency ● Dependency graph management, correct order of operations
  • 37. Deploy with Terraform What we had: 1. Write TF configs 2. Run TF to create infrastructure 3. Take TF outputs & enter them to Jenkins 4. Deploy app itself with Jenkins What we wanted to have: 1. Write TF configs 2. Deploy app
  • 39. Deploy with Terraform !"" vars <-- Environment specific variables # !"" production.tfvars # !"" staging.tfvars !"" main.tf !"" io.tf !"" db.tf !"" etc.tf
  • 40. Single fact you could memorize here Try to have as much declarative infrastructure configs as you can, avoid imperative scripts at all cost.
  • 41. Learnings & further improvements
  • 42. Tests for infrastructure code The more infrastructure code you have - the more bugs you see.
  • 43. Chaos monkey We wrote a Lambda that randomly reboots every instance once a day. This simple tweak ensures that: ● Apps you launch can survive instance failure ● Updates to the cluster setup is easy as you’re sure that you won’t harm anyone by killing outdated machines ● Problem resolution can be simpler sometimes. You can always reboot/kill any instance that behaves abnormally as a first action Apply this to control fleet too
  • 44. No SSH keys distribution over instances ● If you’re using AWS - simply install SSM agent to your instances and disable SSH daemon. You will be able to use SSH console to perform your administrative actions. ● If you’re not in AWS - you can use Hashicorp Vault. It provides you with SSH backend that allows central management & audit of the login identities.
  • 45. Things that work for 3 teams - doesn’t work for 10 Issues # of users
  • 46. Key learnings here: ● Operational work grows exponentially the more teams you add ● 1 new tool/approach for devs at a time ● Conduct educational courses for big new things like Docker, AWS, Terraform ● Gain trust within the team you’re challenging with a change first ● Documentation is paramount, start it as early as possible Things that work for 3 teams - doesn’t work for 10
  • 47. DevOps on Call & transparent SLAs ● We’ve established “on call” schedule ● Agreed on SLAs & shared then among the teams ● Created chat room & Jira board Result: significant reduction of the distraction level, better productivity, happier teams Even you’re doing good now, you can make it even better with this practice
  • 48. Encourage feedback ● Ask for it proactively, show that it's important for you ● Public retrospectives ● Respond to the feedback
  • 50. What we’ve achieved ● Immutable infrastructure (done) ○ ECS with immutable data plane ○ Immutable EC2 instances for stateful instances ● Everything as code (done) ○ Infrastructure as code: Terraform, Cloud-init ○ CI/CD as code: Jenkins Now it’s hard to break & easy to repair things as well as
 easy to track changes.
  • 51. Vlad Fedosov Director of R&D @Namecheap, Inc vlad.fedosov@gmail.com Or just scan it: