This document discusses the journey of transitioning infrastructure management at Namecheap to an immutable infrastructure as code model using tools like Terraform, Docker, and Jenkins. Key points include taking over a project from an outsourcing company, setting up immutable infrastructure with infrastructure as code, configuring CI/CD pipelines as code in Jenkins, and lessons learned around testing, chaos engineering, and encouraging team feedback. The overall goals were to make infrastructure hard to break, easy to repair, and easy to modify.
2. Vladlen Fedosov
Director of R&D @Namecheap
TL;DR:
• 10 years in the industry
• Went path from Junior to Architect
• Amateur DevOps evangelist
• AWS ninja
• Believe in self-organized, cross-functional teams
4. The beginning
Disclaimer: Here I’m talking mostly about my experience and part of the infrastructure that my team
was responsible for. So further statements may not apply to every department in the company. Timeline for the things
mentioned below was changes to simplify storytelling.
9. State of the project after takeover
● Half broken Chef cookbooks
● Sketchy CD pipelines
● Fault tolerance in place
● And… Everything went down after failure of the single (out of 3)
etcd node. We realized that we have fault tolerance only on paper
11. Issues we noticed
- Multiple apps were sharing same OS, language versions, dependencies
- Horizontal scaling was hard
- Sometimes failing chef scripts on random instances mostly due to network
errors & configuration differences
13. Issues we noticed
- Manual sync of the AWS setup between environments
- Manually configured CI/CD
- Easy to break something and hard to repair or modify anything
14. Blue sky vision
● Immutable infrastructure
● Everything as code
○ Infrastructure as code
○ CI/CD as code
17. What is immutable infrastructure?
martinfowler.com/bliki/ImmutableServer.html
18. Why go Immutable?
● Forget about change flow, only “create” matters
● Defeat Configuration Drift
● Use much simpler tools
● Build highly available systems easier
● Fix issues faster
19. How to achieve this?
● Complexity → Docker (or AMI) images
● OS → Docker runtime only
● App/OS configuration methods:
○ K8s pod definitions (or similar)
○ “cloud-init”
● Terraform to define “datacenter config”
20. Main tools we use
● AMI images & Docker images
● Cloud-Init (98%)
● Hashicorp Packer (2%)
● Terraform, Terraform everywhere
21. Single fact you could memorise here
Immutable Infrastructure allows you to significantly simplify
management steps and consequently reduce number of
bugs your customers will face with. Work with images rather
than servers.
25. Everything as code: full list
● Everything as code, it supports:
○ job steps definition as code (via pipelines)
○ jobs creation as code (via job dsl)
○ system configuration as code (via groovy API, XML configs & CasC yml)
● Shared libraries, ability to share common steps between apps
P.S: Talk to me if know better alternative ;)
26. Other factors influenced the choice
● Has deployment dashboard so we can see the state of all the
environments
● Highly extensible
● Elastic EC2 instances as agents
27. Nowadays
● It went beyond one project and now almost every team at Namecheap uses it
● 300+ pipelines
● Around 38 projects
● We expect even more in the future
30. Lessons learned
● Hide all complexity inside. Provide as much logic as you can in a form of
Shared Libraries.
● Documentation & examples are crucial for developers
● If possible - provide standardized pipelines invoked as Shared Library
function
● It takes about 1.5 months for 2 people to setup Jenkins properly for the first
time
31. Single fact you could memorise here
Always keep your CI/CD configuration, written as code,
near the app, in the same repo. It will give an understanding
to everyone in your company on how to build & deploy any
app.
33. How Terraform is different to Ansible/Chef/Puppet
Terraform is not a configuration management tool. It focuses on the higher-level
abstraction of the datacenter (or cloud provider), without sacrificing the ability to
use configuration management tools to do what they do best: bootstrapping
and initializing resources.
resource "aws_instance" "web" {
ami = "ami-dbc3b9aa"
instance_type = "t2.micro"
}
34. Imperative VS Declarative infrastructure code
Declarative
“Can I have a cup of coffee on my
desk at 9AM on Monday morning?”
Imperative
“Go to that machine, then get the
glass jar, then fill it with water, then
put it back in the machine”
…you get the idea...
35. Why go Declarative?
Key challenges this approach solves for us:
● Dealing with “Configuration Drift” / State management
● Idempotency
● Dependency graph management, correct order of operations
37. Deploy with Terraform
What we had:
1. Write TF configs
2. Run TF to create infrastructure
3. Take TF outputs & enter them to
Jenkins
4. Deploy app itself with Jenkins
What we wanted to have:
1. Write TF configs
2. Deploy app
43. Chaos monkey
We wrote a Lambda that randomly reboots every instance once a day. This
simple tweak ensures that:
● Apps you launch can survive instance failure
● Updates to the cluster setup is easy as you’re sure that you won’t harm
anyone by killing outdated machines
● Problem resolution can be simpler sometimes. You can always reboot/kill any
instance that behaves abnormally as a first action
Apply this to control fleet too
44. No SSH keys distribution over instances
● If you’re using AWS - simply install SSM agent to your instances and disable
SSH daemon. You will be able to use SSH console to perform your
administrative actions.
● If you’re not in AWS - you can use Hashicorp Vault. It provides you with
SSH backend that allows central management & audit of the login identities.
45. Things that work for 3 teams - doesn’t work for 10
Issues
# of users
46. Key learnings here:
● Operational work grows exponentially the more teams you add
● 1 new tool/approach for devs at a time
● Conduct educational courses for big new things like Docker, AWS, Terraform
● Gain trust within the team you’re challenging with a change first
● Documentation is paramount, start it as early as possible
Things that work for 3 teams - doesn’t work for 10
47. DevOps on Call & transparent SLAs
● We’ve established “on call” schedule
● Agreed on SLAs & shared then among the teams
● Created chat room & Jira board
Result: significant reduction of the distraction level, better productivity, happier
teams
Even you’re doing good now, you can make it even better with this practice
48. Encourage feedback
● Ask for it proactively, show that it's important for you
● Public retrospectives
● Respond to the feedback
50. What we’ve achieved
● Immutable infrastructure (done)
○ ECS with immutable data plane
○ Immutable EC2 instances for stateful instances
● Everything as code (done)
○ Infrastructure as code: Terraform, Cloud-init
○ CI/CD as code: Jenkins
Now it’s hard to break & easy to repair things as well as
easy to track changes.