Immutable infrastructure wasn't suitable for the consultancy's needs as it led to long deployment times and a lack of visibility into instance configurations. Instead, they developed a hybrid approach using Packer, Puppet, S3, and AWS services that provides faster deployments, self-healing infrastructure, and a known, verifiable state for instances. This allows them to focus on application development rather than infrastructure management.

1. Immutable Infrastructure Isn’t the Answer
Sam Bashton

3. Who am I?
• Sam Bashton
• Ran a cloud (AWS + GCP) consultancy firm until 2016
when it was acquired by Claranet Group
• Working with config management (Puppet) since 2007
• Working with AWS since 2009
• Working with GCP since 2014

5. What is this talk about?
• How we tried to use immutable infrastructure
• How and why it wasn’t right for us
• What we do instead

6. Business Model
• Charge customer for building super reliable infrastructure
• Charge customers a monthly support fee

7. Hard won experience
• Migrated over 1000 apps to public cloud
• Variety of approaches to managing infra and deploying
code
- Including Immutable Infrastructure
• 2011 onwards

11. Terraform

12. AWS Concepts
• Each customer in one or more region
• Each region has two or more data centres (‘availability
zones’)
- Most have three
• SLA says that no more than one data centre will be down
at once in a region
“Region Unavailable” and “Region Unavailability” mean that more than one Availability Zone in
which you are running an instance or task (one or more containers), as applicable, within the same
Region, is “Unavailable” to you.

13. Data Lives in Services
• We use AWS services to store all state
- RDS (MySQL, Postgres, Oracle, MS SQL)
- Elasticache (Redis)
- DynamoDB
- AWS Elasticsearch
• The instances in question are ‘stateless’

15. Immutable
immutable
/ɪˈmjuːtəb(ə)l/
unchanging over time or unable to be
changed

16. What is immutable infrastructure?
• Automatically build a golden image
• New infrastructure using the new image replaces the old
infrastructure

17. Why would I want to do that?
• Unit of deployment becomes a machine image
• Test the artifact and have confidence it’ll be the same in
production

18. Blue/Green Deployments

19. Canary Deployments

20. Fudgetown
• All the images are the same, except..
- We need to specify a different database location in each environment
• And we need to specify it in an XML config file
- We have different sizes of machine in each environment, and need to using
different JVM settings

21. Why not just build lots of images?
• Image building is automatic - why don’t we just build an
image for each environment?

22. Why not just build lots of images?
• Unit of deploy is a machine
image
• Images are created via an
imperative set of
commands
- Shell Script
- Ansible
• What is in each image?
What is different?

23. Immutable-ish
• Scripts at startup handle differences
• Consul cluster?
- consul-template

24. Fudgetown

25. Fudgetown
• Many dozens of microservices
• All with configuration files
- XML, yaml, ini, other

26. Fudgetown
• Multiple processes make up a single ‘service’
• All have to be started in a specific order

27. Fudgetown
• Deploying changes takes much longer
- ~10-15 minutes for a Packer build and deployment to test infra
• Tests on minor changes take a lot longer

28. Fudgetown
• We don’t know what the
state of our instances is,
or should be
• We don’t know whether
config files were written
successfully
• It takes ages to test
things

29. Back to the drawing board
• Doing the thing the ‘cool kids’ say they are doing is not
the path to technical success
• Our customers care whether their app is working, not how

30. What do we actually need?
• Infrastructure and configuration in a known and verifiable
state
• Self-healing
• Fault tolerant - should continue to work even if a whole
data centre (‘AZ’) fails

31. • Autoscaling which works every time
• New instances which provision quickly (autoscaling)
• Automated deployments
- Possibly Canary, Blue/Green
• Nice to have: quick to test changes
What do we actually need?

32. Instance configuration in a known state
• We need a way to describe configuration on the machine
• A declarative language
• Should tell us if something went wrong

34. Except..
• Puppet master doesn’t lend itself to an autoscaling world
- Performance bottleneck bringing up new instances
- Single point of failure
- Especially in the zone failure scenario

35. The rules
• Terminating an instance should always automatically give
you a replacement which works
- Even if external repos are down
• CentOS mirrors
• EPEL
• Elasticsearch yum repo
• Gem
• Pip
• We should expect data centre (‘AZ’) failure

36. How do we do it?
• Packer - base common AMI
• Puppet
• S3
• yum/apt
• Jenkins

37. Jenkins

38. Packer
• Build a base image
• Generally common to all roles
• Sometimes will have per-role AMIs
• pip/gem dependencies generally installed here
- Easier than building a package, even with FPM
• Install big RPMs here to save time at provisioning

39. Masterless Puppet
• Put the Puppet manifests and modules on an instance
• Run puppet apply

40. Distributing Puppet
• Puppet needs to be on every instance
• Build an RPM/DEB containing Puppet manifests/modules
• Add to a RPM/DEB repo in S3
• Script at startup (cloud-init) installs Puppet
• Puppet runs from systemd❤

41. External Repos
• Mirror CentOS, etc repos in S3
• Repos are copied as part of deployment process
- Dev repos continually updated
- When code is promoted to next step (eg staging), repos also copied
- OS upgrades are a part of the normal deployment process

42. Repos in S3
• Puppet, application code
in yum repos in S3
• Repo created from a
Terraform module
• Just drop your RPM in, it
handles metadata
generation
https://registry.terraform.io/
modules/claranet/s3-yum-
repo/aws

43. Config updates
• AWS provides SSM
• SSM triggers updating Puppet RPM, running Puppet
• ~120 seconds from commit to Puppet run finishing

44. Success
• We have been using this approach for 6+ years
• Tried other approaches
• Always came back for apps unsuitable for containerisation

45. Your problems are not my problems
• Have lovely 12 factor apps?
• Why are you wasting time building infrastructure?!

47. Career advice
• You don’t get paid to build infrastructure
• ‘Serverless’ isn’t NoOps
• Understanding distributed systems and their many failure
modes the path to future success

48. Conclusions
• Concentrate on the desired outcome, not what somebody
at a conference said worked for them
• Find the things that will give you the most success most
easily, then iterate

49. • Architect for ease of management
• Don’t be constrained by ‘best practice’
• Don’t be embarrassed by ‘ugly hacks’ when they solve real
problems
Conclusions