Moving from the Iron Age to the Cloud Age in computing is supposed to save us money yet many migrations seem to cost more in the long run and result in infrastructures as complex to manage as what we had before. This is often the result of the so called “lift & shift” approach many take – it’s a short term win that doesn’t address why you wanted to move to the cloud in the first place. The Cloud Age affords us the opportunity not to treat our infrastructure as something special, instead as something disposable. By applying the practices of continuous integration and delivery to our infrastructure and configuration management we can built truly scalable infrastructures to host our application’s wildest dreams. In this talk we will look at the tools and processes that can be adopted to truly make use of the possibilities of the Cloud.

1. Mike Fowler (mike@bashton.com) FLOSS UK Spring 2017
The Disposable Infrastructure

2. ●
Currently a Senior Site Reliability Engineer for
Claranet|Bashton
●
Background in Software Engineering, Systems
Engineering & System Administration
●
Contributed to several open source projects (YAWL,
PostgreSQL & Terraform)
●
Been using Linux & Open Source software since
1999
About Me

3. ●
Typical cloud migration strategies
●
The drawbacks of typical cloud migration strategies
●
The five theses of a disposable infrastructure
●
How we build a disposable infrastructure
Overview

4. ●
Duplicate existing estate in the cloud
●
Use cloud as spare/batch capacity
●
Brave New World
– Greenfield development
– “Version 2.0”
Approaching Cloud Migration

5. ●
Direct mapping of existing infrastructure to the cloud
– Load balancers become Elastic Load Balancers
– SANs become Buckets or Elastic File Systems
●
Minimal operational change required
– Everything is the same just in a new location
●
Perceived as a “quick win” to cloud adoption
– Little AWS/GCP/Azure specific knowledge required
The Appeal of a Lift & Shift

6. ●
No “legacy” baggage
●
Free reign for experimentation
●
Perceived as a “low risk” path to cloud adoption
– If it doesn’t work, switch it off
– “No risk” to existing production environment
The Appeal of a Brave New World

7. ●
Are we just building a traditional but virtual data
centre?
– Lift & Shift is operationally the same
– Brave New World isn’t part of the Real World
●
How are we leveraging the power of a dynamic
infrastructure?
– Our infrastructure is scalable, but is the application?
●
Are we using IAAS where we could be using PAAS?
– e.g. Running a message broker instead of SQS
Are we really “doing cloud”?

8. ●
We’re changing only where our hardware is
– Instance size based on current hardware size
– No change to deployment process
●
Under utilisation of resource
– Still paying for excess capacity
●
Stunted scalability
– We can throw more virtual hardware at it
– Add additional node behind load balancers
The Penalty of a Lift & Shift

9. ●
Organisationally isolated
– Limited impact to existing practices
– Leads to a “Us vs. Them” mentality
●
Focus is usually on application functionality with
infrastructure seen as a necessity
●
Project has a high risk of failure
– Care free scoping leads to an unfocused project
– Significant time can be lost to integrating with the
old world
The Penalty of a Brave New World

10. ●
Conway’s Law: “Organisations which design systems … are
constrained to produce designs which are copies of the
communication structures of these organisations”
●
Kief Morris: “In many cases, applying existing patterns will, at
best, miss out on opportunities to leverage newer technology to
simplify and improve the architecture. At worst, replicating existing
patterns with the newer platforms will involve adding even more
complexity.”
●
Fred Brooks: “A scaling-up of a software entity is not merely a
repetition of the same elements in larger size; it is necessarily an
increase in the number of different elements. In most cases, the
elements interact with each other in some nonlinear fashion, and
the complexity of the whole increases much more than linearly.”
Breaking the Mould

11. 1) The attitude you have to your environment will
determine the limits of your scalability
2) Continuous integration (CI) and delivery (CD) is a must
3) Your applications need to be (re)built to fit a dynamic
infrastructure
4) Dynamic infrastructure must be treated as a first class
citizen in any cloud project
5) Planning to fail will lead to success
The Five Theses

12. ●
The more you care about individual things the more
they will hold your attention
●
In a truly scalable environment you should only care
about the combination of many individual things
Attitude
Thesis 1
The attitude you have to your environment will
determine the limits of your scalability

13. ●
You treat your servers like pets
– You give them names (igloo, husky, snowshoe)
– You give them homes (racks on site or co-located)
– If they fail, you do everything you can to save them
●
Every server is an investment
– Often the best hardware that can be afforded
– Amortised over years
– Excess capacity to allow for growth
●
Provisioning new servers takes weeks
Attitude: Living in the Iron Age

14. ●
You treat your servers like cattle
– They have identifiers
– You care only where they are geographically
– If they fail, you put them down and get a new one
●
Your architecture is your investment
– Configuration is chosen for your current load
– Pay for what you use
– Capacity can be added when required
●
Provisioning new servers takes seconds
Attitude: Living in the Cloud Age

15. ●
Are we simply herding our pets?
– In a Lift & Shift this is almost certainly so
– Scaling groups is a start but it is not the end
●
How are we managing our virtual servers?
– Complex cloud-init scripts?
– Traditional configuration management?
Attitude: Is Pets v Cattle enough?

16. ●
Everything is a package and can be discarded
●
You treat your servers like single use products
– They’re pre-packaged for a particular purpose
– You still care only where they are geographically
– If they fail, you toss it away and grab another
●
You automate everything
– Servers should be immutable
– Never make a manual change
Attitude: The Disposable Infrastructure

17. ●
Repeatability brings reliability and predictability
●
Defining a build pipeline:
– Ensures the same process is followed for every
change
– Provides an audit trail for every change
– Gives visibility of your value stream
Be Continuous
Thesis 2
Continuous integration (CI) and delivery (CD) is a must

18. ●
Your developers probably already practice CI
– It is the standard for code development
– The output of CI can be the start of CD
●
Continuous delivery doesn’t have to mean continuous
deployment
– Build pipelines can have approval stages
– Every change should be deployable
Be Continuous

19. ●
Many applications expect a static infrastructure
– Hard-coded assumptions that an IP address won’t
change once an application is started
●
Many applications are cluster unaware
– Sticky sessions on load balancers can help
– Some protocols don’t load balance well
Reactoring to the Cloud
Thesis 3
Your applications need to be (re)built to fit a dynamic
infrastructure

20. ●
Refactor to contemporary architectural approaches
– Service Oriented Architectures & Microservices
– Transition from stateful services to stateless
●
Package everything using distribution packagers
– The output of your build pipeline is a RPM/DEB
– Your $CM_TOOL already supports this
●
Chose a deployment strategy
– Machine images vs. containers
Reactoring to the Cloud

21. ●
Fear not vendor lock in, savings are to be reaped
leveraging commodity services
– Use SQS instead of automating the installation and
configuration of a message broker and accepting
the operational burden of maintaining it
– Careful abstraction of the API will allow porting to a
different platform if absolutely necessary
Reactoring to the Cloud

22. ●
Design the infrastructure in parallel to the cloud aware
application changes
●
Mandate every instance is part of a scaling group to
enforce cluster awareness
●
Use the same principles for infrastructure development
as you use for applications
Infrastructure is Code
Thesis 4
Dynamic infrastructure must be treated as a first class
citizen in any cloud project

23. ●
Script/encode everything unless there is no API/tooling
support
●
Deploy the same infrastructure in development, test
and production environments
– Sizing can be parameterised
●
Your deployment pipeline becomes the assembly of
application packages and infrastructure configuration
●
High cohesion and loose coupling applies to
infrastructure as much as it does to applications
Infrastructure is Code

24. ●
If it can go wrong, it will go wrong so think in terms of
when and not if
●
Treating our infrastructure and its hosted applications
as disposable in conjunction with CD eliminates a
number of failure scenarios
Planning to fail
Thesis 5
Planning to fail will lead to success

25. ●
Regularly test your disposability
– Terminate instances at random to ensure resiliency
– Block all network access to an instance
– Chaos Monkey & the Simian Army
– Trigger failovers for less disposable services
●
Constantly churning disposable instances helps
prevent configuration drift of immutable servers
Planning to fail

26. ●
Availability and durability cost
– Favour numerous small instances over a handful of
large instances
●
Identify points of failure and assess:
– How often will this failure occur?
– How do I mitigate this failure?
– How do I test this failure to ensure mitigation?
– Is the cost of mitigation worth the customer impact
during failure?
Planning to fail

27. ●
Be honest in assessing the worth of your business
– Do you really need to double your costs to run in
multiple regions?
– Trello, Slack & many other high profile companies –
including Amazon - were affected by the S3 outage
Planning to fail

28. ●
Data is not disposable and is probably more important
than your availability
●
Ship log files to CloudWatch or Stackdriver
●
Make back-ups and regularly test they restore
– Gitlab had 5 separate backup processes, they all
failed
– Consider storing backups in both S3 & Google
– Store backups in multiple regions
Data is not Disposable

29. ●
If you must use persistent disks:
– Use multiple disks and use RAID-1
– Snapshot the disks regularly
●
Test the durability of your data
– User error is your biggest risk
●
“I forgot the WHERE clause”
●
“I thought I was in the test environment”
– Regularly exercise data loss & recovery scenarios
in development and test environments
Data is not Disposable

30. ●
We will have two identical CI pipelines for the
applications, the output of each being AMI images
●
A separate CD pipeline executes infrastructure code
and rolls out the new AMIs
●
Goal is to promote infrastructure and AMIs between
environments
Under Construction
Let us assume we have a front end web application
which places orders in a queue for subsequent
asynchronous fulfilment by a separate application
backed by a database. We’ve already refactored our
applications for the cloud.

31. ●
Source our code from a repo, build and test
●
Package our application as a DEB or RPM
●
Place our artifact into a S3 repository
●
Run Packer to generate a new AMI
Application Pipeline

32. ●
https://packer.io
●
Can create many different machine images
●
Consider creating a base image to control OS updates
●
Use normal configuration management tools
– Support for Ansible, Chef & Puppet
– Can just write shell script if you must
●
Use placeholders for configuration to be filled by
launch scripts
Packer

33. ●
Triggered by new AMIs or Terraform code changes
●
Apply Terraform to update the infrastructure
●
Run integration tests to verify application build
●
Wait for approval before promotion to next
environment
Infrastructure Pipeline

34. ●
https://terraform.io
●
Declarative language for the construction of
infrastructure
●
Supports all major vendors
●
State can be stored in buckets to facilitate sharing
●
Separate out infrastructure layers
– Minimises blast radius of changes
– Keep persistent apart from disposable
Terraform

35. ●
Any instance can be terminated
●
Resilient to zone failure
●
Cross-region read replica allows DR for region failure
– Just need to run Terraform to add the instances
when required and update Route 53
Deployed Infrastructure

36. ●
Have attitude
●
Be continuous
●
Refactor to the Cloud
●
Infrastructure is code
●
Plan to fail
●
Data is King
Summary