1.
Building
EVOLUTIONARY
infrastructure

2.
kief@thoughtworks.com
Infrastructure Technologist@
Twitter: @kief
Book: http://oreil.ly/1JKIBVe
Site: http://infrastructure-as-code.com
Some visuals found from the Vault of the Atomic Space Age
http://bit.ly/2KBTbAd

3.
Why?
Ongoing pressure to deliver while
requirements change

4.
PRODTESTDEV
Inconsistent path to production

5.
Snowflake production

6.
PROD
Continuous, reliable delivery of changes
to all your stuff
TESTDEV

7.
"Since we can't avoid
change, we need to
exploit it"
Building Evolutionary
Architectures
Neal Ford, Rebecca Parsons,
Pat Kua

8.
Speed or Quality?

9.
Fast
Slow Careless
Careful

10.
Move
fast and
fix
things

11.
Fast
Slow Careless
Careful

12.
FastSlow
Careful
Careless

13.
FastSlow
Careless
Prioritize
speed over
correctness
Fragile mess

14.
Slow
Careful
Careless
Prioritize
correctness over
speed of change
Fragile mess

15.
Slow
Careful
Careless
Prioritize
correctness over
speed of change
Fragile mess
Time
pressures
Over-
engineering
Technical
debt
Barriers to
improvement

16.
FastSlow
Careful
Careless
Agile, Lean,
DevOpsBuild the
simplest thing
Build quality in
Optimize for
change
Fast,
frequent
feedback

17.
Infrastructure Tools and
Technology

18.
Dynamic infrastructure platform
Change delivery services
Infrastructure management tools
Operational
management
services
Application Packaging and
Deployment
Development Tooling
Application runtime platform

19.
Dynamic infrastructure platform
Change delivery services
Infrastructure management tools
Operational
management
services
Application Packaging and
Deployment
Development Tooling
Application runtime platform

20.
Dynamic infrastructure platform
Compute StorageNetwork
A pool of resources

21.
Provision resources
• On demand
• Programmatically

22.
Physical Servers
Containers
Serverless (FaaS)
Variations of compute resources
Virtual Machines

23.
Dynamic infrastructure platform
Change delivery services
Infrastructure management tools
Operational
management
services
Application Packaging and
Deployment
Development Tooling
Application runtime platform

24.
Stack management tool
Stack instances
Platform API
Stack definition
(Terraform, CloudFormation, ...)

25.
Server configuration tool
Configuration
tool
Configuration
definitions
Server instance
(Ansible, Chef, Puppet, Saltstack, ...)

26.
Server image creation
Base server image
Custom
image
Server
configuration
definitions
Image
definition
(Packer, ...)

27.
Concept: Infrastructure Stack

28.
Infrastructure stack
A collection of infrastructure
resources provisioned and
updated as a unit

29.
A stack may include
the infrastructure for
a single application

30.
Or, multiple
applications
might run in
a single stack

31.
Or the parts of one
application could be
spread across several
stacks

32.
Stack definition
A stack definition is code used by a tool to
provision stack instances
Stack definition
Stack instanceStack management tool

33.
Stack instances
A single stack definition may be used to
manage one or more stack instances
Stack definition
Stack
instances

34.
Multiple environments

35.
Starting small – single stack instance
One or two
people working
on it,
... the system is
fairly simple,
... and there
are few users,
with low stakes

36.
And then ...
More people
join the team, ... the system becomes
more complex,
... and more
people rely on
the system to
be working

37.
PRODUCTION
QA
TEST
Multiple environments

38.
PRODUCTION
QA
TEST
Multiple environments defined in one
stack instance our_env/
└── test.tf
└── qa.tf
└── production.tf

39.
But the blast radius for a change is too
wide
PRODUCTION
QA
TEST

40.
our_env/
└── test/
└── servers.tf
our_env/
└── qa/
└── servers.tf
our_env/
└── production/
└── servers.tf
PRODUCTION
QA
TEST
Separate stack definition project for
each stack instance

41.
QA
PRODUCTION
TEST
Single definition
project template,
used to create a
separate stack
instance for each
environment
our_env/
└── servers.tf

42.
Pipelines for infrastructure

43.
Automatically test every change before
applying it

44.
Promote changes to environments using
a pipeline
BUILDLOCAL
APPLY
TO QA
APPLY
TO
PROD
Sandbox
QA Production
APPLY
TO TEST
Test

45.
Stack tool is only run by pipeline agents
BUILDLOCAL
APPLY
TO QA
APPLY
TO
PROD
APPLY
TO TEST

46.
Why not apply changes from our
laptops?
Our working environments
may not be consistent
Multiple people may clash
We may shortcut our
process
We may neglect to run all
appropriate tests

47.
"Local" pre-commit sandboxes
VCS
Nita's Instance
Alex's Instance
Freddy's Instance

48.
Parameterizing stacks

49.
server "web" {
name = "web-${env}"
type = "${type}"
}
env = "prod"
type = "t2.large"
env = "qa"
type = "t2.large"
env = "test"
type = "t2.tiny"
web-test
web-qa
web-prod
Parameters per stack instance

50.
Parameters in source project folders
web-test
web-qa
web-prod
.
├── infra
│ ├── web_server.tf
│ ├── subnet.tf
│ └── vpc.tf
└── environments
├── test.tfvars
├── qa.tfvars
└── prod.tfvars

51.
env
=
"test"
type
=
"t2.tiny"
env
=
"qa"
type
=
"t2.large"
env
=
"prod"
type
=
"t2.large"
Parameters in pipeline job definitions
BUILD
APPLY
TO QA
APPLY
TO
PROD
APPLY
TO TEST
web-test web-qa web-prod

52.
Parameters from configuration registry
web-test
web-qa
web-prod
envs/test/type t2.tiny
envs/qa/type t2.large
envs/prod/type t2.large

53.
Recommendation
Minimize the configurable surface area to
reduce variation between environments

54.
Infrastructure testing

55.
An effective testing regime
drives a clean, decoupled,
evolvable system design

56.
Fitness functions
Define the important characteristics of your
system, how to measure these, and how to
assess progress

57.
Example: our-app stack
Infrastructure
stack
JDK
Tomcat
OurApp.war
Application
server

58.
QA PRODTEST ALL
TEST JDK
COOKBOOK
TEST
TOMCAT
COOKBOOK
TEST APP
SERVER
ROLE
BUILD APP
SERVER
AMI
VALIDATE
INFRA
STACK
BUILD
OUR-APP
TEST
INFRA
STACK
TEST OUR-
APP
Example pipeline for our-app

59.
QA PRODTEST ALL
TEST JDK
COOKBOOK
TEST
TOMCAT
COOKBOOK
TEST APP
SERVER
ROLE
APP
SERVER
AMI
VALIDATE
INFRA
STACK
BUILD
OUR-APP
TEST
INFRA
STACK
TEST OUR-
APP
APPLICATION STAGES:
Compile, unit tests,
container tests,
packaging, etc.
(run on build agents)
JUnit, Rspec,
etc.

60.
QA PRODTEST ALL
TEST JDK
COOKBOOK
TEST
TOMCAT
COOKBOOK
TEST APP
SERVER
ROLE
BUILD APP
SERVER
AMI
BUILD
INFRA
STACK
BUILD
OUR-APP
TEST
INFRA
STACK
TEST OUR-
APP
SERVER PACKAGES:
Validate, test individual package
configurations in isolation
SERVER ROLE:
Test aggregated
packages
Typically run on build
agents, perhaps
containerized
test-kitchen, serverspec,
inspec, puppet-rspec,
testinfra, etc.

61.
QA PRODTEST ALL
TEST JDK
COOKBOOK
TEST
TOMCAT
COOKBOOK
TEST APP
SERVER
ROLE
BUILD APP
SERVER
AMI
VALIDATE
INFRA
STACK
BUILD
OUR-APP
TEST
INFRA
STACK
TEST OUR-
APP
SERVER IMAGE:
Build and test
serverspec,
inspec, etc.

62.
QA PRODTEST ALL
TEST JDK
COOKBOOK
TEST
TOMCAT
COOKBOOK
TEST APP
SERVER
ROLE
BUILD APP
SERVER
AMI
VALIDATE
INFRA
STACK
BUILD
OUR-APP
TEST
INFRA
STACK
TEST OUR-
APP
INFRASTRUCTURE STACK
Validate (e.g. syntax,
provision non-application
specific infrastructure
elements
awspec, inspec, etc.

63.
QA PRODTEST ALL
TEST JDK
COOKBOOK
TEST
TOMCAT
COOKBOOK
TEST APP
SERVER
ROLE
BUILD APP
SERVER
AMI
VALIDATE
INFRA
STACK
BUILD
OUR-APP
TEST
INFRA
STACK
TEST OUR-
APP
INTEGRATED STAGES:
Test the various
components in
combination
(deployment environments)
selenium, etc.

64.
Recommendations
Those who write the code write the tests
Avoid reflexive testing (AKA testing your tools)
Test where there is risk, complexity
Test contracts

65.
Splitting stacks into
microstacks

66.
Monolithic stack

67.
Coordinating changes
Team A could make a change that breaks
things for team B

68.
Divide infrastructure into multiple,
independently changeable stacks

69.
Each stack has its own pipeline
to deliver changes

70.
Strategies for drawing
boundaries

71.
Draw boundaries to minimize
changes that cross stacks

72.
Infrastructure code boundaries !=
network boundaries

73.
Infrastructure code boundaries !=
network boundaries

74.
Infrastructure code boundaries !=
network boundaries

75.
Group things together when they tend to
be changed together

76.
Align team boundaries and change
boundaries

77.
Patterns for infrastructure
stacks

78.
Shared infrastructure
components

79.
Base infrastructure module pipeline
Application stack pipelines
Components shared across stacks

80.
Challenges with shared components
Increases
coupling

81.
Shared infrastructure stacks

82.
Shared stack instance
Shared
network
resource stack
Multiple
application
infrastructure
stacks

83.
Shared stack pipeline
Application infrastructure pipelines

84.
Shared infrastructure stacks
The provider
stack should not
have knowledge
of its consumers
Becomes a
bottleneck as the
number of
consumers
grows
Should be kept
simple, minimal
Split into smaller
stacks to avoid
monoliths

85.
Shared nothing stacks

86.
Each stack is self-contained

87.
Each stack is independently releasable

88.
Techniques for integrating
stacks

89.
Example of integrated stacks
Shared
infrastructure
stack
Application
infrastructure
stack
vpc_id = "vpc-abcde"
private_subnets = ["sn-12345", "sn-54321"]

90.
Integration at apply-time
vpc_id = "vpc-abcde"
private_subnets = ["sn-12345", "sn-54321"]

91.
Integration by resource discovery
data "aws_vpc" "vpc" {
filter {
name = "tag:Environment"
values = ["${var.env}"]
}
}
data "aws_subnet" "private_subnet" {
filter {
name = "tag:Environment"
values = ["${var.env}"]
}
}

92.
Integration by configuration registry
envs/qa/vpc_id vpc-abcde
envs/qa/private_subnets "sn-12345", "sn-54321"

93.
Integration by remote state

94.
Recommendations
Stack integration points are contracts
Follow good practice for API design when making
changes to contracts
Use fitness functions (automated tests) to ensure the
integrity of contracts

95.
kief@thoughtworks.com
Infrastructure Technologist@
Twitter: @kief
Book: http://oreil.ly/1JKIBVe
Site: http://infrastructure-as-code.com