Immutable infrastructure has changed the way we think about system lifecycle: compute machines live for days instead of months or years, and applications live for hours or less. With the proliferation of CI/CD systems, and infrastructure as a service, the increased churn in production systems has hastened the immediate need for tools that prioritize experimentation - is your next development iteration really better than the last? In such a volatile world, traditional notions of compute “environments” and mutable approaches to experimentation can be found lacking. In large systems, emergent behaviors are near impossible to replicate in isolation, so experimenting in production systems is the only way to effectively measure hypothesis. This session covers different schemes for experimentation and the primitives required to make converged infrastructure work for real systems.

1. Experimentation with
Immutable Infrastructure.
Timothy Perrett @ Hashiconf 2017

2. Hello.
!

3. Immutable Infrastructure.
thou shall not change thy deployment!

4. Converged Infrastructure.
datacenter resources are finite.

5. Converged Infrastructure.
staging production
static provisioning

6. Converged Infrastructure.
machine
application
datacenter
operations
infrastructure
product

7. Converged Infrastructure.
machine
scheduler
container
mesh
application
datacenter
operations
infrastructure
product

8. Converged Infrastructure.
resources
scheduling substrate
workloads

9. Workloads.
• Users demand:
• Blazing fast iteration.
• High degree of observability.
• Seamless traffic migration between application revisions.
• Automated application cleanup.
• Self-service DevOps experience.

10. Workloads.
• Users demand:
• Blazing fast iteration.
• High degree of observability.
• Seamless traffic migration between application revisions.
• Automated application cleanup.
• Self-service DevOps experience.
2017 minimum
table stakes!

11. Ex·per·i·men·ta·tion
Process of performing a scientific procedure to determine something.

12. Ex·per·i·men·ta·tion
Process of performing a scientific procedure to determine something.
Testing in production

14. Testing in Production.
everyone does it. nobody wants to own up to it.

15. Emergent Behaviors.
observation of unpredictable, non-deterministic runtime artifacts.
Testing in production.

16. – Conway’s Law
“Any organization that designs a system will produce a design
whose structure is a copy of the organization's
communication structure”

17. – Hyrum’s Law
“With a sufficient number of users of an interface,
all observable behaviors of your interface will
be depended upon by somebody.”

18. Microservice Complexity.
local containerized testing is a fable.
Testing in production.

19. Inconceivable Scale.
black friday. super bowl. popular cat memes.
Testing in production.

20. Human Error.
embrace the inevitable.
http://queue.acm.org/detail.cfm?id=2839461
Testing in production.

21. Prior Art.

22. Segmentation.
total
population
experiment
group

23. Segmentation.
control
treatment

24. Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.

25. Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
interception

26. Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
segmentation

27. Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
request
flow

28. Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
telemetry

29. Traffic Control.
split. shift. shadow.

30. 1.1.21.1.1
Ingress Load Balancer

31. 1.1.21.1.1
Ingress Load Balancer

32. 1.1.21.1.1
Ingress Load Balancer

33. 1.1.2
Ingress Load Balancer

34. 1.1.2
Ingress Load Balancer
Considered
harmful!
X

40. I’m sold!
where do I sign?

41. Scheduling Substrate.
nomad. kubernetes. mesos.

42. Data Plane.
envoy. linkerd. nginx+.

46. Control Plane.
istio. linkerd. nelson.

48. Control Plane.

49. Challenges.
• How you collect results affects how you read them.
• Twyman’s Law: “Any piece of data or evidence that looks interesting
or unusual is probably wrong!”.
• Disparate data collection in many organizations.
• Instrument everything with segment identifiers.
• Modify all applications to be experiment aware.
• Such systems breed organizational change. Embrace it.

50. Conclusions.
• Experimentation infrastructure makes your workloads mobile.
• Not only does this make experimentation viable and low-impact,
but it makes your operation vastly more flexible.
• Codify and automate an experimentation process.
• You’re already doing it manually.
• Tools are already available that form the building blocks.

51. EOF
timperrett
github.com/timperrett

52. Links
• Nelson: https://verizon.github.io/nelson/
• Istio: https://istio.io/
• Research:
• Microsoft (2013): http://dl.acm.org/citation.cfm?id=2488217
• Etsy (2012): https://www.slideshare.net/danmckinley/design-for-continuous-
experimentation
• Google (2010): https://research.google.com/pubs/pub36500.html
• Microsoft (2009): http://dl.acm.org/citation.cfm?id=2339653