Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Experimentation with 

Immutable Infrastructure.
Timothy Perrett @ Hashiconf 2017
Hello.
!
Immutable Infrastructure.
thou shall not change thy deployment!
Converged Infrastructure.
datacenter resources are finite.
Converged Infrastructure.
staging production
static provisioning
Converged Infrastructure.
machine
application
datacenter
operations
infrastructure
product
Converged Infrastructure.
machine
scheduler
container
mesh
application
datacenter
operations
infrastructure
product
Converged Infrastructure.
resources
scheduling substrate
workloads
Workloads.
• Users demand:
• Blazing fast iteration.
• High degree of observability.
• Seamless traffic migration between a...
Workloads.
• Users demand:
• Blazing fast iteration.
• High degree of observability.
• Seamless traffic migration between a...
Ex·per·i·men·ta·tion
Process of performing a scientific procedure to determine something.
Ex·per·i·men·ta·tion
Process of performing a scientific procedure to determine something.
Testing in production
Testing in Production.
everyone does it. nobody wants to own up to it.
Emergent Behaviors.
observation of unpredictable, non-deterministic runtime artifacts.
Testing in production.
– Conway’s Law
“Any organization that designs a system will produce a design
whose structure is a copy of the organization...
– Hyrum’s Law
“With a sufficient number of users of an interface, 

all observable behaviors of your interface will 

be de...
Microservice Complexity.
local containerized testing is a fable.
Testing in production.
Inconceivable Scale.
black friday. super bowl. popular cat memes.
Testing in production.
Human Error.
embrace the inevitable.
http://queue.acm.org/detail.cfm?id=2839461
Testing in production.
Prior Art.
Segmentation.
total

population
experiment
group
Segmentation.
control
treatment
Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
interception
Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
segmentation
Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
request
flow
Edge
Experiment
Manager
Assignment
Foo
Bar Qux
Design.
telemetry
Traffic Control.
split. shift. shadow.
1.1.21.1.1
Ingress Load Balancer
1.1.21.1.1
Ingress Load Balancer
1.1.21.1.1
Ingress Load Balancer
1.1.2
Ingress Load Balancer
1.1.2
Ingress Load Balancer
Considered

harmful!
X
I’m sold!
where do I sign?
Scheduling Substrate.
nomad. kubernetes. mesos.
Data Plane.
envoy. linkerd. nginx+.
Control Plane.
istio. linkerd. nelson.
Control Plane.
Challenges.
• How you collect results affects how you read them.
• Twyman’s Law: “Any piece of data or evidence that looks...
Conclusions.
• Experimentation infrastructure makes your workloads mobile.
• Not only does this make experimentation viabl...
EOF
timperrett
github.com/timperrett
Links
• Nelson: https://verizon.github.io/nelson/
• Istio: https://istio.io/
• Research:
• Microsoft (2013): http://dl.acm...
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Online Experimentation with Immutable Infrastructure
Upcoming SlideShare
Loading in …5
×

Online Experimentation with Immutable Infrastructure

1,178 views

Published on

Immutable infrastructure has changed the way we think about system lifecycle: compute machines live for days instead of months or years, and applications live for hours or less. With the proliferation of CI/CD systems, and infrastructure as a service, the increased churn in production systems has hastened the immediate need for tools that prioritize experimentation - is your next development iteration really better than the last? In such a volatile world, traditional notions of compute “environments” and mutable approaches to experimentation can be found lacking. In large systems, emergent behaviors are near impossible to replicate in isolation, so experimenting in production systems is the only way to effectively measure hypothesis. This session covers different schemes for experimentation and the primitives required to make converged infrastructure work for real systems.

Published in: Technology
  • Be the first to comment

Online Experimentation with Immutable Infrastructure

  1. 1. Experimentation with 
 Immutable Infrastructure. Timothy Perrett @ Hashiconf 2017
  2. 2. Hello. !
  3. 3. Immutable Infrastructure. thou shall not change thy deployment!
  4. 4. Converged Infrastructure. datacenter resources are finite.
  5. 5. Converged Infrastructure. staging production static provisioning
  6. 6. Converged Infrastructure. machine application datacenter operations infrastructure product
  7. 7. Converged Infrastructure. machine scheduler container mesh application datacenter operations infrastructure product
  8. 8. Converged Infrastructure. resources scheduling substrate workloads
  9. 9. Workloads. • Users demand: • Blazing fast iteration. • High degree of observability. • Seamless traffic migration between application revisions. • Automated application cleanup. • Self-service DevOps experience.
  10. 10. Workloads. • Users demand: • Blazing fast iteration. • High degree of observability. • Seamless traffic migration between application revisions. • Automated application cleanup. • Self-service DevOps experience. 2017 minimum
 table stakes!
  11. 11. Ex·per·i·men·ta·tion Process of performing a scientific procedure to determine something.
  12. 12. Ex·per·i·men·ta·tion Process of performing a scientific procedure to determine something. Testing in production
  13. 13. Testing in Production. everyone does it. nobody wants to own up to it.
  14. 14. Emergent Behaviors. observation of unpredictable, non-deterministic runtime artifacts. Testing in production.
  15. 15. – Conway’s Law “Any organization that designs a system will produce a design whose structure is a copy of the organization's 
 communication structure”
  16. 16. – Hyrum’s Law “With a sufficient number of users of an interface, 
 all observable behaviors of your interface will 
 be depended upon by somebody.”
  17. 17. Microservice Complexity. local containerized testing is a fable. Testing in production.
  18. 18. Inconceivable Scale. black friday. super bowl. popular cat memes. Testing in production.
  19. 19. Human Error. embrace the inevitable. http://queue.acm.org/detail.cfm?id=2839461 Testing in production.
  20. 20. Prior Art.
  21. 21. Segmentation. total
 population experiment group
  22. 22. Segmentation. control treatment
  23. 23. Edge Experiment Manager Assignment Foo Bar Qux Design.
  24. 24. Edge Experiment Manager Assignment Foo Bar Qux Design. interception
  25. 25. Edge Experiment Manager Assignment Foo Bar Qux Design. segmentation
  26. 26. Edge Experiment Manager Assignment Foo Bar Qux Design. request flow
  27. 27. Edge Experiment Manager Assignment Foo Bar Qux Design. telemetry
  28. 28. Traffic Control. split. shift. shadow.
  29. 29. 1.1.21.1.1 Ingress Load Balancer
  30. 30. 1.1.21.1.1 Ingress Load Balancer
  31. 31. 1.1.21.1.1 Ingress Load Balancer
  32. 32. 1.1.2 Ingress Load Balancer
  33. 33. 1.1.2 Ingress Load Balancer Considered
 harmful! X
  34. 34. I’m sold! where do I sign?
  35. 35. Scheduling Substrate. nomad. kubernetes. mesos.
  36. 36. Data Plane. envoy. linkerd. nginx+.
  37. 37. Control Plane. istio. linkerd. nelson.
  38. 38. Control Plane.
  39. 39. Challenges. • How you collect results affects how you read them. • Twyman’s Law: “Any piece of data or evidence that looks interesting or unusual is probably wrong!”. • Disparate data collection in many organizations. • Instrument everything with segment identifiers. • Modify all applications to be experiment aware. • Such systems breed organizational change. Embrace it.
  40. 40. Conclusions. • Experimentation infrastructure makes your workloads mobile. • Not only does this make experimentation viable and low-impact, but it makes your operation vastly more flexible. • Codify and automate an experimentation process. • You’re already doing it manually. • Tools are already available that form the building blocks.
  41. 41. EOF timperrett github.com/timperrett
  42. 42. Links • Nelson: https://verizon.github.io/nelson/ • Istio: https://istio.io/ • Research: • Microsoft (2013): http://dl.acm.org/citation.cfm?id=2488217 • Etsy (2012): https://www.slideshare.net/danmckinley/design-for-continuous- experimentation • Google (2010): https://research.google.com/pubs/pub36500.html • Microsoft (2009): http://dl.acm.org/citation.cfm?id=2339653

×