Using a service mesh based on the Envoy proxy to inspect and tune customer experience under steady state and injected faults.

1. © Turbine Labs. Confidential and proprietary.
Chaos Proxies
1

2. © Turbine Labs. Confidential and proprietary.
Usual Engineering Loop
2
Define Desired Behavior
Observe System
System is Destabilized
Firefight
Resolve
Baseline
Incident
Planet of the Crepes - @shinypb

3. © Turbine Labs. Confidential and proprietary.
Chaos Engineering Loop
3
Define Desired Behavior
Observe System
Inject Fault
Analyze System
Tune System
Baseline
Hardening
Planet of the Crepes - @shinypb

4. © Turbine Labs. Confidential and proprietary.
Service Meshes and Chaos
4
•The more service oriented you are, the more central the network
becomes in your fault model.
•The bad news: networks are notoriously chao@c.
•The good news: the service mesh lets you observe behavior and
quickly apply policy changes.
Food truck name idea: Calamari Damacy - @shinypb

5. © Turbine Labs. Confidential and proprietary.
Envoy and the Loop
5
•Envoy as the data plane in a service mesh gives you a great set
of capabili'es.
• Global observability
• Global resilience policies
• Fault injec@on
•The control plane gives you faster feedback loops.
•You s@ll need to build prac'ces on top of these primi@ves.
A post apocalyp@c bakery called Judge Bredd - @mccv

6. © Turbine Labs. Confidential and proprietary.
Choose Your Fighter
6
If a request
exceeds a .me
limit, stop and
return an error.
Timeout
On an error, try
again.
Retry
If too many
requests are in
progress, stop
and return an
error.
Break
If an endpoint
is unhealthy,
don’t use it.
Eject
A falafel truck called Kebab Kebarker - @9len

7. Dealing With Failure
I'd like to open up a food truck called Pasta La Vista. - @shinypb

8. © Turbine Labs. Confidential and proprietary.
Dealing With Failure - Ejection
8Korma Police - @shinypb
Simplicity Scalability Fidelity
Health Checks Very Simple Low Low
Outlier Detection Not Simple High High
Serivce Discovery ? ? ?

9. © Turbine Labs. Confidential and proprietary.
Dealing With Failure - Ejection
9A food truck called Move Fast and Bake Things - @mccv

10. © Turbine Labs. Confidential and proprietary.
On the Exponential Benefits of Retries
10
A food truck converted from one of those tac@cal swat trucks
called Meal Team 6 - @mccv
90.0%
92.5%
95.0%
97.5%
100.0%
2 Tries 3 Tries 4 Tries
99%
95%
90%
85%
80%
75%
70%
Base Success Rate

11. © Turbine Labs. Confidential and proprietary.
On the Exponential Risks of Failure
11
0.0%
25.0%
50.0%
75.0%
100.0%
2 3 4 5 6 7
99%
95%
90%
85%
80%
75%
70%
Base Success Rate
Process Steps
Food truck name idea: Entrée the Giant - @shinypb

12. © Turbine Labs. Confidential and proprietary.
Dealing With Failure - Retries
12Did we already do a breakfast truck called Cereal Bus? - @mccv

13. Dealing With Slow
A fancy cheese shop called float like a camembert s@nk like a brie - @9len

14. © Turbine Labs. Confidential and proprietary.
A Typical Latency Curve
14A cinnamon-raisin bread truck called Fruity and the Yeast - @9len

15. © Turbine Labs. Confidential and proprietary.
Fixing the Latency Curve
15
•Add @meouts to abandon long running requests.
• Timeouts trade success rate for speed.
•Add retries to recover success rate.
•Warnings:
• This won’t get you to 100% success rate.
• This increases overall load on your system.
A medieval-themed hamburger truck called “Knight Slider” - @9len

16. © Turbine Labs. Confidential and proprietary.
Fixing the Latency Curve
16A bakery truck called “Yeast Common Denominator” - @mccv

17. Beyond Reactive
food truck idea: serves hand-craged sog drinks named "Helter Seltzer" - @slyphon

18. © Turbine Labs. Confidential and proprietary.
Steady State is a Myth
18
•We’ve tuned to provide good service in good condi@ons
•Bad things happen to good networks
•If we don’t tune for degrada@on, service will suffer
tracked vehicle food truck named BlintzKrieg - @mccv

19. © Turbine Labs. Confidential and proprietary.
Running an Example Exercise
19
•The scenario
• Inject packet loss.
• Analyze customer
experience.
• Tune Envoy to restore
customer experience.
A punk rock food truck called sid's dishes - @mccv
•The ergonomics
• Low risk fault injec@on
• Deep visibility into system
behavior
• Fast feedback loop for system
tuning

20. © Turbine Labs. Confidential and proprietary.
Our Stack
20
Define Desired Behavior
Observe System
Inject Fault
Analyze System
Tune System
Baseline
Hardening
Food truck name idea: Slaw & Order - @shinypb

21. © Turbine Labs. Confidential and proprietary. 21
Inject
Much Ado About Muffins - @RSGAT

22. © Turbine Labs. Confidential and proprietary. 22
Inject
Food truck name idea: Vegan Be Heroes (Just For One Day) - @shinypb

23. © Turbine Labs. Confidential and proprietary.
Attacking and Hardening
23Korma Police - @shinypb
Before/After Graphs

24. © Turbine Labs. Confidential and proprietary. 24
Analyze
Fantas@c Beets and Where to Find Them - @shinypb

25. © Turbine Labs. Confidential and proprietary. 25
Tune
Pastry food truck name idea: The Filling Is Mutual - @shinypb

26. © Turbine Labs. Confidential and proprietary. 26
Observe
Shwarmageddon - @shinypb
Attack 1 Attack 2
Tune Policies

27. © Turbine Labs. Confidential and proprietary.
Thank you!
27
Me:
mark@turbinelabs.io
twirer.com/mccv
Company:
turbinelabs.io
twirer.com/goturbine
Envoy project:
envoyproxy.github.io
blog.envoyproxy.io/
Lessons from Envoy at scale:
learnenvoy.io
(reply to a DND-themed food truck thread) - “THAC0 Truck” - @zuecrher