Instead of randomly injecting faults ( i.e. Chaos Monkey), what if we could order our experiments to perform min number of experiments for maximum yield? We present a solution(& results) to the problem of experiment selection using Lineage Driven Fault Injection to reduce the search space of faults. Lineage Driven Fault Injection (LDFI) is a state of the art technique in chaos engineering experiment selection. LDFI since its inception has used an SAT solver under the hood which presents solutions to the decision problem (which faults to inject) in no particular order. As SRE’s we would like to perform experiments that reveal the bugs that the customers are most likely to hit first. In this talk, we present new improvements to LDFI that orders the experiment suggestions. In the first the half of the talk we will show LDFI is a technique that can be widely used within an enterprise. We present the motivation for ordering the chaos experiments along with some prioritization we utilized while conducting the experiments. We also highlight how ordering is a general purpose technique that we can use to encode the peculiarities of a heterogeneous microservices architecture. LDFI can work in an enterprise by harnessing the observability infrastructure to model the redundancy of the system. Next, we present experiments conducted within our organization using ordered LDFI and some preliminary results. We show examples of services where we discovered bugs, and how carefully controlling the order of experiments allowed LDFI to avoid running unnecessary experiments. We also present an example of an application where we declared the service shippable under crash stop model. We also present a comparison with Chaos Monkey and show how LDFI found the known bugs in a given application using orders of magnitude fewer experiments than a random fault injection tool like Chaos Monkey. Finally, we discuss how we plan to take LDFI forward. We discuss open problems and possible solutions for scalarizing probabilities of failure, latency injection, integration with service mesh technologies like envoy for fine-grained fault injection, fault injection for stateful systems. Key takeaways: 1) Understand how LDFI can be integrated in the enterprise by harnessing the observability infrastructure. 2) Limitations of LDFI w.r.t unordered solutions and why ordering matters for chaos engineering experiments. 3) Preliminary results of prioritized LDFI and a future direction for the community.

1. Madaari
Ordering For The Monkeys

2. 2

3. Agenda
● Distributed Systems and Chaos Engineering : State Of The Union
● Lineage Driven Fault Injection : A Brief Primer
● LDFI : Ordering Of Faults
● Bringing LDFI to the Enterprise
● Results
● Future Work
3

4. Industry + Academia = Win !!
Joint work between eBay and Disorderly Labs
● Dr. Peter Alvaro ( UCSC )
● Kamala Ramasubramanian ( UCSC )
● eBay SRE Team
Madaari : a trainer who teaches a monkey to perform tricks
4

5. The Problem : Testing Distributed
Systems
Combinatorial Space of FailuresMicroservices Death Star
Consider 100 Services
Fault Search Space : 2100
5
Fault
Cardinality
Possible
Faults
1 100
4 3 Million

6. Chaos Engineering : A Possible Solution
● Failure is inevitable, let’s fail in a controlled environment
● Proactively inject failure in your system to reveal weaknesses
● Perturbation and observation of large-scale systems
6

7. Chaos Engineering : A Brief Primer
Doesn’t
scale well !!
7
A genius holds the
mental model of the
system
Guided Fault Injection
No Model Of The
System
Random Fault
Injection
Can’t quantify
progress

8. Lineage Driven Fault Injection aka LDFI
CLAIM : Fault Tolerance = Redundancy
● Use explanations of successful outcomes to search for faults that can drive the system
into a bad state
● Observing successful executions enables LDFI to build a model of the redundancy of the
system
8

9. Lineage Driven Fault Injection aka LDFI
Why did a good thing happen?
Consider its lineage.
9
What could have gone wrong?
Faults are cuts in the lineage graph.
Is there a cut that breaks all supports?

10. Lineage Driven Fault Injection aka LDFI
(RepA OR Bcast1)
10
AND (RepA OR Bcast2)
AND (RepB OR Bcast2)
AND (RepB OR Bcast1)

11. Lineage Driven Fault Injection aka LDFI
(RepA OR Bcast1)
AND (RepA OR Bcast2)
AND (RepB OR Bcast2)
AND (RepB OR Bcast1)
Hypothesis: {Bcast1, Bcast2}
11

12. LDFI : Building Blocks
● Witnessing a large number of successful
executions allows LDFI to build a model
of redundancy of the system
● How? Because it can reason about why
faults were tolerated
12

13. LDFI : Building Blocks
Recipe:
1. Start with a successful outcome. Work
backwards.
2. Ask why it happened ? Ans. Lineage (Traces)
3. Convert lineage to a CNF formula and solve
the decision problem ( using a SAT solver )
4. Lather, rinse, repeat
13

14. Encoding the Lineage
(A v B v C v D v E)
14
A
B
C
ED
(A v C v D v E)
(A v B v C v D v E) ^ (A v C v D v E)
A
C
D E
B

15. Injecting Faults That Matter
● Drawbacks of existing approach
○ LDFI (using SAT) reduces the search space but the search space might still be still
large
○ LDFI is a decision problem, solutions are returned in no particular order
● We want to order solutions (run experiments) to:
○ Find the most likely faults before users do!
○ Reduce the search space as much as possible
15

16. Ordering Faults : Injecting Faults That
Matter
16
LDFI assumes all faults are equally likely,
the reality differs !!
Intuition : Some faults are more likely than
others; incident history usually backs this
claim
We want to encode our intuition of failure
in LDFI
A
B
C
ED
F

17. Ordering Of Faults
(A ∨ B ∨ C ) ∧ (C ∨ D ∨ E ∨ F) ∧ (D ∨ E ∨ F ∨ G)
∧ (H ∨ I)
(A, B, C), (C, D, E, F), (D, E, F, G), (H, I)
17

18. Ordering Of Faults : Minimal Hitting Set
(A ∨ B ∨ C ) ∧ (C ∨ D ∨ E ∨ F) ∧ (D ∨ E ∨ F ∨ G) ∧ (H ∨ I)
(A, B, C), (C, D, E, F), (D, E, F, G), (H, I)
18
e.g (C,E,H)

19. Ordering Of Faults : Minimal Hitting Set
(A ∨ B ∨ C ) ∧ (C ∨ D ∨ E ∨ F) ∧ (D ∨ E ∨ F )
Maximise: XAlog(PA) + XBlog(PB) + XClog(PC) + XDlog(PD) + XElog(PE) +
XFlog(PF)
Subject to:
XA + XB + XC >= 1
XC + XD + XE + XF >= 1
XD + XE + XF >= 1 19

20. Ordering Faults : Injecting Faults That
Matter
20
A
B
Use the structure of the Trace to prune the Solution Space :
1. Rank Of the Service ( distance from the root )
2. Size Of the sub graph of the Service
3. If we survive the failure of C, we will surely survive the failure
of D, E and F
A
B
C
ED
F

21. Ordering Faults : Injecting Faults That
Matter
● All services are not created equal, some services fail more than others
● Likelihood and Containment :
○ P(Node failure) > P(Rack Failure) >> P(Data center failure)
● Historical measures :
○ Time since last release
○ History Of Failure and Bug Rate
21

22. LDFI in the Enterprise
Explanations
Models Of
Redundancy
Fault Injection
22

23. Traces = Explanations
● Distributed Tracing
○ Call graphs come for free
● Less Ideal (but OK) : Structured
Logging
○ We did this too !!
23
What are traces anyway ?
○ Ordered Events with context
stitched together
○ Create the call graphs using
service names and endpoints

24. Fault Injection Tool
● We rolled our own ( Mowgli )
○ Inspired by Trogdor ( Kafka’s FIT
Tool)
○ Circuit breaker aware fault injection
tool, deals with services and
databases
○ Built in safety mechanisms
○ Hooks for AZ level, node level fault
injection
○ Audit and Tracking capabilities
24
● Lots of open source options available
○ Start simple, a script to drop
network traffic is also OK
○ https://github.com/dastergon/awes
ome-chaos-engineering
● Tip : Be safe by default
○ Always have a rollback strategy

25. Interaction Replay
● Ability to replay interactions ( Tip : E2E Tests )
● Measure of Success
○ A unique binary (yes or no ) way of saying whether the execution was successful or not
● Works for Eventually Consistent systems as well, as long as there is finite
upper bound on the eventuality
25

26. LDFI in the Enterprise
Traces/Structured
Logs LDFI FIT Tool
To Call
Graphs
Encode For
The Solver Fault
Suggestion
● PyCoSAT
● PULP
● SAT4J
26

27. Results : Finding Bugs
27

28. Comparison With Chaos Monkey
28
Strategy Fault Experiment Runs
(avg.)
Standard Deviation
Ordered LDFI 17 0
Uniform Random 210.35 111.42
How long did it take to find those 5 bugs? A few hours
(An experiment takes ~2 minute, and we did retries to get around our infrastructure)

29. Results : Finding No Bugs
29

30. Madaari : The Road Ahead
● Scalarizing Probabilities of Failure
● SLA verification using strategic Delay Injection
● Reason about Stateful systems
● Fine Grained Fault Injection
● Microservices Only ?
○ Databases, Containers, Service Mesh .. Let’s Go !!
30

31. LDFI : The Road Ahead
3 W’s For Fault Injection
1. What to inject ? ( type of fault we want to inject )
2. Where to inject ? (the target component )
3. When to inject ? ( inject when there are exactly 5 items in the cart !! )
31

32. LDFI : The Road Ahead
A Journey from Time to State and back
1. What’s time anyway ??
2. Applications have state and change of state gives you implicit order.
3. A rendezvous of state and time gives us precision for fault injection.
32

33. Madaari : Key Takeaways
● Industry and Academia can work together for fun(d) and profit
● Limitations of LDFI w.r.t unordered solutions and why ordering matters for
chaos engineering experiments
● Understand how LDFI can be integrated in the enterprise by harnessing the
observability infrastructure
● Preliminary results of prioritized LDFI and a future direction for the community
● Evangelising new techniques is hard; start small and stay simple
33

34. Discussion
34
Reach us at :
@ashutoshraina
@palvaro
@KamalRamas
https://disorderlylabs.github.io