Chaos Engineering is used in a distributed system to test integrally all the application by simulating error conditions within the system and observes how the application reacts to that stimulus. With all this information and analyzing it correctly, you can write applications more resilient to the failures. This talk will provide an introduction to the principles of Chaos Engineering, how to perform experiments, identify the weakness of the architecture and fix these problems. Come to this session to learn different tools like Istio, Chaos Toolkit or Glooshot to run Chaos Engineering in Kubernetes and what strategies you can use to prevent chaos from taking over your system.

1. @alexsotob
Chaos Engineering in
Kubernetes
Alex Soto
Director of Developer Experience Red Hat
(@alexsotob)

2. @alexsotob
2
Buenas tardes, buenas noches,
señoritas y señores
To be here with you tonight.
Brings me joy, que alegria
— Miguel
“

3. @alexsotob
3
Who Am I?
Alex Soto

4. @alexsotob 4
Questions

5. @alexsotob5
I think that is the
most stupidest thing
I ever heard it.
— Gideon Grey
“

6. @alexsotob6
MyApp
Monolith

7. @alexsotob7
Modules

8. @alexsotob8
Components

9. @alexsotob9
Microservices

10. @alexsotob10
Microservices

11. @alexsotob11
Microservices

12. @alexsotob12
Network of Services

13. @alexsotob13
This is the most beautiful
miracle I’ve ever seen.
— Vanellope Von
Schweetz
“

14. @alexsotob14
Failure of a Service

15. @alexsotob15
Cascading Failure

16. @alexsotob16
Production is not sacrosanct anymore

17. @alexsotob17
- Unit Tests
- Component Tests
- Static Analysis
- Coverage Tests
- Benchmark Tests
- Contract Tests
- Acceptance Tests
- Mutation Tests
- Smoke Tests
- UI/UX Tests
- Penetration Tests
- Integration Tests
- Tap Compare
- Load Tests
- Shadowing
- Config Tests
- Canarying
- Dark Canaries
- Monitoring
- Feature Flagging
- Exception Tracking
- Feature Graduation
- Teeing
- Profiling
- Logs
- Chaos Testing
- Monitoring
- A/B Testing
- Tracing
- Auditing
- OnCall Experience
- Journey tests
Cindy Sridharan
Pre-Production
Testing In Production
Deploy Release Post Release
The New ¿Pyramid?

18. @alexsotob18

19. @alexsotob19
All of this has happened
before,
and it will all happen
again.
— Peter Pan
“

20. @alexsotob20

21. @alexsotob21
The flower that bloom in adversity
is the most rare
and beautiful of all.
— The Emperor
“

22. @alexsotob22
Chaos Engineering
https://principlesofchaos.org/
Break Your System on purpose
Find out how it behaves and fix if necessary

23. @alexsotob23
The human world…
It’s a mess.
— Sebastian
“

24. @alexsotob24
Phases of chaos
Steady State
Hypothesis
Run
Validate
Fix

25. @alexsotob25
cogs Steady State

26. @alexsotob26
Hypothesis
What happen in case of …
Service starts returning Error Codes
Latency increased to 500 ms
Database is not available
Database is not available
Time travel
Partially deleting Kafka topics

27. @alexsotob27
Run
Canary Release
X v1
X v2
user
90%
10%
Dark Canaries
X v1
X v2
user
*
[10.0.X.Y]
Containerise the experiment
Define Expected Behaviour
Make it public within the organisation

28. @alexsotob28
Validate
Compare between current state and steady state
System recover to steady state
Identify the problems

29. @alexsotob29
Worked - You are good.
Escalate and Start Developing
Not Blame
Fix

30. @alexsotob30

31. @alexsotob31

32. @alexsotob32
Image:
quay.io/images/custservice:1.1.0
Replicas:
2
Labels:
customerservice=prod,ci_build=1213
ConfigMap:
cust_config

33. @alexsotob33

34. @alexsotob34

35. @alexsotob35
Defining Steady State"steady-state-hypothesis": {
"title": "Services are all available and healthy",
"probes": [
{
"type": "probe",
"name": "application-should-be-alive-and-healthy",
"tolerance": true,
"provider": {
"type": "python",
"module": "chaosk8s.probes",
"func": "microservice_available_and_healthy",
"arguments": {
"name": “greetings-app",
"ns": "default"
}
}
},
{
"type": "probe",
"name": "application-must-respo
"tolerance": 200,
"provider": {
"type": "http",
"verify_tls": false,
"url": “https://app.greetin
}
}
]
},

36. @alexsotob36
Defining Experiment
"method": [
{
"type": "action",
"name": "terminate-db-master",
"provider": {
"type": "python",
"module": "chaosk8s.pod.actions",
"func": "terminate_pods",
"arguments": {
"label_selector": "spilo-role=master",
"name_pattern": “greetings-db-[0-9]$",
"rand": true,
"ns": "default"
}
},
"pauses": {
"after": 2
}
},

37. @alexsotob37
Verifying Experiment{
"type": "probe",
"ref": "application-must-respond"
},
{
"type": "probe",
"name": "fetch-patroni-operator-logs",
"provider": {
"type": "python",
"module": "chaosk8s.pod.probes",
"func": "read_pod_logs",
"arguments": {
"label_selector": "name=postgres-operator",
"last": "20s",
"ns": "default"
}
}
}
],
"rollbacks": []chaos run experiment.json

38. @alexsotob38
AWS
{
"name": "stop-an-ec2-instance-in-az-at-random",
"provider": {
"type": "python",
"module": "chaosaws.ec2.actions",
"func": "stop_instance",
"arguments": {
"az": "us-west-1"
}
}
}

39. @alexsotob39

40. @alexsotob40

41. @alexsotob41
metadata:
name: abort-ratings-metric
namespace: bookinfo
spec:
spec:
duration: 60s
failureConditions:
- trigger:
prometheus:
customQuery: |
scalar(sum(rate(istio_requests_total{ source_app=“productpage",response_code="500"
reporter="destination",destination_app="reviews",destination_version!="v1"}[1
thresholdValue: 0.01
comparisonOperator: ">"
faults:
- destinationServices:
- name: ratings
namespace: app
fault:
abort:
httpStatus: 500
percentage: 100
targetMesh:
kubectl create -f experiment.yml

42. @alexsotob
It's time to see
what I can do
To test the limits
and break through.
— Elsa
“

43. @alexsotob43
Put on your Sunday clothes
there's lots of world
out there.
— Wall-E
“
[https://github.com/lordofthejars/chaos-quarkus]

44. @alexsotob44
What's the lesson?
What is the take-away?
— Maui
“

45. @alexsotob45
Every adventure
requires a first step.
— Alice
“

46. @alexsotob46
Start with the most sympathetic and innovative groups
MarketGrowth
Time
2.5% 13,5% 34% 34% 16%
The Chasm
Early
Adopters
Innovators
Early
Majority
Late
Majority
Laggards
The Technology Adoption Curve (Source: Moore and McKenna, Crossing The Chasm)

47. @alexsotob47
Start small
As close as possible to production
Communicate with everybody
Have a failover plan
Start Manual
To get started

48. @alexsotob48
This is the circle of sadness.
Your job is to make sure
that all sadness stays
inside of it.
— Joy
“

49. @alexsotob49
Oh yes,
the past can hurt.
But the way I see it,
you can either run from it
or learn from it.
— Rafiki
“

50. @alexsotob
Hay un amigo en mí,
cuando salgan a volar,
hay un amigo en mí
— Toy Story
“
@alexsotob
asotobue@redhat.com
http://www.lordofthejars.com/
lordofthejars