Download as PDF, PPTX
![@alexsotob36
Run
Canary Release
X v1
X v2
user
90%
10%
Dark Canaries
X v1
X v2
user
*
[10.0.X.Y]
Containerise the experiment
Define Expected Behaviour
Make it public within the organisation
volume-up](https://image.slidesharecdn.com/chaosengineeringkubernetes-200131193046/75/Chaos-Engineering-Kubernetes-36-2048.jpg)
![@alexsotob49
Defining Experiment
"method": [
{
"type": "action",
"name": "terminate-db-master",
"provider": {
"type": "python",
"module": "chaosk8s.pod.actions",
"func": "terminate_pods",
"arguments": {
"label_selector": "spilo-role=master",
"name_pattern": “greetings-db-[0-9]$",
"rand": true,
"ns": "default"
}
},
"pauses": {
"after": 2
}
},](https://image.slidesharecdn.com/chaosengineeringkubernetes-200131193046/75/Chaos-Engineering-Kubernetes-49-2048.jpg)
![@alexsotob50
Verifying Experiment
{
"type": "probe",
"ref": "application-must-respond"
},
{
"type": "probe",
"name": "fetch-patroni-operator-logs",
"provider": {
"type": "python",
"module": "chaosk8s.pod.probes",
"func": "read_pod_logs",
"arguments": {
"label_selector": "name=postgres-operator",
"last": "20s",
"ns": "default"
}
}
}
],
"rollbacks": []
chaos run experiment.json](https://image.slidesharecdn.com/chaosengineeringkubernetes-200131193046/75/Chaos-Engineering-Kubernetes-50-2048.jpg)
![@alexsotob54
apiVersion: glooshot.solo.io/v1
kind: Experiment
metadata:
name: abort-ratings-metric
namespace: bookinfo
spec:
spec:
duration: 60s
failureConditions:
- trigger:
prometheus:
customQuery: |
scalar(sum(rate(istio_requests_total{ source_app=“productpage",response_code="500",
reporter="destination",destination_app="reviews",destination_version!="v1"}[1m])))
thresholdValue: 0.01
comparisonOperator: ">"
faults:
- destinationServices:
- name: ratings
namespace: app
fault:
abort:
httpStatus: 500
percentage: 100
targetMesh:
name: istio-istio-system
namespace: app
kubectl create -f experiment.yml](https://image.slidesharecdn.com/chaosengineeringkubernetes-200131193046/75/Chaos-Engineering-Kubernetes-54-2048.jpg)
![@alexsotob56
Put on your Sunday clothes
there's lots of world
out there.
— Wall-E
“
[https://github.com/lordofthejars/chaos-quarkus]](https://image.slidesharecdn.com/chaosengineeringkubernetes-200131193046/75/Chaos-Engineering-Kubernetes-56-2048.jpg)
Uploaded byAlex Soto
Chaos Engineering Kubernetes
The document discusses chaos engineering in Kubernetes led by Alex Soto from Red Hat, detailing the importance of intentionally breaking systems to understand their behavior and increase resilience. It outlines various testing methods, phases of chaos, and defines processes for running and validating chaos experiments in production environments. The talk emphasizes communication, starting small, and embracing the learning opportunities that arise from failures.
More Related Content
Similar to Chaos Engineering Kubernetes
Chaos Engineering Kubernetes
- 1. @alexsotob Chaos Engineering in Kubernetes AlexSoto Director of Developer Experience Red Hat (@alexsotob)
- 2. @alexsotob 2 Buenas tardes, buenasnoches, señoritas y señores To be here with you tonight. Brings me joy, que alegria — Miguel “
- 3.
- 4.
- 5. @alexsotob5 I think thatis the most stupidest thing I ever heard it. — Gideon Grey “
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20. @alexsotob20 This is themost beautiful miracle I’ve ever seen. — Vanellope Von Schweetz “
- 21. @alexsotob21 Failure of aService
- 22.
- 23. @alexsotob23 Production is notsacrosanct anymore
- 24. @alexsotob24 - Unit Tests -Component Tests - Static Analysis - Coverage Tests - Benchmark Tests - Contract Tests - Acceptance Tests - Mutation Tests - Smoke Tests - UI/UX Tests - Penetration Tests - Integration Tests - Tap Compare - Load Tests - Shadowing - Config Tests - Canarying - Dark Canaries - Monitoring - Feature Flagging - Exception Tracking - Feature Graduation - Teeing - Profiling - Logs - Chaos Testing - Monitoring - A/B Testing - Tracing - Auditing - OnCall Experience - Journey tests Cindy Sridharan Pre-Production Testing In Production Deploy Release Post Release The New ¿Pyramid?
- 25. @alexsotob25 There is alot of grey area in make me a prince. — Gene “
- 26.
- 27. @alexsotob27 All of thishas happened before, and it will all happen again. — Peter Pan “
- 28.
- 29. @alexsotob29 The flower thatbloom in adversity is the most rare and beautiful of all. — The Emperor “
- 30. @alexsotob30 Chaos Engineering https://principlesofchaos.org/ Break YourSystem on purpose Find out how it behaves and fix if necessary
- 31. @alexsotob31 The human world… It’sa mess. — Sebastian “
- 32. @alexsotob32 Phases of chaos SteadyState Hypothesis Run Validate Fix
- 33.
- 34.
- 35. @alexsotob35 Hypothesis What happen incase of … Service starts returning Error Codes Latency increased to 500 ms Database is not available Time travel Partially deleting Kafka topics
- 36. @alexsotob36 Run Canary Release X v1 Xv2 user 90% 10% Dark Canaries X v1 X v2 user * [10.0.X.Y] Containerise the experiment Define Expected Behaviour Make it public within the organisation volume-up
- 37. @alexsotob37 Validate Compare between currentstate and steady state System recover to steady state Identify the problems
- 38. @alexsotob38 Worked - Youare good. Escalate and Start Developing Not Blame Fix
- 39.
- 40.
- 41.
- 42. @alexsotob42 Istio — ‘Sail’ (Kubernetes— the ‘Helmsman’)
- 43. @alexsotob43 Pod Container JVM Service A Pod Container JVM Service C Pod Container JVM ServiceB Microservices Externalizing Capabilities The service mesh intercepts all network traffic
- 44. @alexsotob44 Resiliency Chaos Retries Bulkhead Circuit Breaker PoolEjection Request Timeout Fault Injection Fault Delay Istio and Chaos
- 45. @alexsotob45 Defining Failure apiVersion: networking.istio.io/v1alpha3 kind:VirtualService metadata: name: ratings ... spec: hosts: - ratings http: - fault: abort: httpStatus: 500 percentage: value: 100 match: - headers: end-user: exact: alex route: - destination: host: ratings kubectl apply -f failure.yml
- 46.
- 47.
- 48. @alexsotob48 Defining Steady State "steady-state-hypothesis":{ "title": "Services are all available and healthy", "probes": [ { "type": "probe", "name": "application-should-be-alive-and-healthy", "tolerance": true, "provider": { "type": "python", "module": "chaosk8s.probes", "func": "microservice_available_and_healthy", "arguments": { "name": “greetings-app", "ns": "default" } } },
- 49. @alexsotob49 Defining Experiment "method": [ { "type":"action", "name": "terminate-db-master", "provider": { "type": "python", "module": "chaosk8s.pod.actions", "func": "terminate_pods", "arguments": { "label_selector": "spilo-role=master", "name_pattern": “greetings-db-[0-9]$", "rand": true, "ns": "default" } }, "pauses": { "after": 2 } },
- 50. @alexsotob50 Verifying Experiment { "type": "probe", "ref":"application-must-respond" }, { "type": "probe", "name": "fetch-patroni-operator-logs", "provider": { "type": "python", "module": "chaosk8s.pod.probes", "func": "read_pod_logs", "arguments": { "label_selector": "name=postgres-operator", "last": "20s", "ns": "default" } } } ], "rollbacks": [] chaos run experiment.json
- 51. @alexsotob51 AWS { "name": "stop-an-ec2-instance-in-az-at-random", "provider": { "type":"python", "module": "chaosaws.ec2.actions", "func": "stop_instance", "arguments": { "az": "us-west-1" } } }
- 52.
- 53.
- 54. @alexsotob54 apiVersion: glooshot.solo.io/v1 kind: Experiment metadata: name:abort-ratings-metric namespace: bookinfo spec: spec: duration: 60s failureConditions: - trigger: prometheus: customQuery: | scalar(sum(rate(istio_requests_total{ source_app=“productpage",response_code="500", reporter="destination",destination_app="reviews",destination_version!="v1"}[1m]))) thresholdValue: 0.01 comparisonOperator: ">" faults: - destinationServices: - name: ratings namespace: app fault: abort: httpStatus: 500 percentage: 100 targetMesh: name: istio-istio-system namespace: app kubectl create -f experiment.yml
- 55. @alexsotob55 The way toget started is to quit talking and begin doing. — Walt Disney “
- 56. @alexsotob56 Put on yourSunday clothes there's lots of world out there. — Wall-E “ [https://github.com/lordofthejars/chaos-quarkus]
- 57. @alexsotob57 What's the lesson? Whatis the take-away? — Maui “
- 58. @alexsotob58 Every adventure requires afirst step. — Alice “
- 59. @alexsotob59 Start with themost sympathetic and innovative groups MarketGrowth Time 2.5% 13,5% 34% 34% 16% The Chasm Early Adopters Innovators Early Majority Late Majority Laggards The Technology Adoption Curve (Source: Moore and McKenna, Crossing The Chasm)
- 60.
- 61. @alexsotob61 Start small As closeas possible to production Communicate with everybody Have a failover plan Start Manual To get started
- 62. @alexsotob62 Don’t you know there’spart of me that longs to go… Into the Unknown — Elsa “
- 63.
- 64. @alexsotob64 This is thecircle of sadness. Your job is to make sure that all sadness stays inside of it. — Joy “
- 65. @alexsotob65 Oh yes, the pastcan hurt. But the way I see it, you can either run from it or learn from it. — Rafiki “
- 66. @alexsotob Hay un amigoen mí, cuando salgan a volar, hay un amigo en mí — Toy Story “ @alexsotob asotobue@redhat.com http://www.lordofthejars.com/ lordofthejars