Downloaded 10 times
More Related Content
Chaos engineering intro
- 1.
- 2.
- 3. What is Gremlin? ›Practitioners of Chaos Engineering › Builds software that helps organizations build resilient systems › Offers 11 ways to inject chaos for Chaos Engineering experiments
- 4. Agenda › Welcome › ResilientSystems › Introduction to Chaos Engineering › Chaos Engineering at other companies › Chaos Engineering with Pumba & Pod-Reaper › Demo
- 5. HELLO! I am ShantanuDeshpande You can find me at: Twitter: @ishantanu_ LinkedIn: shantanud10 Website: inherentlychaotic.me
- 6.
- 7. 1. What is aResilient System?
- 8. Resilient System › Highlyavailable and durable system › Can maintain acceptable level of service in the face of failure › Can weather the storm
- 10. 2. How do wemeasure System’s Resiliency?
- 11. Measuring System’s Resiliency ›Mean Time to Failure (MTTF) › Mean Time to Recovery (MTTR) › Mean Time between Failure (MTBF)
- 12. 3. How do wecreate More Resilient Systems?
- 13.
- 14. “ Chaos Engineering isthe discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production. ” What is Chaos Engineering?
- 15. “ Inject something harmful,in order to build an immunity “
- 16.
- 17. Prerequisites › High SeverityIncident Management › Monitoring › Measure the Impact of Downtime
- 18. 4. Prerequisite #1: High SeverityIncident Management
- 19. “ Practice of recording,triaging, tracking and assigning business value to problems that impact critical systems “ - gremlin High SEV incident management blog post
- 20.
- 21. What are SEVs? SEVLevel Description Target Resolution Time Who is notified SEV 0 Catastrophic service impact Resolve within 15 min Entire Company SEV 1 Critical Service Impact Resolve within 8 hours Teams working on SEV and CTO SEV 2 High Service Impact Resolve within 24 hours Teams working on SEV
- 22.
- 23. Why Monitor? › Analyzinglong-term trends › Comparing over time or experiment groups › Alerting › Building dashboards › Conducting ad hoc retrospective analysis
- 24.
- 25. The Four GoldenSignals - The Google SRE Book Monitoring Signal Description Target Resolution Time Latency The time taken to serve a request HTTP 500 error triggered due to loss of DB connection Traffic A measure of how much demand is being placed on your system HTTP requests per second Errors Rate of requests that fails either implicitly or explicitly Catching HTTP 500s at LB Saturation How full your service is. Should also signal impending saturation It looks like you DB will fill it’s hard drive in 4 hours.
- 26. 4. Prerequisite #3: Measure theImpact of Downtime
- 27. Measure the impactof downtime System Impact: › Availability › Durability Customer Impact › Outcome › Cost › Time
- 28. 5. Getting Started with ChaosEngineering
- 29. Getting Started › Builda hypothesis around steady state behavior › Vary real-world events › Run experiments in production › Automate experiments to run continuously › Minimize blast radius.
- 30.
- 31.
- 32. Chaos at Netflix -Used Simian army to keep the cloud safe, secure and highly available. - Chaos Monkey - Janitor Monkey - Conformity Monkey
- 33. Chaos at Google -Running DRTs from many years.
- 34. Chaos at LinkedIn -Project Waterbear which provides “application resilience” as a service - Application Failure (LinkedOut) - Infrastructure Failure (FireDrill)
- 35. 6. Chaos Engineering with Pumba& pod-reaper
- 36. What is Pumba? -Well-know supporting character (warthog) from Disney’s animated film The lion king - Pumba means: to be foolish, silly, weak-minded, careless, negligent - An open-source chaos testing tool for Docker containers
- 37. What can Pumbado? - Can disturb docker runtime environment by injecting failures - The victim container can be specified, providing name(s)/regex. - Random selection supported (--random) - Can disturb either single docker host, swarm cluster, and Kubernetes cluster
- 38. What is Pod-Reaper? -Designed to kill pods that meet specific conditions
- 39. Time to breaksome containers
- 40.