Addo reducing trauma in organizations with SLOs and chaos engineering
•
0 likes•206 views
This document discusses establishing service level objectives (SLOs) and indicators (SLIs) to quantify user experience and prioritize work. It recommends using chaos engineering to validate SLOs and dependencies by injecting failures. Key points: - SLOs quantify goals for SLIs to measure user experience quality like load times and errors - Error budgets set thresholds for acceptable failures to meet SLOs - Chaos engineering tests new features and validates SLOs and dependencies by inducing failures - Incidents provide opportunities to revisit SLOs and prioritize work to improve experience
More Related Content
What's hot
What's hot (20)
Similar to Addo reducing trauma in organizations with SLOs and chaos engineering
Similar to Addo reducing trauma in organizations with SLOs and chaos engineering (20)
More from Mandi Walls
More from Mandi Walls (20)
Recently uploaded
Recently uploaded (20)
Addo reducing trauma in organizations with SLOs and chaos engineering
- 1. Proprietary & Confidential Reducing Trauma in Organizations with SLOs and Chaos Engineering Mandi Walls DevOps Advocate PagerDuty @Lnxchk Julie Gunderson Sr. Reliability Advocate Gremlin @julie_gund
- 2. Introduction Centering user experience is key to prioritizing what will best serve the users and increase engagement. We do this with qualitative practices like Full Service Ownership and quantitative practices like SLIs and SLOs.
- 3. Measuring the Cost of Downtime Cost = R + E + C + ( B + A ) During the Outage R = Revenue Lost E = Employee Productivity After the Outage C = Customer Chargebacks (SLA Breaches) Unquantifiable B = Brand Defamation A = Employee Attrition Amazon is estimated to lose $13.22MM/hour or $220,000/min Your company? Average is $300,000/hour
- 4. What are SLAs, SLIs, and SLOs?
- 5. SLIs and SLOs Indicators - our metrics Objectives - our goals for those metrics
- 7. Product Development Capacity Planning Testing & Release Procedures Post-incident Analysis Incident Response Monitoring & Observability E.g. SLOs & SLIs E.g. Blam eless Postm ortem s E.g. Canary Deploym ents E.g. Error Budgets
- 9. Proprietary & Confidential The Users are the Point What is important to the users? How do you know? Is it different for different parts of the application?
- 10. Focus on What Users Care About News Stream Loads Fast on Scroll What else? No Missing Images Center Module Loads First No Errors on Main Page Fast Load Time
- 11. Translate User Experience to Useful Metrics Photo by Luke Chesser on Unsplash
- 12. How do we know if our SLO/SLIs are working as expected? ?
- 13. We inject failure proactively to validate SLOs/SLIs. ✓
- 14. Prerequisites for setting SLIs and SLOs
- 15. Telemetry • Monitoring - keep track of what you know • Logging - scan for errors after the fact • Tracing - follow the user through the service ecosystem • Observability - the baseline characteristic Photo by Mikail McVerry on Unsplash
- 16. Dependency Mapping Success of your SLOs depends on the SLOs of your service upstream dependencies
- 17. Creating SLIs, SLOs, and Error Budgets
- 18. [SLI][SLO][t] SLI = (good/valid) * 100 eb = 100 - SLI
- 19. Example Valid Events Good Events SLO Error Budget Allowable Bad Events 100 99 99% 1% 1 1000 999 99.9% 0.1% 1 1000 990 99% 1% 10 10,000 9999 99.99% 0.01% 1 100,000 99,000 99% 1% 1000 100,000 99,999 99.999% 0.001% 1
- 20. Perfect: 100% of web requests have 0ms latency all the time! 30
- 21. Perfect: 100% of web requests have 0ms latency all the time! SLA: 90% of web requests have latency <500ms for the month… or customer gets money back. 31
- 22. Perfect: 100% of web requests have 0ms latency all the time! SLA: 90% of web requests have latency <500ms for the month… or customer gets money back. SLO: 95% of web requests have latency <500ms over a rolling month. 32
- 23. Perfect: 100% of web requests have 0ms latency all the time! SLA: 90% of web requests have latency <500ms for the month… or customer gets money back. SLO: 95% of web requests have latency <500ms over a rolling month. SLI: web requests latency <500ms 33
- 24. Instance Downtime Occurs Datadog picks up that instance is down for SLI calculation (metric), auto tracks SLO is now impacted (monitor) PagerDuty fires an alert that the uptime SLO has been breached Gremlin Downtime SLO Scenario is run on staging
- 25. Revising and Revisiting Photo by Jonathan Kemper on Unsplash These are internal tools! You can change them if they no longer work for you!
- 26. Use Chaos Engineering to test out new features and focus on your SLOs 40
- 28. Working with Upstream Dependencies Do your dependencies publish their own SLOs? Can you defensively code around bad performance? Do you need to explore alternatives?
- 29. Use Chaos Engineering to validate your dependencies ?
- 30. Unplanned Work Incidents can indicate that work needs to be done Your SLOs and error budgets are part of your postmortem discussion Revisit and prioritize work based on the outcomes of a major incident
- 31. Lifecycle • Research user behavior • Measure and monitor for reliability and performance • Set goals, write SLIs, establish SLOs • Work to keep SLOs in the green • Verify SLOs and error budgets in incident post mortems • Adjust SLOs to new business requirements
- 32. Summary SLIs prioritize the User Experience SLOs quantify “good” vs “bad” experience to a quantitative goal Error Budgets tell your team where you stand They all feed back into the work prioritization process You can change them when they no longer work for you
- 33. Resources Talks at SLOConf: https://www.sloconf.com/talks Google’s SRE Books are available online: https://sre.google/books/ Implementing SLOs: https://www.oreilly.com/library/view/implementing-service-level/9781492076803/ Gremlin Free: https://www.gremlin.com/buttons/ Sign up for a PagerDuty trial at https://pagerduty.com/sign-up Gremlin Certified Chaos Engineering Professional certification: https://www.gremlin.com/certification
- 34. Proprietary & Confidential Julie’s random slides
- 35. Moving to the cloud Verify host failure, autoscaling rules, and memory. Migrating to microservices Validate that each new service can fail independently. Protect against cascading failures and knock-on effects. Adopting Kubernetes The devil is the in details. Have you configured everything correctly? Are you running one large cluster? Find your monitoring gaps, reduce signal to noise “We’ll get paged if that breaks”, until you don’t. A false sense of security is worse than nothing. Train your teams We run fire drills, train firefights, and first responders. Are you investing in your operations teams? 55