Download to read offline
Uploaded byDevOpsDays DFW
Detangling complex systems with compassion & production excellence
The document discusses the complexities of production systems and emphasizes the importance of evolving strategies in DevOps. It advocates for investments in people, culture, and process to achieve production excellence, which involves clear measurement, collaboration, and addressing risks systematically. The key message is that creating reliable systems requires collective effort and a focus on user needs, rather than individual heroism.
More Related Content
Similar to Detangling complex systems with compassion & production excellence
Detangling complex systems with compassion & production excellence
- 1. Detangling complex systems LizFong-Jones @lizthegrey #DevOpsDays DFW August 20, 2019 with compassion & production excellence 1w/ illustrations by @emilywithcurls!
- 2. @lizthegrey at #DevOpsDaysDFW Production is increasingly complex. 2
- 3. @lizthegrey at #DevOpsDaysDFW Especially for hybrid systems. 3
- 4. @lizthegrey at #DevOpsDaysDFW What does uptime mean? 4
- 5. @lizthegrey at #DevOpsDaysDFW Is it measured in servers? 5
- 6. @lizthegrey at #DevOpsDaysDFW Is it measured in complaints? 6
- 7. @lizthegrey at #DevOpsDaysDFW How about juggling everything else? 7
- 8. @lizthegrey at #DevOpsDaysDFW Our strategies need to evolve. 8
- 9. @lizthegrey at #DevOpsDaysDFW Don't "buy" DevOps. 9
- 10. @lizthegrey at #DevOpsDaysDFW When we order the alphabet soup... 10
- 11. @lizthegrey at #DevOpsDaysDFW Noisy alerts. Grumpy engineers. 11
- 12. @lizthegrey at #DevOpsDaysDFW Walls of meaningless dashboards. 12
- 13. @lizthegrey at #DevOpsDaysDFW Incidents take forever to fix. 13
- 14. @lizthegrey at #DevOpsDaysDFW Everyone bugs the "expert". 14
- 15. @lizthegrey at #DevOpsDaysDFW Deploys are unpredictable. 15
- 16. @lizthegrey at #DevOpsDaysDFW There's no time to do projects... 16
- 17. @lizthegrey at #DevOpsDaysDFW and when there's time, there's no plan. 17
- 18. @lizthegrey at #DevOpsDaysDFW The team is struggling to hold on. 18
- 19. @lizthegrey at #DevOpsDaysDFW What are we missing? 19
- 20. @lizthegrey at #DevOpsDaysDFW We forgot who operates systems. 20
- 21. @lizthegrey at #DevOpsDaysDFW Tools aren't magical. 21
- 22. @lizthegrey at #DevOpsDaysDFW Invest in people, culture, & process. 22
- 23. @lizthegrey at #DevOpsDaysDFW Enter the art of Production Excellence. 23
- 24. @lizthegrey at #DevOpsDaysDFW Make systems more reliable & friendly. 24
- 25. @lizthegrey at #DevOpsDaysDFW ProdEx takes planning. 25
- 26. @lizthegrey at #DevOpsDaysDFW Measure and act on what matters. 26
- 27. @lizthegrey at #DevOpsDaysDFW Involve everyone. 27
- 28. @lizthegrey at #DevOpsDaysDFW Build everyone's confidence. Encourage asking questions. 28
- 29. @lizthegrey at #DevOpsDaysDFW How do we get started? 29
- 30. @lizthegrey at #DevOpsDaysDFW Know when it's too broken. 30
- 31. @lizthegrey at #DevOpsDaysDFW & be able to debug, together when it is. 31
- 32. @lizthegrey at #DevOpsDaysDFW Eliminate (unnecessary) complexity. 32
- 33. @lizthegrey at #DevOpsDaysDFW Our systems are always failing. 33
- 34. @lizthegrey at #DevOpsDaysDFW What if we measure too broken? 34
- 35. @lizthegrey at #DevOpsDaysDFW We need Service Level Indicators 35
- 36. @lizthegrey at #DevOpsDaysDFW SLIs and SLOs are common language. 36
- 37. @lizthegrey at #DevOpsDaysDFW Think in terms of events in context. 37
- 38. @lizthegrey at #DevOpsDaysDFW Is this event good or bad? 38
- 39. @lizthegrey at #DevOpsDaysDFW Are users grumpy? Ask your PM. 39
- 40. @lizthegrey at #DevOpsDaysDFW What threshold buckets events? 40
- 41. @lizthegrey at #DevOpsDaysDFW HTTP Code 200? Latency < 300ms? 41
- 42. @lizthegrey at #DevOpsDaysDFW How many eligible events did we see? 42
- 43. @lizthegrey at #DevOpsDaysDFW Availability: Good / Eligible Events 43
- 44. @lizthegrey at #DevOpsDaysDFW Set a target Service Level Objective. 44
- 45. @lizthegrey at #DevOpsDaysDFW Use a window and target percentage. 45
- 46. @lizthegrey at #DevOpsDaysDFW 99.9% of events good in past 30 days. 46
- 47. @lizthegrey at #DevOpsDaysDFW A good SLO barely keeps users happy. 47
- 48. @lizthegrey at #DevOpsDaysDFW Drive alerting with SLOs. 48
- 49. @lizthegrey at #DevOpsDaysDFW Error budget: allowed unavailability 49
- 50. @lizthegrey at #DevOpsDaysDFW How long until I run out? 50
- 51. @lizthegrey at #DevOpsDaysDFW Page if it's hours. 51 Ticket if it's days.
- 52. @lizthegrey at #DevOpsDaysDFW Data-driven business decisions. 52
- 53. @lizthegrey at #DevOpsDaysDFW Is it safe to do this risky experiment? 53
- 54. @lizthegrey at #DevOpsDaysDFW Should we invest in more reliability? 54
- 55. @lizthegrey at #DevOpsDaysDFW Perfect SLO > Good SLO >>> No SLO 55
- 56. @lizthegrey at #DevOpsDaysDFW Measure what you can today. 56
- 57. @lizthegrey at #DevOpsDaysDFW Iterate to meet user needs. 57
- 58. @lizthegrey at #DevOpsDaysDFW Only alert on what matters. 58
- 59. @lizthegrey at #DevOpsDaysDFW SLIs & SLOs are only half the picture... 59
- 60. @lizthegrey at #DevOpsDaysDFW Our outages are never identical. 60
- 61. @lizthegrey at #DevOpsDaysDFW Failure modes can't be predicted. 61
- 62. @lizthegrey at #DevOpsDaysDFW Support debugging novel cases. In production. 62
- 63. @lizthegrey at #DevOpsDaysDFW Allow forming & testing hypotheses. 63
- 64. @lizthegrey at #DevOpsDaysDFW Dive into data to ask new questions. 64
- 65. @lizthegrey at #DevOpsDaysDFW Our services must be observable. 65
- 66. @lizthegrey at #DevOpsDaysDFW Can you examine events in context? 66
- 67. @lizthegrey at #DevOpsDaysDFW Can you explain the variance? 67
- 68. @lizthegrey at #DevOpsDaysDFW Can you mitigate impact & debug later? 68
- 69. @lizthegrey at #DevOpsDaysDFW SLOs and Observability go together. 69
- 70. @lizthegrey at #DevOpsDaysDFW But they alone don't create collaboration. 70
- 71. @lizthegrey at #DevOpsDaysDFW Heroism isn't sustainable. 71
- 72. @lizthegrey at #DevOpsDaysDFW Debugging is not a solo activity. 72
- 73. @lizthegrey at #DevOpsDaysDFW Debugging is for everyone. 73
- 74. @lizthegrey at #DevOpsDaysDFW Collaboration is interpersonal. 74
- 75. @lizthegrey at #DevOpsDaysDFW Lean on your team. 75
- 76. @lizthegrey at #DevOpsDaysDFW We learn better when we document. 76
- 77. @lizthegrey at #DevOpsDaysDFW Fix hero culture. Share knowledge. 77
- 78. @lizthegrey at #DevOpsDaysDFW Reward curiosity and teamwork. 78
- 79. @lizthegrey at #DevOpsDaysDFW Learn from the past. Reward your future self. 79
- 80. @lizthegrey at #DevOpsDaysDFW Outages don't repeat, but they rhyme. 80
- 81. @lizthegrey at #DevOpsDaysDFW Risk analysis helps us plan. 81
- 82. @lizthegrey at #DevOpsDaysDFW Quantify risks by frequency & impact. 82
- 83. @lizthegrey at #DevOpsDaysDFW Which risks are most significant? 83
- 84. @lizthegrey at #DevOpsDaysDFW Address risks that threaten the SLO. 84
- 85. @lizthegrey at #DevOpsDaysDFW Make the business case to fix them. 85
- 86. @lizthegrey at #DevOpsDaysDFW And prioritize completing the work. 86
- 87. @lizthegrey at #DevOpsDaysDFW Don't waste time chrome polishing. 87
- 88. @lizthegrey at #DevOpsDaysDFW Lack of observability is systemic risk. 88
- 89. @lizthegrey at #DevOpsDaysDFW So is lack of collaboration. 89
- 90. @lizthegrey at #DevOpsDaysDFW 90 Success doesn't demand heroism.
- 91. @lizthegrey at #DevOpsDaysDFW Season the alphabet soup with ProdEx 91
- 92. @lizthegrey at #DevOpsDaysDFW Production Excellence brings teams closer together. Measure. Debug. Collaborate. Fix. 92 lizthegrey.com; @lizthegrey