Voxxed Days Thesaloniki 2016 - Herding cats to a firefight
•
0 likes•98 views
Voxxed Days Thesaloniki 2016 - Herding cats to a firefight ( The evolution of an engineering on call team ) by Grace Chang
Recommended
More Related Content
Similar to Voxxed Days Thesaloniki 2016 - Herding cats to a firefight
Similar to Voxxed Days Thesaloniki 2016 - Herding cats to a firefight (8)
More from Voxxed Days Thessaloniki
More from Voxxed Days Thessaloniki (10)
Recently uploaded
Recently uploaded (20)
Voxxed Days Thesaloniki 2016 - Herding cats to a firefight
- 1. H E R D I N G C AT S T O A F I R E F I G H T T H E E V O L U T I O N O F A N E N G I N E E R I N G O N - C A L L T E A M G . C H A N G @ G R E Y S C H A L E E U R U K O 2 0 1 6
- 2. T H E Y E A R 1 B . C . ( B E F O R E C AT S )
- 3. In the beginning, there was only darkness.
- 4. But suddenly, out of the darkness, there came a sound...
- 6. One person was on-call. All day. And night. Every day. Every week. Forever.
- 7. (not really, but close enough)
- 8. Why not start having a rotation? "We don't need no stinkin' on-call rotation!"
- 9. Bullshit.
- 10. "Hi, sorry to be calling at this hour. I'm from Yammer, I work with _____. Can I please speak with him?" Date: Friday, xxth of XXX, 2013 Time: 03:00 AM GMT -0800
- 11. T H E Y E A R 1 A . D . ( A F T E R D I S A S T E R )
- 12. How to maths?!?! • Given: • Given: • Given: (1 + 15 ± 5 ) * 2 • ((4 / 1) * ((1 + 15 ± 5) * 2)) = ??? Answer:
- 14. How to acronyms?!?! M T B FM T T RA A RS L A
- 15. • MTBF: Mean Time Between Failures • MTTR: Mean Time To Recovery • SLA: Service Level Agreement • AAR: After Action Review • IR: Incident Report • OMGWTFBBQAFK
- 16. M T B F M T T R less frequent faster recovery requires more stable systems needs good response training engineers interrupted less often engineers gain broad knowledge possibly more disastrous issues possibly more frequent issues
- 18. • Google Docs Forms • Yammer Notes • JIRA ❎ (hard to read reports) ❎ (hard to analyse) ✅ (not perfect...but sort of works)
- 20. Hey, we're starting to get this!. . . . . . . . . . . .
- 21. Actually, not yet. • System grows faster than we can learn about it • Silos appear when you don't share knowledge • Who's cleaning up this mess, anyway? • Burnout is real
- 22. T H E R E N A I S S A N C E ( G R O W I N G PA I N S )
- 23. Do more by doing less • Split responsibilities by stack • Added London office for follow-the-sun coverage • Onboard everybody to the process • Practice, practice, practice
- 24. All hands on deck • Keep all alerts in a configuration repo • Managers aren't doing anything, anyway -- make them Incident Managers! • Runbooks, runbooks everywhere (and a unified one) • Make the initial response as simple as possible
- 25. B A C K T O T H E F U T U R E ( T H E P R E S E N T )
- 26. Combined schedules • Fewer rotations • Team is unified, so schedules should be too
- 27. Post-mortems and retrospectives • What? Where? Who? Why? How? • NO blame game
- 28. Weekly hand-overs and monthly reviews • Previous week engineers to current week engineers • Track top alerts and resolutions (or lack of) • Focus on the noisiest services • Timezones are hard
- 29. Bi-monthly surveys • Summarise overall preparedness • Make sure we're improving • ...and that nobody is actually burned out
- 30. Fix ALL the alerts • Noisy • Flaky • Real
- 31. W H E R E A R E T H E C AT S N O W ? !
- 32. The end game • 1 alert per person per day • Service owners are on-call for those services • The world is full of kittens!
- 33. Isn't on-call just for Ops? • No • Responsibility for our code • Pride in our code • No pain, no gain
- 34. Isn't on-call just for Ops? • No • Responsibility for our code • Pride in our code • No pain, no gain
- 35. After all... we are all cats being herded.
- 36. T H A N K Y O U @ G R E Y S C H A L E G . C H A N G @ G R E Y S C H A L E E U R U K O 2 0 1 6