Recommended
Recommended
More Related Content
What's hot
What's hot (10)
Similar to SREcon 2017 From Firefighting to Proactive Work: the Journey of a Small Infrastructure Team in a Hyper Growth Environment
Similar to SREcon 2017 From Firefighting to Proactive Work: the Journey of a Small Infrastructure Team in a Hyper Growth Environment (20)
Recently uploaded
Recently uploaded (20)
SREcon 2017 From Firefighting to Proactive Work: the Journey of a Small Infrastructure Team in a Hyper Growth Environment
- 1. From firefighting to proactive work The journey of a small SRE team in a hyper growth environment
- 6. •Multiple downtimes •CPU usage hitting 80% regularly •Commit latency trending up •66 % increase in query volume in 6 months Things started getting worse
- 7. SQL Fingerprinting Query Service Line of code Responsible Team
- 9. •High number of pages in/out office hours •Multiple downtimes because of our main datastore •No obvious way to mitigate Tipping point
- 11. The safety of our team and our availability were at risk
- 13. Work mode Database Health PreventiveFirefighting Proactive
- 14. Firefighting
- 15. •New project supported by all the engineering teams •More than 25 Engineers involved (4 SREs) •Status Meeting every 3 hours •One Goal: Keep Intercom UP A massive effort
- 16. Moved queries to read replicas Got rid of bad database usages Vertical partitioning Reviewed top queries Schema optimizations Keep Intercom UP
- 17. •No downtime •Query volume reduced by 25 % Huge impact in 3 days
- 18. Huge impact in 3 days
- 19. Work mode Database Health PreventiveFireFighting Proactive
- 20. Preventive
- 21. •Project focused •Weekly lunch and learn to share new ideas •Continuous Improvement •Regular lessons learned sessions New organization
- 22. But it didn’t work out
- 24. Work mode Database Health Team Health
- 26. Work mode Team Health Database Health Preventive Firefighting Proactive
- 28. Firefighting
- 29. •No clear target •Too many pages Misalignment
- 30. No head space to think strategically
- 31. 80% Drop
- 32. Alignment Current Situation Definition of done The DB is under constant pressure Majority of the read operations are against a replica CPU is routinely > 70 % Adding new replicas to spread the load is easy AMI rollouts push us closer to overall connection limit Master CPU is always below 50% Bad database usages: queues, analytics, large tables Master handles less than 10.000 SELECTS/s Connection limit is not a concern
- 34. Preventive
- 35. Split read / write load Tuned some database settings Removed a lot of bad database usages MySQL training Improved our dashboards and alarms Definition of done
- 36. •Hit our definition of done •Reduced our number of pages by 80 % •Great atmosphere and morale Great milestone
- 38. Proactive
- 39. •What’s next ? •What should we prioritise ? •Do we still need someone oncall 24/7 ? •Team morale was trending down High level of ambiguity
- 41. •Hackathon •Chats about the team •Karting / Diner Team Week
- 43. •New projects •Mission, Vision, Values •New Team organisation •New T-shirts Team Week
- 44. •Never underestimate the work and the iterative process needed for moving to a new work mode •Always align your organization with the current situation and the morale of the team •Good news and good results also come with consequences and can imply radical changes Lessons learned