Downloaded 11 times
Uploaded byTodd Palino
Code Yellow: Helping Operations Top-Heavy Teams the Smart Way
The document outlines strategies for managing operations in top-heavy teams, shared by LinkedIn leaders Jeff Weiner, Michael Kehoe, and Todd Palino. It discusses challenges like technical debt, resource allocation, and alert fatigue while providing a structured approach for improvement through communication, success criteria, and resource acquisition. Key learnings emphasize measuring toil, prioritizing efforts to reduce overhead, and effective communication with stakeholders.
More Related Content
Similar to Code Yellow: Helping Operations Top-Heavy Teams the Smart Way
Code Yellow: Helping Operations Top-Heavy Teams the Smart Way
- 1. Helping operations top-heavy teamsthe smart way Jeff Weiner Chief Executive Officer Michael Kehoe Staff Site Reliability Engineer Todd Palino Sr Staff Site Reliability Engineer
- 2. This Is TheOnly Slide You May Need a Picture Of slideshare.net/ToddPalino slideshare.net/MichaelKehoe3
- 3. Michael Kehoe $ WHOAMI •Staff Site Reliability Engineer @ LinkedIn • Production-SRE Team • Funny accent = Australian + 4 years American • Former Network Engineer at the University of Queensland
- 4. Todd Palino $ WHOAMI •Senior Staff SRE @ LinkedIn • Capacity Engineering Team • Co-Author of Kafka: The Definitive Guide • Late of VeriSign Infrastructure Engineering
- 5. When Operations Isn’tPerfect Code Yellow https://devops.com/code-yellow-when-operations-isnt-perfect/
- 6. • How toquickly erase all your technical debt • How to change your engineering culture This talk is not
- 7. • How toidentify team anti-patterns • How to work through high toil • How to create sustainable workloads This talk is
- 8. Today’s agenda 1 Background 2 Scenario1: Traffic-SRE 3 Scenario 2: Kafka-SRE 4 Building A Formula For Success 5 Key Learnings 6 Q&A
- 9.
- 10. Personal Experience inthe past 19 months ASSISTANCE RENDERED • Traffic-SRE: Technical Debt/ Resource Allocation • Voyager-SRE: Technical Debt • Capacity War-room • Espresso-SRE: Reliability • Kafka-SRE: Capacity and Alert Fatigue
- 11.
- 12. Problem Statement Technical Debt •Written documentation needed improvement • Deployment infrastructure needed investment • Alert Fatigue Traffic-SRE
- 13. Problem Statement Resource Allocations •Backlog of work for clients • Staff shortage
- 14.
- 16. Problem Statement Capacity Planning •Multi-tenant Infrastructure • No resource controls • Unclear resource ownership • Ad-hoc capacity planning • Sudden 100% increase in traffic
- 17. Problem Statement Alert Fatigue •Multiple applications overutilized • No time for proactive work • Most alerts non-actionable
- 18. Building a formulafor success
- 19.
- 20. Building a formulafor success Define the areas that need attacking Problem Statement Communicate expectations with clients & partners Communication & Partnerships Define success criteria Exit Criteria Get the help that you require Resource Acquisition Plan for short-term & long-term Planning
- 21. Define the areasthat need attacking Problem Statement • Admit there is a problem • Measure the problem • Understand the problem • Determines underlying causes that need to be fixed Building a formula for success
- 22. Define success criteria ExitCriteria • Define concrete goals • Define concrete success criteria • Measure via an operational metric • Measure via a project being completed • Define timelines for completion Building a formula for success
- 23. Get the helpyou require Resource Acquisition • Ask other teams for help • Get dedicated engineers/ project managers/ other roles as required • Set exit-date for resources Building a formula for success
- 24. Plan for theshort-term & long-term Planning • Plan out short-term work • Plan out longer-term projects • Do they need to be rescheduled? • Prioritize work that will reduce toil & burnout (Automation + Measurement) Building a formula for success
- 25. Communicate expectations with clients& partners Communicatio n & Partnerships • Communicate problem statement & exit criteria • Send regular progress updates • Ensure that stakeholders understand delays & expected outcomes Building a formula for success
- 26.
- 27. Key Learnings Measure toil/ overhead Measure Prioritizeefforts to remove overhead/toil Prioritize Communicate with partners & teams Communicate
- 28.