SlideShare a Scribd company logo

Helping operations top-heavy teams the smart way

SRE teams can sometimes run into periods of time where they have staff burnout, technical debt or poor reliability. As SRE’s, we’re programmed to keep fighting through the issues, when sometimes it’s best to step back, assess the situation; and ask for help to put the team back on a successful pathway. This talk will discuss three separate experiences where teams needed some extra help to stabilize their services and oncall. We’ll discuss how to identify struggling teams; get the right assistance; and build a strategy for the team to succeed.

1 of 27
Download to read offline
Helping operations top-heavy
teams the smart way
(Lessons from my experience being loaned out to SRE teams)
Michael Kehoe
Staff Site Reliability Engineer
Michael Kehoe
$ WHOAMI
• Staff Site Reliability Engineer @ LinkedIn
• Production-SRE Team
• Funny accent = Australian + 4 years
American
• Former Network Engineer at the
University of Queensland
Production-SRE Team @ LinkedIn
$ WHOAMI
• Disaster Recovery - Planning &
Automation
• Incident Response – Process &
Automation
• Visibility Engineering – Making use of
operational data
• Reliability Principles – Defining best
practice & automating it
• How to quickly erase all your
technical debt
• How to change your engineering
culture
This talk is not
• How to identify team anti-patterns
• How to work through high-toil
• How to create sustainable
workloads
This talk is
Today’s
agenda
1 Background
2 Scenario 1: Resource Allocation
3 Scenario 2: Technical Debt
4 Scenario 3: High Toil
5 Building A Formula For Success
6 Key Learnings
7 Q&A
Ad

Recommended

The Snowman - visualising Agile project roles & responsibilities
The Snowman - visualising Agile project roles & responsibilitiesThe Snowman - visualising Agile project roles & responsibilities
The Snowman - visualising Agile project roles & responsibilitiesAbdul Ghafoor
 
Building effective teams in Amdocs TECC - Presentation
Building effective teams in Amdocs TECC - PresentationBuilding effective teams in Amdocs TECC - Presentation
Building effective teams in Amdocs TECC - PresentationMilind Gokhale
 
Software Project Success Through Value Assurance
Software Project Success Through Value AssuranceSoftware Project Success Through Value Assurance
Software Project Success Through Value AssuranceValueware
 
Operations impossible
Operations impossibleOperations impossible
Operations impossibleRamesh Victor
 
Tipu: agile service improvement
Tipu: agile service improvementTipu: agile service improvement
Tipu: agile service improvementRob England
 
The Snowman - visualising Agile project roles & responsibilities
The Snowman - visualising Agile project roles & responsibilities The Snowman - visualising Agile project roles & responsibilities
The Snowman - visualising Agile project roles & responsibilities Abdul Ghafoor
 
Vygantas Kazlauskas - How Agile saved Christmas in Estonia
Vygantas Kazlauskas - How Agile saved Christmas in EstoniaVygantas Kazlauskas - How Agile saved Christmas in Estonia
Vygantas Kazlauskas - How Agile saved Christmas in EstoniaAgile Lietuva
 

More Related Content

What's hot

Making a Project a Complete Success with Post-Implementation Strategies | Jul...
Making a Project a Complete Success with Post-Implementation Strategies | Jul...Making a Project a Complete Success with Post-Implementation Strategies | Jul...
Making a Project a Complete Success with Post-Implementation Strategies | Jul...Katie Elliott
 
Agile network India | Dysfunctions in a Scrum Master's Role | Soja Nizam
Agile network India | Dysfunctions in a Scrum Master's Role | Soja NizamAgile network India | Dysfunctions in a Scrum Master's Role | Soja Nizam
Agile network India | Dysfunctions in a Scrum Master's Role | Soja NizamAgileNetwork
 
Using OEE to Improve Production - Interphex 2012
Using OEE to Improve Production - Interphex 2012Using OEE to Improve Production - Interphex 2012
Using OEE to Improve Production - Interphex 2012Adrian Pask
 
Blackbaud CRM After Go-Live
Blackbaud CRM After Go-LiveBlackbaud CRM After Go-Live
Blackbaud CRM After Go-LiveBlackbaud
 
Top tips for a successful traceability system implemention paula peterson 2015
Top tips for a successful traceability system implemention paula peterson 2015Top tips for a successful traceability system implemention paula peterson 2015
Top tips for a successful traceability system implemention paula peterson 2015Paula Peterson
 
Agile Balanced Scorecard -Agile Tour 2011 Pune
Agile Balanced Scorecard -Agile Tour 2011 PuneAgile Balanced Scorecard -Agile Tour 2011 Pune
Agile Balanced Scorecard -Agile Tour 2011 PuneAsheesh Mehdiratta
 
Why lean can't succeed without operational discipline
Why lean can't succeed without operational disciplineWhy lean can't succeed without operational discipline
Why lean can't succeed without operational disciplineCalvin L Williams
 
Agile Network India | Disciplined Agile Through Case Study | Nagaraja Gundappa
Agile Network India | Disciplined Agile Through Case Study | Nagaraja GundappaAgile Network India | Disciplined Agile Through Case Study | Nagaraja Gundappa
Agile Network India | Disciplined Agile Through Case Study | Nagaraja GundappaAgileNetwork
 
Project management career seminar
Project management career seminarProject management career seminar
Project management career seminarOjiugo Ajunwa
 
Procensol Breakfast Forum Launch - Modern Business Transformation
Procensol Breakfast Forum Launch - Modern Business TransformationProcensol Breakfast Forum Launch - Modern Business Transformation
Procensol Breakfast Forum Launch - Modern Business TransformationProcensol
 
Go Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchio
Go Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchioGo Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchio
Go Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchioKatie Elliott
 
City of Canning: 4 Key Success Factors to Drive Engagement and Build Momentum
City of Canning: 4 Key Success Factors to Drive Engagement and Build MomentumCity of Canning: 4 Key Success Factors to Drive Engagement and Build Momentum
City of Canning: 4 Key Success Factors to Drive Engagement and Build MomentumEileenTan67
 
Agile project management - everything you want to know but were afraid to ask...
Agile project management - everything you want to know but were afraid to ask...Agile project management - everything you want to know but were afraid to ask...
Agile project management - everything you want to know but were afraid to ask...Association for Project Management
 
Agile Network India | Data driven approach to Retrospectives | Sandhya Bhayana
Agile Network India | Data driven approach to Retrospectives | Sandhya BhayanaAgile Network India | Data driven approach to Retrospectives | Sandhya Bhayana
Agile Network India | Data driven approach to Retrospectives | Sandhya BhayanaAgileNetwork
 

What's hot (18)

Implementing infinity hr (katie cuthriell)
Implementing infinity hr (katie cuthriell)Implementing infinity hr (katie cuthriell)
Implementing infinity hr (katie cuthriell)
 
Making a Project a Complete Success with Post-Implementation Strategies | Jul...
Making a Project a Complete Success with Post-Implementation Strategies | Jul...Making a Project a Complete Success with Post-Implementation Strategies | Jul...
Making a Project a Complete Success with Post-Implementation Strategies | Jul...
 
Agile network India | Dysfunctions in a Scrum Master's Role | Soja Nizam
Agile network India | Dysfunctions in a Scrum Master's Role | Soja NizamAgile network India | Dysfunctions in a Scrum Master's Role | Soja Nizam
Agile network India | Dysfunctions in a Scrum Master's Role | Soja Nizam
 
Using OEE to Improve Production - Interphex 2012
Using OEE to Improve Production - Interphex 2012Using OEE to Improve Production - Interphex 2012
Using OEE to Improve Production - Interphex 2012
 
Blackbaud CRM After Go-Live
Blackbaud CRM After Go-LiveBlackbaud CRM After Go-Live
Blackbaud CRM After Go-Live
 
Top tips for a successful traceability system implemention paula peterson 2015
Top tips for a successful traceability system implemention paula peterson 2015Top tips for a successful traceability system implemention paula peterson 2015
Top tips for a successful traceability system implemention paula peterson 2015
 
Agile Balanced Scorecard -Agile Tour 2011 Pune
Agile Balanced Scorecard -Agile Tour 2011 PuneAgile Balanced Scorecard -Agile Tour 2011 Pune
Agile Balanced Scorecard -Agile Tour 2011 Pune
 
Why lean can't succeed without operational discipline
Why lean can't succeed without operational disciplineWhy lean can't succeed without operational discipline
Why lean can't succeed without operational discipline
 
Agile Network India | Disciplined Agile Through Case Study | Nagaraja Gundappa
Agile Network India | Disciplined Agile Through Case Study | Nagaraja GundappaAgile Network India | Disciplined Agile Through Case Study | Nagaraja Gundappa
Agile Network India | Disciplined Agile Through Case Study | Nagaraja Gundappa
 
Project management career seminar
Project management career seminarProject management career seminar
Project management career seminar
 
Procensol Breakfast Forum Launch - Modern Business Transformation
Procensol Breakfast Forum Launch - Modern Business TransformationProcensol Breakfast Forum Launch - Modern Business Transformation
Procensol Breakfast Forum Launch - Modern Business Transformation
 
Lean testing
Lean testingLean testing
Lean testing
 
IT Outsourcing Best Practices
IT Outsourcing Best PracticesIT Outsourcing Best Practices
IT Outsourcing Best Practices
 
Go Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchio
Go Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchioGo Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchio
Go Live is Just the Start - Managing AX Improvement Projects | Carlo DiPucchio
 
TPM: Focused Improvement (Kobetsu Kaizen) Poster
TPM: Focused Improvement (Kobetsu Kaizen) PosterTPM: Focused Improvement (Kobetsu Kaizen) Poster
TPM: Focused Improvement (Kobetsu Kaizen) Poster
 
City of Canning: 4 Key Success Factors to Drive Engagement and Build Momentum
City of Canning: 4 Key Success Factors to Drive Engagement and Build MomentumCity of Canning: 4 Key Success Factors to Drive Engagement and Build Momentum
City of Canning: 4 Key Success Factors to Drive Engagement and Build Momentum
 
Agile project management - everything you want to know but were afraid to ask...
Agile project management - everything you want to know but were afraid to ask...Agile project management - everything you want to know but were afraid to ask...
Agile project management - everything you want to know but were afraid to ask...
 
Agile Network India | Data driven approach to Retrospectives | Sandhya Bhayana
Agile Network India | Data driven approach to Retrospectives | Sandhya BhayanaAgile Network India | Data driven approach to Retrospectives | Sandhya Bhayana
Agile Network India | Data driven approach to Retrospectives | Sandhya Bhayana
 

Similar to Helping operations top-heavy teams the smart way

Code Yellow: Helping operations top-heavy teams the smart way
Code Yellow: Helping operations top-heavy teams the smart wayCode Yellow: Helping operations top-heavy teams the smart way
Code Yellow: Helping operations top-heavy teams the smart wayMichael Kehoe
 
Code Yellow: Helping Operations Top-Heavy Teams the Smart Way
Code Yellow: Helping Operations Top-Heavy Teams the Smart WayCode Yellow: Helping Operations Top-Heavy Teams the Smart Way
Code Yellow: Helping Operations Top-Heavy Teams the Smart WayTodd Palino
 
Helping operations top-heavy teams the smart way
Helping operations top-heavy teams the smart wayHelping operations top-heavy teams the smart way
Helping operations top-heavy teams the smart wayMichael Kehoe
 
American Electric Power Ercot kickoff
American Electric Power Ercot kickoffAmerican Electric Power Ercot kickoff
American Electric Power Ercot kickoffJohn Napier
 
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...ssuser835d1a
 
AVATA Webinar: Solutions to Common Demantra & ASCP Challenges
AVATA Webinar: Solutions to Common Demantra & ASCP ChallengesAVATA Webinar: Solutions to Common Demantra & ASCP Challenges
AVATA Webinar: Solutions to Common Demantra & ASCP ChallengesAVATA
 
Methodology lean IT transformation mission
Methodology   lean IT transformation missionMethodology   lean IT transformation mission
Methodology lean IT transformation missionJean-François Nguyen
 
The Dashlane Agile Journey
The Dashlane Agile JourneyThe Dashlane Agile Journey
The Dashlane Agile JourneyDashlane
 
Engineering Teams and Systems for Velocity
Engineering Teams and Systems for VelocityEngineering Teams and Systems for Velocity
Engineering Teams and Systems for VelocityJean Barmash
 
Lean Six Sigma-An Execution Engine
Lean Six Sigma-An Execution EngineLean Six Sigma-An Execution Engine
Lean Six Sigma-An Execution EngineMark Cichonski
 
Fundamentals of agile tntu (2015-04-27)
Fundamentals of agile   tntu (2015-04-27)Fundamentals of agile   tntu (2015-04-27)
Fundamentals of agile tntu (2015-04-27)Oleg Nazarevych
 
R a ci & innovation
R a ci & innovationR a ci & innovation
R a ci & innovationAlan Culler
 
BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...
BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...
BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...Business of Software Conference
 
CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...
CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...
CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...TheConnectedCause
 
Practical Enterprise Architecture in Medium-size Corporation using TOGAF
Practical Enterprise Architecture in Medium-size Corporation using TOGAFPractical Enterprise Architecture in Medium-size Corporation using TOGAF
Practical Enterprise Architecture in Medium-size Corporation using TOGAFMichael Sukachev
 
Agile ncr pramila hitachi consulting_future_coaching
Agile ncr pramila hitachi consulting_future_coachingAgile ncr pramila hitachi consulting_future_coaching
Agile ncr pramila hitachi consulting_future_coachingAgileNCR2016
 
The Five Phases of Agile Maturity (Part 2): Phase 3 and 4
The Five Phases of Agile Maturity (Part 2): Phase 3 and 4The Five Phases of Agile Maturity (Part 2): Phase 3 and 4
The Five Phases of Agile Maturity (Part 2): Phase 3 and 4Cprime
 
Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6
Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6
Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6PrescienceTechnology
 
Changing culture and building efficiencies by applying the Lean principles to...
Changing culture and building efficiencies by applying the Lean principles to...Changing culture and building efficiencies by applying the Lean principles to...
Changing culture and building efficiencies by applying the Lean principles to...Association for Project Management
 

Similar to Helping operations top-heavy teams the smart way (20)

Code Yellow: Helping operations top-heavy teams the smart way
Code Yellow: Helping operations top-heavy teams the smart wayCode Yellow: Helping operations top-heavy teams the smart way
Code Yellow: Helping operations top-heavy teams the smart way
 
Code Yellow: Helping Operations Top-Heavy Teams the Smart Way
Code Yellow: Helping Operations Top-Heavy Teams the Smart WayCode Yellow: Helping Operations Top-Heavy Teams the Smart Way
Code Yellow: Helping Operations Top-Heavy Teams the Smart Way
 
Helping operations top-heavy teams the smart way
Helping operations top-heavy teams the smart wayHelping operations top-heavy teams the smart way
Helping operations top-heavy teams the smart way
 
American Electric Power Ercot kickoff
American Electric Power Ercot kickoffAmerican Electric Power Ercot kickoff
American Electric Power Ercot kickoff
 
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
103240-The-New-Way-of-Thinking-Our-Implementation-experience-with-Oracle-HCM-...
 
AVATA Webinar: Solutions to Common Demantra & ASCP Challenges
AVATA Webinar: Solutions to Common Demantra & ASCP ChallengesAVATA Webinar: Solutions to Common Demantra & ASCP Challenges
AVATA Webinar: Solutions to Common Demantra & ASCP Challenges
 
Methodology lean IT transformation mission
Methodology   lean IT transformation missionMethodology   lean IT transformation mission
Methodology lean IT transformation mission
 
The Dashlane Agile Journey
The Dashlane Agile JourneyThe Dashlane Agile Journey
The Dashlane Agile Journey
 
Engineering Teams and Systems for Velocity
Engineering Teams and Systems for VelocityEngineering Teams and Systems for Velocity
Engineering Teams and Systems for Velocity
 
Lean Six Sigma-An Execution Engine
Lean Six Sigma-An Execution EngineLean Six Sigma-An Execution Engine
Lean Six Sigma-An Execution Engine
 
Fundamentals of agile tntu (2015-04-27)
Fundamentals of agile   tntu (2015-04-27)Fundamentals of agile   tntu (2015-04-27)
Fundamentals of agile tntu (2015-04-27)
 
R a ci & innovation
R a ci & innovationR a ci & innovation
R a ci & innovation
 
BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...
BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...
BoS2015 Jeff Szczepanski – COO, Stack Exchange - Stack Overflow. Scaling a Te...
 
CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...
CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...
CRMready Webinar Series - Part 3 - How to Make Your Nonprofit’s CRM Implement...
 
Practical Enterprise Architecture in Medium-size Corporation using TOGAF
Practical Enterprise Architecture in Medium-size Corporation using TOGAFPractical Enterprise Architecture in Medium-size Corporation using TOGAF
Practical Enterprise Architecture in Medium-size Corporation using TOGAF
 
Agile ncr pramila hitachi consulting_future_coaching
Agile ncr pramila hitachi consulting_future_coachingAgile ncr pramila hitachi consulting_future_coaching
Agile ncr pramila hitachi consulting_future_coaching
 
The Five Phases of Agile Maturity (Part 2): Phase 3 and 4
The Five Phases of Agile Maturity (Part 2): Phase 3 and 4The Five Phases of Agile Maturity (Part 2): Phase 3 and 4
The Five Phases of Agile Maturity (Part 2): Phase 3 and 4
 
Effective Scrum
Effective ScrumEffective Scrum
Effective Scrum
 
Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6
Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6
Webinar - Integrating InEight Hard Dollar and Oracle Primavera P6
 
Changing culture and building efficiencies by applying the Lean principles to...
Changing culture and building efficiencies by applying the Lean principles to...Changing culture and building efficiencies by applying the Lean principles to...
Changing culture and building efficiencies by applying the Lean principles to...
 

More from Michael Kehoe

QConSF 2018: Building Production-Ready Applications
QConSF 2018: Building Production-Ready ApplicationsQConSF 2018: Building Production-Ready Applications
QConSF 2018: Building Production-Ready ApplicationsMichael Kehoe
 
AllDayDevops: What the NTSB teaches us about incident management & postmortems
AllDayDevops: What the NTSB teaches us about incident management & postmortemsAllDayDevops: What the NTSB teaches us about incident management & postmortems
AllDayDevops: What the NTSB teaches us about incident management & postmortemsMichael Kehoe
 
Linux Container Basics
Linux Container BasicsLinux Container Basics
Linux Container BasicsMichael Kehoe
 
Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops
Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet DropsPapers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops
Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet DropsMichael Kehoe
 
What the NTSB teaches us about incident management & postmortems
What the NTSB teaches us about incident management & postmortemsWhat the NTSB teaches us about incident management & postmortems
What the NTSB teaches us about incident management & postmortemsMichael Kehoe
 
PyBay 2018: Production-Ready Python Applications
PyBay 2018: Production-Ready Python ApplicationsPyBay 2018: Production-Ready Python Applications
PyBay 2018: Production-Ready Python ApplicationsMichael Kehoe
 
The Next Wave of Reliability Engineering
The Next Wave of Reliability EngineeringThe Next Wave of Reliability Engineering
The Next Wave of Reliability EngineeringMichael Kehoe
 
Building Production-Ready Microservices: DevopsExchangeSF
Building Production-Ready Microservices: DevopsExchangeSFBuilding Production-Ready Microservices: DevopsExchangeSF
Building Production-Ready Microservices: DevopsExchangeSFMichael Kehoe
 
SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...
SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...
SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...Michael Kehoe
 
SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...
SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...
SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...Michael Kehoe
 
SRECon-Europe-2017: Networks for SREs
SRECon-Europe-2017: Networks for SREsSRECon-Europe-2017: Networks for SREs
SRECon-Europe-2017: Networks for SREsMichael Kehoe
 
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleVelocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleMichael Kehoe
 
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedInReducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedInMichael Kehoe
 
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...Michael Kehoe
 
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInCouchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInMichael Kehoe
 
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016Michael Kehoe
 
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsMichael Kehoe
 
SRECon USA 2016: Growing your Entry Level Talent
SRECon USA 2016: Growing your Entry Level TalentSRECon USA 2016: Growing your Entry Level Talent
SRECon USA 2016: Growing your Entry Level TalentMichael Kehoe
 

More from Michael Kehoe (20)

eBPF Workshop
eBPF WorkshopeBPF Workshop
eBPF Workshop
 
eBPF Basics
eBPF BasicseBPF Basics
eBPF Basics
 
QConSF 2018: Building Production-Ready Applications
QConSF 2018: Building Production-Ready ApplicationsQConSF 2018: Building Production-Ready Applications
QConSF 2018: Building Production-Ready Applications
 
AllDayDevops: What the NTSB teaches us about incident management & postmortems
AllDayDevops: What the NTSB teaches us about incident management & postmortemsAllDayDevops: What the NTSB teaches us about incident management & postmortems
AllDayDevops: What the NTSB teaches us about incident management & postmortems
 
Linux Container Basics
Linux Container BasicsLinux Container Basics
Linux Container Basics
 
Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops
Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet DropsPapers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops
Papers We Love Sept. 2018: 007: Democratically Finding The Cause of Packet Drops
 
What the NTSB teaches us about incident management & postmortems
What the NTSB teaches us about incident management & postmortemsWhat the NTSB teaches us about incident management & postmortems
What the NTSB teaches us about incident management & postmortems
 
PyBay 2018: Production-Ready Python Applications
PyBay 2018: Production-Ready Python ApplicationsPyBay 2018: Production-Ready Python Applications
PyBay 2018: Production-Ready Python Applications
 
The Next Wave of Reliability Engineering
The Next Wave of Reliability EngineeringThe Next Wave of Reliability Engineering
The Next Wave of Reliability Engineering
 
Building Production-Ready Microservices: DevopsExchangeSF
Building Production-Ready Microservices: DevopsExchangeSFBuilding Production-Ready Microservices: DevopsExchangeSF
Building Production-Ready Microservices: DevopsExchangeSF
 
SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...
SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...
SF Chaos Engineering Meetup: Building Disaster Recovery via Resilience Engine...
 
SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...
SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...
SRECon-Europe-2017: Reducing MTTR and False Escalations: Event Correlation at...
 
SRECon-Europe-2017: Networks for SREs
SRECon-Europe-2017: Networks for SREsSRECon-Europe-2017: Networks for SREs
SRECon-Europe-2017: Networks for SREs
 
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleVelocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
 
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedInReducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
 
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
 
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInCouchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
 
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016
 
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production Systems
 
SRECon USA 2016: Growing your Entry Level Talent
SRECon USA 2016: Growing your Entry Level TalentSRECon USA 2016: Growing your Entry Level Talent
SRECon USA 2016: Growing your Entry Level Talent
 

Recently uploaded

My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17Ana-Maria Mihalceanu
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsInflectra
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Umar Saif
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, GoogleISPMAIndia
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Daniel Toomey
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Bit N Build Poland
Bit N Build PolandBit N Build Poland
Bit N Build PolandGDSC PJATK
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxMaarten Balliauw
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?MENGSAYLOEM1
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch TuesdayIvanti
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Q1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIQ1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIMemory Fabric Forum
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...shaiyuvasv
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIEDanBrown980551
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Q1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product LineupQ1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product LineupMemory Fabric Forum
 
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)Memory Fabric Forum
 

Recently uploaded (20)

My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
 
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
"The Transformative Power of AI and Open Challenges" by Dr. Manish Gupta, Google
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
Bit N Build Poland
Bit N Build PolandBit N Build Poland
Bit N Build Poland
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptx
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch Tuesday
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Q1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIQ1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AI
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIE
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Q1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product LineupQ1 Memory Fabric Forum: SMART CXL Product Lineup
Q1 Memory Fabric Forum: SMART CXL Product Lineup
 
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
Q1 Memory Fabric Forum: Intel Enabling Compute Express Link (CXL)
 

Helping operations top-heavy teams the smart way

  • 1. Helping operations top-heavy teams the smart way (Lessons from my experience being loaned out to SRE teams) Michael Kehoe Staff Site Reliability Engineer
  • 2. Michael Kehoe $ WHOAMI • Staff Site Reliability Engineer @ LinkedIn • Production-SRE Team • Funny accent = Australian + 4 years American • Former Network Engineer at the University of Queensland
  • 3. Production-SRE Team @ LinkedIn $ WHOAMI • Disaster Recovery - Planning & Automation • Incident Response – Process & Automation • Visibility Engineering – Making use of operational data • Reliability Principles – Defining best practice & automating it
  • 4. • How to quickly erase all your technical debt • How to change your engineering culture This talk is not
  • 5. • How to identify team anti-patterns • How to work through high-toil • How to create sustainable workloads This talk is
  • 6. Today’s agenda 1 Background 2 Scenario 1: Resource Allocation 3 Scenario 2: Technical Debt 4 Scenario 3: High Toil 5 Building A Formula For Success 6 Key Learnings 7 Q&A
  • 8. Personal Experience in the past 15 months ASSISTANCE RENDERED • Traffic-SRE: Resource Allocation • Voyager-SRE: Technical Debt • Capacity War-room • Espresso-SRE: Reliability
  • 10. Problem Statement Resource Allocations • Lack of written documentation • Backlog of work for clients • Alert Fatigue
  • 12. Problem Statement Technical Debt • New frontend service • Understanding performance is complicated • Management of dependent services was difficult
  • 14. Problem Statement High Toil • Large multi-tenant/ multi-cluster database team • Lack of maturity in team-specific automation • Alert Fatigue
  • 15. Building a formula for success
  • 17. Building a formula for success Define the areas that need attacking Problem Statement Communicate expectations with clients & partners Commutation & Partnerships Define success criteria Exit Criteria Get the help that you require Resource Acquisition Plan for short-term & long-term Planning
  • 18. Define the areas that need attacking Problem Statement • Admit there is a problem • Measure the problem • Understand the problem • Determines underlying causes that need to be fixed Building a formula for success
  • 19. Define success criteria Exit Criteria • Define concrete goals • Define concrete success criteria • Measure via an operational metric • Measure via a project being completed • Define timelines for completion Building a formula for success
  • 20. Get the help you require Resource Acquisition • Ask other teams for help • Get dedicated engineers/ project managers/ other roles as required • Set exit-date for resources Building a formula for success
  • 21. Plan for the short-term & long-term Planning • Plan out short-term work • Plan out longer-term projects • Do they need to be rescheduled? • Prioritize work that will reduce toil & burnout (Automation + Measurement) Building a formula for success
  • 22. Communicate expectations with clients & partners Communicatio n & Partnerships • Communicate problem statement & exit criteria • Send regular progress updates • Ensure that stakeholders understand delays & expected outcomes Building a formula for success
  • 23. When Operations Isn’t Perfect Code Yellow https://devops.com/code-yellow-when-operations-isnt-perfect/
  • 25. Key Learnings Measure toil/ overhead Measure Prioritize efforts to remove overhead/toil Prioritize Communicate with partners & teams Communicate
  • 26. Q&A

Editor's Notes

  1. Michael So we’re apart of a team at LinkedIn called Production-SRE The key tenants of production-sre at LinkedIn is: Assist in restoring stability during site-critical issues Developing applications to reduce MTTD and MTTR Provide direction and guidelines for site-troubleshooting Build tools for efficient site-issue troubleshooting, issue detection and correlation As this presentation goes on, you’ll notice how an Event Correlation system fits into these
  2. This talk isn’t how to magically erase all of your technical debt Neither is it a talk on changing your engineering culture
  3. This talk is How to identify team anti-patterns How to work through high-toil How to create sustainable workloads
  4. Michael So we’re apart of a team at LinkedIn called Production-SRE The key tenants of production-sre at LinkedIn is: Assist in restoring stability during site-critical issues Developing applications to reduce MTTD and MTTR Provide direction and guidelines for site-troubleshooting Build tools for efficient site-issue troubleshooting, issue detection and correlation As this presentation goes on, you’ll notice how an Event Correlation system fits into these
  5. So the first scenario I want to discuss is when I got pulled into the Traffic team due to severe resource allocation issues: We had a team that had a lack of written documentation on how their platform worked and was deployed They had a large backlog of work for clients And there was a large amount of alert fatigue due to a some poorly defined alerts and some infrastructure that needed upgrading but they hadn’t gotten to it yet Ontop of that, 4/5 team members left in a short period of time and started doing reliability operations at another company together So we’re in a bit of a pickle here…. So in response, we took 5 staff SRE’s from other teams and dedicated them to the traffic team for a period of 3 months Stopped all non-critical client work for a number of weeks Completely recreated all monitoring systems Spent a large chunk of time removing complexity Focused on infrastructure reliability
  6. The second team I worked with was our frontend API service team
  7. Thousands of instances Lack of maturity in automation for the team Alert fatigue given the size of their infrastructure Poor visibility into ops metrics