SlideShare a Scribd company logo

Ignite (10m) how to not burn out your monitoring team

Bad monitoring, alerting and logging has made Gil Zellner very frustrated in some of his previous positions. It seems that almost nobody gets this exactly right. This will be a talk about the most annoying issues he has come across and advice for how to fix them.

1 of 56
Download to read offline
Relaxing picture of Yoga
PagerDuty Alert
Ignite (10m)   how to not burn out your monitoring team
hunt through logs for 2
hours
Ignite (10m)   how to not burn out your monitoring team
Ignite (10m)   how to not burn out your monitoring team
Ignite (10m)   how to not burn out your monitoring team
How to not burn out your
production team
Gil Zellner
(CloudifyDev at Gigaspaces)
Twitter: @Heathenaspargus
Who am I?
Now:
Past:
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
cost of hiring new employee is 1.5-3x their monthly
salary
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
Next day
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
frustration - I am unable to complete my task
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
Time spent inefficiently
@Heathenaspargus
Repetitive tasks
@Heathenaspargus
Working Alone
@Heathenaspargus
Yak
Shaving
@Heathenaspargus
https://www.ergoflex.co.uk/blog/category/sleep-research/sleeponomics-could-sleep-deprivation-be-the-real-reason-politicians-make-bad-decisions
@Heathenaspargus
Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus
Mandatory Half day-off after night production issue
@Heathenaspargus
Allocate weekly time to resolve or automate issues that kept us up at night
@Heathenaspargus
Wider rotation (more people do on-call)
@Heathenaspargus
Knowledge Matrix
Deploy System Mobile Link Backend
Gil V V
Karen V V
Ari V V
@Heathenaspargus
@Heathenaspargus
Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
Ignite (10m)   how to not burn out your monitoring team
@Heathenaspargus
solution: alert only things that meet the following
criteria:
1) Alert on symptoms, not suspected "causes"
2) Actionable
3) Business breaking
@Heathenaspargus
Alerte générale!
@Heathenaspargus
Solution: direct alerts to relevant parties
@Heathenaspargus
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
What are your KPIs ?
@Heathenaspargus
@Heathenaspargus
Netflix stream starts per second
@Heathenaspargus
Picking how to measure things
Ignite (10m)   how to not burn out your monitoring team
Diagnosis
@Heathenaspargus
Make heal script
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
Facebook Auto Remediation
https://www.facebook.com/notes/facebook-engineering/making-facebook-self-healing/10150275248698920
@Heathenaspargus
Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
@Heathenaspargus
Bad artists copy, great artists steal
email:
Gil.Zellner@gmail.com
Twitter:
@Heathenaspargus

Recommended

BigData - Hadoop -by 侯圣文@secooler
BigData - Hadoop -by 侯圣文@secooler BigData - Hadoop -by 侯圣文@secooler
BigData - Hadoop -by 侯圣文@secooler Shengwen HOU(侯圣文)
 
GIS for People & Parks
GIS for People & ParksGIS for People & Parks
GIS for People & ParksJereme Monteau
 
From Ops to Dev and Back Again - Gil Zellner - DevOpsDays Tel Aviv 2017
From Ops to Dev and Back Again - Gil Zellner - DevOpsDays Tel Aviv 2017From Ops to Dev and Back Again - Gil Zellner - DevOpsDays Tel Aviv 2017
From Ops to Dev and Back Again - Gil Zellner - DevOpsDays Tel Aviv 2017DevOpsDays Tel Aviv
 
9. Media Exposure- DNA News
9. Media Exposure- DNA News9. Media Exposure- DNA News
9. Media Exposure- DNA NewsJaymin Gajjar
 
Agile architecture
Agile architectureAgile architecture
Agile architectureandy_butcher
 
#AI For Earth: When we started and where we're headed (and why we need your h...
#AI For Earth: When we started and where we're headed (and why we need your h...#AI For Earth: When we started and where we're headed (and why we need your h...
#AI For Earth: When we started and where we're headed (and why we need your h...World Resources Institute (WRI)
 
Kubernetes YAML + Science = Sanity
Kubernetes YAML + Science = SanityKubernetes YAML + Science = Sanity
Kubernetes YAML + Science = SanityVMware Tanzu
 

More Related Content

Viewers also liked

Become a Learning Masterchef
Become a Learning MasterchefBecome a Learning Masterchef
Become a Learning MasterchefKineoPacific
 
Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016Unmetric
 
Getting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem SolvingGetting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem SolvingHock Leng PUAH
 
How to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of BubblegumHow to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of BubblegumGreg Gifford
 
Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.kulibin
 
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...Marcel Lebrun
 
Common Mistakes Travelers Make
Common Mistakes Travelers MakeCommon Mistakes Travelers Make
Common Mistakes Travelers MakeVisitorsCoverage
 
Stepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value CreationStepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value CreationRobin Teigland
 
How to deal with bitter criticism
How to deal with bitter criticismHow to deal with bitter criticism
How to deal with bitter criticismXenia Y
 
SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012Candy Gourlay
 
Présentation Xebicon15 - Akamis - Marketing & IT
Présentation Xebicon15 -  Akamis - Marketing & ITPrésentation Xebicon15 -  Akamis - Marketing & IT
Présentation Xebicon15 - Akamis - Marketing & ITAKAMIS
 
Accelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPOAccelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPOVitaly Golomb
 
Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1Annabel Lee
 
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...Kapost
 
Time is Life. Life is Time
Time is Life. Life is TimeTime is Life. Life is Time
Time is Life. Life is TimeShirley Williams
 
Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)Julio Nuñez
 
The Science of Webinars
The Science of WebinarsThe Science of Webinars
The Science of WebinarsHubSpot
 
23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarte23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarteAna Sek
 

Viewers also liked (20)

Become a Learning Masterchef
Become a Learning MasterchefBecome a Learning Masterchef
Become a Learning Masterchef
 
Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016
 
Getting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem SolvingGetting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem Solving
 
How to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of BubblegumHow to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of Bubblegum
 
Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.
 
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
 
Common Mistakes Travelers Make
Common Mistakes Travelers MakeCommon Mistakes Travelers Make
Common Mistakes Travelers Make
 
Stepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value CreationStepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value Creation
 
How to deal with bitter criticism
How to deal with bitter criticismHow to deal with bitter criticism
How to deal with bitter criticism
 
SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012
 
Présentation Xebicon15 - Akamis - Marketing & IT
Présentation Xebicon15 -  Akamis - Marketing & ITPrésentation Xebicon15 -  Akamis - Marketing & IT
Présentation Xebicon15 - Akamis - Marketing & IT
 
Accelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPOAccelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPO
 
Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1
 
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
 
Time is Life. Life is Time
Time is Life. Life is TimeTime is Life. Life is Time
Time is Life. Life is Time
 
Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)
 
The Science of Webinars
The Science of WebinarsThe Science of Webinars
The Science of Webinars
 
KZero Radar chart Q1 2013
KZero Radar chart Q1 2013KZero Radar chart Q1 2013
KZero Radar chart Q1 2013
 
Disc
DiscDisc
Disc
 
23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarte23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarte
 

More from Gil Zellner

Lessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineeringLessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineeringGil Zellner
 
5m the unsung glory of internal tools (1)
5m   the unsung glory of internal tools (1)5m   the unsung glory of internal tools (1)
5m the unsung glory of internal tools (1)Gil Zellner
 
Full talk from ops to dev and back again
Full talk   from ops to dev and back againFull talk   from ops to dev and back again
Full talk from ops to dev and back againGil Zellner
 
How to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS BillHow to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS BillGil Zellner
 
Rinse and repeat
Rinse and repeatRinse and repeat
Rinse and repeatGil Zellner
 
A newborn as a production system
A newborn as a production systemA newborn as a production system
A newborn as a production systemGil Zellner
 
How to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quitHow to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quitGil Zellner
 

More from Gil Zellner (8)

HA coffee
HA coffeeHA coffee
HA coffee
 
Lessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineeringLessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineering
 
5m the unsung glory of internal tools (1)
5m   the unsung glory of internal tools (1)5m   the unsung glory of internal tools (1)
5m the unsung glory of internal tools (1)
 
Full talk from ops to dev and back again
Full talk   from ops to dev and back againFull talk   from ops to dev and back again
Full talk from ops to dev and back again
 
How to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS BillHow to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS Bill
 
Rinse and repeat
Rinse and repeatRinse and repeat
Rinse and repeat
 
A newborn as a production system
A newborn as a production systemA newborn as a production system
A newborn as a production system
 
How to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quitHow to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quit
 

Recently uploaded

fat and edible oil processsing.ppt, refining
fat and edible oil processsing.ppt, refiningfat and edible oil processsing.ppt, refining
fat and edible oil processsing.ppt, refiningteddymebratie
 
Renewable sources to produce electricity , Wind Energy
Renewable sources to produce electricity , Wind EnergyRenewable sources to produce electricity , Wind Energy
Renewable sources to produce electricity , Wind EnergyMaryam79507
 
Forged Fitting Socket Welding Standard- ASME-B16.11-2001.pdf
Forged Fitting Socket Welding Standard- ASME-B16.11-2001.pdfForged Fitting Socket Welding Standard- ASME-B16.11-2001.pdf
Forged Fitting Socket Welding Standard- ASME-B16.11-2001.pdfVikasKumar11936
 
Unit_1.pdf computer networks and computer topology
Unit_1.pdf computer networks and computer topologyUnit_1.pdf computer networks and computer topology
Unit_1.pdf computer networks and computer topology22i261
 
Amplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation TechniquesAmplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation TechniquesRich171473
 
21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS
21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS
21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaSDr. Shivashankar
 
A tale for cyber security phishing by Rima daqch
A tale for cyber security phishing by Rima daqchA tale for cyber security phishing by Rima daqch
A tale for cyber security phishing by Rima daqchRimaDaqch
 
Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!
Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!
Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!Mathias Magdowski
 
Drone design for agricultural applications
Drone design for agricultural applicationsDrone design for agricultural applications
Drone design for agricultural applicationsturbomachinesproject
 
IT8073 _Information Security _UNIT I Full notes
IT8073 _Information Security _UNIT I Full notesIT8073 _Information Security _UNIT I Full notes
IT8073 _Information Security _UNIT I Full notesAsst.prof M.Gokilavani
 
Haifa University Student Centre .pptxsaa
Haifa University Student Centre .pptxsaaHaifa University Student Centre .pptxsaa
Haifa University Student Centre .pptxsaassuser1cd9c9
 
B111_FTS_2011.06.01 Fuel Tank Safety.pdf
B111_FTS_2011.06.01 Fuel Tank Safety.pdfB111_FTS_2011.06.01 Fuel Tank Safety.pdf
B111_FTS_2011.06.01 Fuel Tank Safety.pdfKhoiTruong19
 
SOLAR PANEL POWERED BATTERY E VEHICLE....
SOLAR PANEL POWERED BATTERY E VEHICLE....SOLAR PANEL POWERED BATTERY E VEHICLE....
SOLAR PANEL POWERED BATTERY E VEHICLE....DeepakK547422
 
HB Self-Body characteristics UHV understanding
HB Self-Body characteristics UHV understandingHB Self-Body characteristics UHV understanding
HB Self-Body characteristics UHV understandingLeoRaju4
 
Super Duplex Steel 32750 Fasteners, UNS S32750 Fasteners
Super Duplex Steel 32750 Fasteners, UNS S32750 FastenersSuper Duplex Steel 32750 Fasteners, UNS S32750 Fasteners
Super Duplex Steel 32750 Fasteners, UNS S32750 Fastenersfastenersboltss
 
Checklist to troubleshoot CD moisture profiles.docx
Checklist to troubleshoot CD moisture profiles.docxChecklist to troubleshoot CD moisture profiles.docx
Checklist to troubleshoot CD moisture profiles.docxNoman khan
 
Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...AshishChanchal1
 
TYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUM
TYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUMTYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUM
TYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUMupamatechverse
 
Basic Concepts of Material Science for Electrical and Electronic Materials ...
Basic Concepts of Material Science for  Electrical and Electronic Materials  ...Basic Concepts of Material Science for  Electrical and Electronic Materials  ...
Basic Concepts of Material Science for Electrical and Electronic Materials ...PeopleFinder
 

Recently uploaded (20)

Présentation IIRB 2024 Prévibest T. Leborgne
Présentation IIRB 2024 Prévibest T. LeborgnePrésentation IIRB 2024 Prévibest T. Leborgne
Présentation IIRB 2024 Prévibest T. Leborgne
 
fat and edible oil processsing.ppt, refining
fat and edible oil processsing.ppt, refiningfat and edible oil processsing.ppt, refining
fat and edible oil processsing.ppt, refining
 
Renewable sources to produce electricity , Wind Energy
Renewable sources to produce electricity , Wind EnergyRenewable sources to produce electricity , Wind Energy
Renewable sources to produce electricity , Wind Energy
 
Forged Fitting Socket Welding Standard- ASME-B16.11-2001.pdf
Forged Fitting Socket Welding Standard- ASME-B16.11-2001.pdfForged Fitting Socket Welding Standard- ASME-B16.11-2001.pdf
Forged Fitting Socket Welding Standard- ASME-B16.11-2001.pdf
 
Unit_1.pdf computer networks and computer topology
Unit_1.pdf computer networks and computer topologyUnit_1.pdf computer networks and computer topology
Unit_1.pdf computer networks and computer topology
 
Amplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation TechniquesAmplitude modulation and Demodulation Techniques
Amplitude modulation and Demodulation Techniques
 
21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS
21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS
21 Scheme_21EC53_MODULE-5_CCN_Dr. ShivaS
 
A tale for cyber security phishing by Rima daqch
A tale for cyber security phishing by Rima daqchA tale for cyber security phishing by Rima daqch
A tale for cyber security phishing by Rima daqch
 
Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!
Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!
Robust, Precise, Fast - Chose Two for Radiated EMC Measurements!
 
Drone design for agricultural applications
Drone design for agricultural applicationsDrone design for agricultural applications
Drone design for agricultural applications
 
IT8073 _Information Security _UNIT I Full notes
IT8073 _Information Security _UNIT I Full notesIT8073 _Information Security _UNIT I Full notes
IT8073 _Information Security _UNIT I Full notes
 
Haifa University Student Centre .pptxsaa
Haifa University Student Centre .pptxsaaHaifa University Student Centre .pptxsaa
Haifa University Student Centre .pptxsaa
 
B111_FTS_2011.06.01 Fuel Tank Safety.pdf
B111_FTS_2011.06.01 Fuel Tank Safety.pdfB111_FTS_2011.06.01 Fuel Tank Safety.pdf
B111_FTS_2011.06.01 Fuel Tank Safety.pdf
 
SOLAR PANEL POWERED BATTERY E VEHICLE....
SOLAR PANEL POWERED BATTERY E VEHICLE....SOLAR PANEL POWERED BATTERY E VEHICLE....
SOLAR PANEL POWERED BATTERY E VEHICLE....
 
HB Self-Body characteristics UHV understanding
HB Self-Body characteristics UHV understandingHB Self-Body characteristics UHV understanding
HB Self-Body characteristics UHV understanding
 
Super Duplex Steel 32750 Fasteners, UNS S32750 Fasteners
Super Duplex Steel 32750 Fasteners, UNS S32750 FastenersSuper Duplex Steel 32750 Fasteners, UNS S32750 Fasteners
Super Duplex Steel 32750 Fasteners, UNS S32750 Fasteners
 
Checklist to troubleshoot CD moisture profiles.docx
Checklist to troubleshoot CD moisture profiles.docxChecklist to troubleshoot CD moisture profiles.docx
Checklist to troubleshoot CD moisture profiles.docx
 
Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...Series of training sessions by our experts for you to provide necessary insig...
Series of training sessions by our experts for you to provide necessary insig...
 
TYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUM
TYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUMTYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUM
TYPES OF PROPAGATION FOR TRANSMISSION OF DATA IN UNGUIDED MEDIUM
 
Basic Concepts of Material Science for Electrical and Electronic Materials ...
Basic Concepts of Material Science for  Electrical and Electronic Materials  ...Basic Concepts of Material Science for  Electrical and Electronic Materials  ...
Basic Concepts of Material Science for Electrical and Electronic Materials ...
 

Ignite (10m) how to not burn out your monitoring team

Editor's Notes

  1. before we begin, lets all relax a bit, dim the lights
  2. Pagerduty Alert - add alert
  3. What is this nonsense ?
  4. Find something not worth waking me up over.
  5. Find some app sending nonsense errors
  6. Why: lower capacity when starting equipment etc. External cost to other engineers
  7. 1st on-call
  8. 10th on call if nobody is dying leave me alone
  9. Why mandatory: people don’t allocate rest enough for themselves. A person would simply accumulate more vacation days.
  10. Dedicated time in the week to prevent things from waking us up at night. Simple fixes, even things like a jenkins job to do reboots of components. At similarWeb, production engineers spend their on-call week only working on ease-of-operation related issue.
  11. Alert all the things! Why this is bad: boy who cried wolf syndrome, alert fatigue
  12. Attention is a depletable resource
  13. Alert everyone! why this is bad tragedy of the commons: everyone ends up turning off their phones thinking “someone else will take of this”
  14. only people who can fix the problem get alerted, others get emails system needs to be smart enough to make the choice, and fixed when it makes a mistake in waking up the wrong person
  15. Log nothing Log to /dev/null Log to ephemeral storage
  16. measuring the wrong things. hosts are irrelevant, so is cpu and memory.
  17. mention multi AZ
  18. Tweet at me! Tell me your monitoring horror stories!