Mean Time to Sleep: Quantifying the On-Call Experience

@lozzd • @ryan_frantz
Mean Time to Sleep
Quantifying the on-call experience

Laurie Denness
@lozzd
Ryan Frantz
@ryan_frantz

Who is in an on-call rotation?

Who is on call right now?

Who feels like on-call sucks?

Let’s help our people sleep

Make on-call more
bearable

Incremental Changes

Email to
Acknowledge

Email to Acknowledge
• Replying “ack” with some context makes it appear in
IRC too

Email Only Alerts
• Do you care if RAID becomes degraded in the middle of
the night?

Email Only Alerts
the night?
• Do you care if one of your web/hadoop/X boxes dies in
the middle of the night?

Email Only Alerts
the night?
• Do you care if one of your web/hadoop/X boxes dies in
the middle of the night?
• Can it wait until the morning?

Added Context
• Previous service state
• Duration in that state

Added Context

Added Context
• Alert recipients

Added Context
• Notes

Added Context
• Notes
• Link to Runbook

Added Context
• Notes
• Link to runbook

Alert Storms
• Reduce noise when 200 things go wrong by aggregating

Alert Storms
• Reduce noise when 200 things go wrong by aggregating
• Trigger alert percentage of pool over threshold

Low friction downtime
• IRC commands to downtime hosts/sets of hosts

Downtime Reminders
• Help prevent false notifications

Event Handlers
• Teach Nagios to augment the team

Event Handlers
• Restarting services (nscd)

Event Handlers
• Re-running jobs (transient errors)

Event Handlers
• Re-running jobs (transient errors)
• Duplicate crons (Chef)

Incremental Improvements?
• Maybe

• Maybe
• More ideas; hoped they’d stick

• Maybe
• More ideas; hoped they’d stick
• We didn’t know because we didn’t measure

Measure Everything
• “You can’t manage what you can’t measure.”
- Deming (not really)

Measure Everything
• “You can’t manage what you can’t measure.”
- Deming (not really)
• But, we weren’t measuring anything

What should we measure?

• Volume of alerts (total, by severity)

• Alert categorization (actionable vs not)

• Alert times: Off-hours?

• Alert times: Off-hours?
• Noisy hosts/services

Opsweekly

@lozzd • @ryan_frantzWe have data.

Aggregate alerts
1. Look at reports

Aggregate alerts
1. Look at reports
2.Wow, look at all those alerts for the same thing

Aggregate alerts
1. Look at reports
3.Aggregate alerts

Aggregate alerts
1. Look at reports
3.Aggregate alerts
4.Profit

Parent relationships
• Prevent alerts due to upstream issues (downed switch)

• Standard Nagios feature

• Standard Nagios feature
• Computers can do this for us!

• signalvnoise.com

• LLDP on host shows switch info

• Put switch info into Chef using ohai

• Put switch info into Chef using ohai
• Create Nagios host configs based on data

Service Dependencies
• Hundreds of Graphite-sourced checks

Service Dependencies
• Hundreds of Graphite-sourced checks
• Created new template that sets a servicegroup that
depends on the Graphite service.

Keep on analyzing
• It’s okay to just identify and delete alerts that don’t
mean anything!

Keep on analyzing
• It’s okay to just identify and delete alerts that don’t
mean anything!
• Or move them to email only

More Quantification!

Reviewing the Year
• Use reports

Reviewing the Year
• Use reports
• Use search

Reviewing the Year
• Use reports
• Use search
• Identify noisiest alerts

Reviewing the Year
YEARLY REPORT SCREENSHOTS

• Great time to look at this data and make improvements
Nagios Hack Day/Week

• Great time to look at this data and make improvements
• If Disk Space is the worst. Can we rethink that?
Nagios Hack Day/Week

Outsource Your Alerts
• Etsy’s Search Team has on-call rotation

• A whole subset of alerts that don’t go to Ops

• A whole subset of alerts that don’t go to Ops
• More teams starting this but Search Team is at 100%

Sleep Tracking

Did it work?

Did it work?
• Yes.

Did it work?
• Yes.
• Signal to noise ratio is much better

Did it work?
• Yes.
• Okay, so it’s a little more complicated than that

Did it work?
• Yes.
• Adding alerts all the time means new “annoying” things

Did it work?
• Yes.
• Adding alerts all the time means new “annoying” things
• Keep monitoring

What’s next?

• We focus on people’s sleep
The Effect of Sleep

• But not the effect on the person when they come to
work the next day
The Effect of Sleep

work the next day
• How do we measure the impact of sleep loss/
deprivation?
The Effect of Sleep

work the next day
• How do we measure the impact of sleep loss/
deprivation?
The Effect of Sleep
• Subjective: Pittsburgh Sleepiness Scale
• Objective: Psychomotor vigilance task (PVT) to measure
alertness

Beyond Opsweekly
• Employee wellness program

Beyond Opsweekly
• Employee wellness program
• Security have started using past sleep data to check for
weird logins to systems

More context: nagios-herald

More reports
• We have a bunch of data, we can build better reports,
drill down to analyze alerting trends

More reports
• We have a bunch of data, we can build better reports,
drill down to analyze alerting trends
• Can we attribute particular actions to reduced noise
volume?
• Aggregate alerts
• Non-downtimed alerts

Thanks

Etsy Ops Team

SewMona

Open Source/Links
• http://ryanfrantz.com/mtts
• https://github.com/etsy/opsweekly
• https://github.com/etsy/nagios-herald
• https://github.com/jonlives/jawboneup_to_graphite
• http://codeascraft.com

Questions?

Mean Time to Sleep: Quantifying the On-Call Experience

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (9)

Similar to Mean Time to Sleep: Quantifying the On-Call Experience

Similar to Mean Time to Sleep: Quantifying the On-Call Experience (16)

Recently uploaded

Recently uploaded (20)

Mean Time to Sleep: Quantifying the On-Call Experience