• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic
 

FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic

on

  • 1,139 views

The alerts in New Relic have always been an essential feature for our customers, starting in the early days when we watched only web applications through the evolution of availability monitoring and ...

The alerts in New Relic have always been an essential feature for our customers, starting in the early days when we watched only web applications through the evolution of availability monitoring and server alerting. Coming next, though, is the biggest improvement to alerting yet. In this talk we'll cover the latest features with special insights and pro-tips, as well as an inside look at the new policy based features. Get ready to take your alerts to a whole new level.

Statistics

Views

Total Views
1,139
Views on SlideShare
1,139
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic Presentation Transcript

    • Wake Up! A New Day Dawns for Alerts in New Relic Bill Kayser Wednesday, November 6, 13
    • You are always monitoring all the things. Wednesday, November 6, 13
    • Welcome Bill Kayser @bravoking bkayser@newrelic.com Covered in this talk: Overview of alerting in New Relic Alert Policies, other new features Tips for getting the most out of alerts Wednesday, November 6, 13
    • What do we expect from alerting systems? Wednesday, November 6, 13
    • Photo credit: U.S. Coast Guard / Foter / CC BY-NC-ND Wednesday, November 6, 13
    • Moehre1992 CC BY-SA 3.0 Wednesday, November 6, 13
    • Randy Chiu CC BY 2.0 Wednesday, November 6, 13
    • Zinnmann CC BY-SA 3.0 Wednesday, November 6, 13
    • What to Watch Wednesday, November 6, 13
    • What to Watch Wednesday, November 6, 13
    • What to Watch Wednesday, November 6, 13
    • What to Watch Wednesday, November 6, 13
    • What to Watch Wednesday, November 6, 13
    • Define Conditions Wednesday, November 6, 13
    • Define Conditions Applications Wednesday, November 6, 13
    • Define Conditions Key Transactions Applications Wednesday, November 6, 13
    • Define Conditions Key Transactions Servers Applications Wednesday, November 6, 13
    • What Action to Take Wednesday, November 6, 13
    • What Action to Take Wednesday, November 6, 13
    • What Action to Take Wednesday, November 6, 13
    • What Action to Take 37 Signals’ Campfire Wednesday, November 6, 13
    • What Action to Take 37 Signals’ Campfire Wednesday, November 6, 13
    • What Action to Take 37 Signals’ Campfire Mobile Alerts Wednesday, November 6, 13
    • What Action to Take 37 Signals’ Campfire Mobile Alerts Wednesday, November 6, 13
    • What Action to Take Notification Channels Mobile Alerts Wednesday, November 6, 13 37 Signals’ Campfire
    • Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • Thresholds and Notifications - Today Wednesday, November 6, 13
    • New in New Relic: Alert Policies Wednesday, November 6, 13
    • New Feature: Alert Policies An Alert Policy specifies: What to watch What action to take Wednesday, November 6, 13
    • Alert Policies Wednesday, November 6, 13
    • Alert Policies Policies apply to: Applications Key Transactions Servers Wednesday, November 6, 13
    • Alert Policies Wednesday, November 6, 13
    • Alert Policies Threshold settings and other configuration Wednesday, November 6, 13
    • Alert Policies Threshold settings and other configuration Wednesday, November 6, 13 Notification channels
    • Alert Policies ops-puppet-01 Server policies are mapped to servers Wednesday, November 6, 13 Threshold settings and other configuration Notification channels
    • Application Alert Policy Wednesday, November 6, 13
    • Server Alert Policy Wednesday, November 6, 13
    • Key Transaction Alert Policy Wednesday, November 6, 13
    • Editing Alert Policies Wednesday, November 6, 13
    • Editing Alert Policies Wednesday, November 6, 13
    • Editing Alert Policies Wednesday, November 6, 13
    • Editing Alert Policies Wednesday, November 6, 13
    • Alert Policies Named Policies Wednesday, November 6, 13
    • Alert Policies Default Policy Wednesday, November 6, 13
    • Default Policies Wednesday, November 6, 13
    • Default Policies Applications with defaults Wednesday, November 6, 13
    • Default Policies Wednesday, November 6, 13 Default settings
    • Mapping Policies Wednesday, November 6, 13
    • Mapping Policies Wednesday, November 6, 13
    • Defining Channels Wednesday, November 6, 13
    • Defining Channels Wednesday, November 6, 13
    • Defining Channels Wednesday, November 6, 13
    • Defining Channels Wednesday, November 6, 13
    • Channel Settings Define distinct endpoints for each channel type Wednesday, November 6, 13
    • Channel Settings Define distinct endpoints for each channel type Designate channels to be used strictly for downtime alerts Wednesday, November 6, 13
    • Notification Groups Define a set of notification channels for use across multiple policies Include e-mail addresses, PagerDuty services, HipChat rooms, etc Wednesday, November 6, 13
    • Policies: Putting it All Together Wednesday, November 6, 13
    • Policies: Putting it All Together Wednesday, November 6, 13
    • Policies: Putting it All Together Wednesday, November 6, 13
    • Policies: Putting it All Together Wednesday, November 6, 13
    • Policies: Putting it All Together Wednesday, November 6, 13
    • Examining Alerts Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • What’s the Problem? Problems are events Critical: tell me right away Wednesday, November 6, 13
    • What’s the Problem? Problems are events Critical: tell me right away Caution: note it in supporting detail Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Alert Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Alerts in the Event List Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Alert History Wednesday, November 6, 13
    • Tips for Leveraging New Alerts Wednesday, November 6, 13
    • ‘Alarm fatigue’ worries hospitals “...a patient’s life could be hanging in the balance.” San Francisco Chronicle, October 23, 2013 Wednesday, November 6, 13
    • Avoid Threshold Sensitivity Increase the lag time for triggering problems Wednesday, November 6, 13
    • Avoid Threshold Sensitivity Increase the lag time for triggering problems Wednesday, November 6, 13
    • Avoid Threshold Sensitivity Increase the lag time for triggering downtime Wednesday, November 6, 13
    • Improve Threshold Sensitivity Add substring requirement to downtime settings Wednesday, November 6, 13
    • Improve Threshold Sensitivity Apdex Score: 0.84 - 0.91 T = 800 ms Wednesday, November 6, 13 Mean Response Time
    • Narrow the Scope Define Key Transactions Watch error rate and apdex Customize apdex T value Wednesday, November 6, 13
    • Key Transaction Policies Wednesday, November 6, 13
    • Key Transaction Policies Wednesday, November 6, 13
    • Ignore Transaction Noise Java: NewRelic.ignoreTransaction(), @NewRelicIgnoreTransaction Ruby: newrelic_ignore [ :only => action..] [ :except => action...] DotNet: NewRelic.Api.Agent.NewRelic.IgnoreTransaction(); PHP: newrelic_ignore_transaction (  ) Python: newrelic.agent.ignore_transaction(flag=True) Wednesday, November 6, 13
    • Stop waking me up Wednesday, November 6, 13
    • Stop waking me up (Wake up Jonathan instead) Wednesday, November 6, 13
    • Stop waking me up (Wake up Jonathan instead) Wednesday, November 6, 13
    • Alerts API Wednesday, November 6, 13
    • Alerts API Wednesday, November 6, 13
    • Alerts API Wednesday, November 6, 13
    • Alerts API Wednesday, November 6, 13
    • Alerts API Wednesday, November 6, 13
    • Alerts V2: Summary of Improvements Alert Policies Notification Channels Notification Groups Server Downtime Lag time setting New API Wednesday, November 6, 13
    • What do we expect from alerting? Get the right people involved at the appropriate time Use well defined notification channels Group channels together for easy re-use Take advantage of PagerDuty High signal, low noise Alert on key transactions Edit lag times Distinguishing critical and caution problems Adjusting apdex T Wednesday, November 6, 13
    • What do we expect from alerting Flexibility Define policies according to specific operational requirements Use defaults, define overrides API for integration, management and auditing Simplicity Watch only the things you care about most Wednesday, November 6, 13
    • Migrating to use Alert Policies Accounts will be upgraded in November One policy will be created for each application and key transaction One policy will be created for each unique set of threshold settings across all servers Existing alert behavior is preserved Wednesday, November 6, 13
    • Wednesday, November 6, 13
    • Thanks! Bill Kayser bkayser@newrelic.com @bravoking Wednesday, November 6, 13