FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic

Wake Up!
A New Day Dawns for Alerts in New Relic
Bill Kayser

Wednesday, November 6, 13

You are always monitoring
all the things.


Welcome
Bill Kayser
@bravoking
bkayser@newrelic.com

Covered in this talk:
Overview of alerting in New Relic
Alert Policies, other new features
Tips for getting the most out of alerts


What do we expect
from alerting systems?


Photo credit: U.S. Coast Guard / Foter / CC BY-NC-ND


Moehre1992 CC BY-SA 3.0

Randy Chiu CC BY 2.0

Zinnmann CC BY-SA 3.0

What to Watch


Define Conditions


Define Conditions

Applications


Define Conditions
Key
Transactions

Applications


Define Conditions
Key
Transactions

Servers
Applications


What Action to Take


What Action to Take

37 Signals’ Campﬁre


What Action to Take


What Action to Take

Mobile Alerts


What Action to Take

Notification
Channels

Mobile Alerts



Thresholds and Notifications - Today


New in New Relic:
Alert Policies


New Feature: Alert Policies
An Alert Policy specifies:
What to watch
What action to take


Alert Policies


Alert Policies
Policies apply to:
Applications
Key Transactions
Servers


Alert Policies

Threshold settings
and other
configuration


Alert Policies

Threshold settings
and other
configuration


Notification
channels

Alert Policies

ops-puppet-01

Server policies are
mapped to servers


Threshold settings
and other
configuration

Notification
channels

Application Alert Policy


Server Alert Policy


Key Transaction Alert Policy


Editing Alert Policies


Alert Policies

Named Policies


Alert Policies
Default Policy


Default Policies


Default Policies

Applications with defaults


Default Policies


Default settings

Mapping Policies


Defining Channels


Channel Settings
Define distinct
endpoints for each
channel type


Channel Settings
Define distinct
endpoints for each
channel type

Designate channels to
be used strictly for
downtime alerts


Notification Groups
Define a set of notification
channels for use across multiple
policies
Include e-mail addresses,
PagerDuty services, HipChat
rooms, etc


Policies: Putting it All Together


Examining Alerts


What’s the Problem?
Problems are events
Critical: tell me right away


What’s the Problem?
Problems are events
Critical: tell me right away
Caution: note it in supporting detail


Alert

Alerts in the Event List


Alert History


Tips for Leveraging
New Alerts


‘Alarm fatigue’
worries hospitals
“...a patient’s life could be hanging in the balance.”
San Francisco Chronicle, October 23, 2013


Avoid Threshold Sensitivity
Increase the lag time for
triggering problems


Avoid Threshold Sensitivity

Increase the lag time for triggering downtime


Improve Threshold Sensitivity
Add substring requirement to downtime settings


Improve Threshold Sensitivity
Apdex Score: 0.84 - 0.91

T = 800 ms


Mean Response Time

Narrow the Scope
Define Key Transactions
Watch error rate and apdex
Customize apdex T value


Key Transaction Policies


Ignore Transaction Noise
Java: NewRelic.ignoreTransaction(),
@NewRelicIgnoreTransaction

Ruby: newrelic_ignore [ :only => action..]
[ :except => action...]

DotNet: NewRelic.Api.Agent.NewRelic.IgnoreTransaction();
PHP: newrelic_ignore_transaction ( )
Python: newrelic.agent.ignore_transaction(flag=True)


Stop waking me up


Stop waking me up
(Wake up Jonathan instead)


Alerts API


Alerts V2: Summary of Improvements
Alert Policies
Notification Channels
Notification Groups
Server Downtime
Lag time setting
New API


What do we expect from alerting?
Get the right people involved at the
appropriate time
Use well defined notification channels
Group channels together for easy re-use
Take advantage of PagerDuty

High signal, low noise
Alert on key transactions
Edit lag times
Distinguishing critical and caution problems
Adjusting apdex T


What do we expect from alerting
Flexibility
Define policies according to specific operational
requirements
Use defaults, define overrides
API for integration, management and auditing

Simplicity
Watch only the things you care about most


Migrating to use Alert Policies
Accounts will be upgraded in November
One policy will be created for each application and key transaction
One policy will be created for each unique set of threshold settings across all servers

Existing alert behavior is preserved


Thanks!
Bill Kayser
bkayser@newrelic.com
@bravoking


FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (10)

More from New Relic

More from New Relic (20)

Recently uploaded

Recently uploaded (20)

FUTURESTACK13: Wake Up! A New Day Dawns for Alerts in New Relic from Bill Kayser, Distinguished Engineer at New Relic