Measure Twice, Alert Once (or Less)

Measure Twice, Alert Once (or Less)
How Priceline Leverages Catchpoint

2
© Catchpoint. Confidential. All rights reserved.
priceline.com is part of the Priceline
Group – sister companies are Agoda,
Booking.com, Kayak, Opentable,
Rentalcars.com
priceline.com has been using Catchpoint
since 2010; several sister companies have
since joined us
Your Presenter
Tom Gerhard
Technical Fellow (Logging and monitoring), priceline.com

3
5 Years
Record-level data storage for in-
depth historical analysis and
flexible, multi-dimensional analytics
One Focus
To help businesses deliver
amazing end user experiences
Best Analytics
Deepest, fastest analytics, with raw data
to slice and dice on any dimension and
pre-built or customizable visualizations
to consume data your preferred way
700+
Vantage points comprising the
world’s largest footprint to test
where your end users are, across
geographies, consumer ISPs,
mobile, and backbone networks
440+
Customers including leaders like
Google, Honeywell, Qualtrics,
Priceline, Kate Spade, Comcast,
Zscaler, and LinkedIn
HQ
In New York, NY
(Offices in LA, Seattle, Boston,
London, Hamburg, and Bangalore)
50+
Vantage points in China’s 20
largest cities, on all 4 major ISPs
and top cloud providers
18
Monitor types – the most
extensive monitor types for
complete coverage
About Catchpoint

4
Extensive Monitor Types for Complete Coverage

5
.
Intro/Overview
How we use Catchpoint
• Identify key pages (and infrastructure, such as DNS)
• Create tests, alerts, and reports
• Sit back and enjoy (maybe not!)
That’s not what this talk is about!
We’re going to discuss some non-traditional applications for synthetic
monitoring.

6
Situation: We do our primary alerting from a set of backbone nodes.
When they alert, the #1 question is “Is it us, or the internet?”
Solution: “First mile” nodes; one in each of our data centers.
Run the same test on each of the first mile nodes, using overrides to
route the test to the local (to the node) data center.
When an alert fires, it’s a simple matter to compare the backbone results
with the first mile test.
#1 First Mile Baseline Tests

7
First Mile Node

8
Test Setup

9

10

11
If you have third-party objects, they won’t observe your overrides; if your test
results depend on them, they could go south.
Consider using zones and alerting only on your core zone(s)
What can go Wrong?
(Settings -> Zones)

12
Inadvertent testing of your outbound connection
• Most likely will be obvious if your first mile tests fail and your backbone
tests don’t
• If they all fail, you probably have a real issue, and the first mile tests have
given you additional data points
You don’t have an Enterprise node? Consider using a node close to your
data center.
What Can Go Wrong?

13
Situation: Infrastructure changes are being contemplated. What will the
impact be?
Solution: Run variant tests using overrides.
Examples:
• CDN Evaluation
• Measure impact of third-party content
• Worse case testing of non-responding objects
#2 What-if Scenarios

14
Change in CDN configuration, comparison of CDNs, etc.
#2a CDN Evaluation
Your CDN’s DNS alias or IP

15
Example: Block google publisher tags:
#2b Impact of Third-Party Content
… or delay using (Request Delay)

16
Move specific objects to another domain (perhaps a CDN?)
#2c Other Overrides
Can also use ${randomlist(a,b)} macro to substitute from a list
CDN’s DNS alias or IP

17
Situation: You’d like to enable all the add-on features, but worry about point
consumption, and you only need the extra information occasionally
Solution: Run a stripped-down test at your alerting frequency and a shadow
test on fewer nodes and/or less frequently with additional features (screen
shots, filmstrips, headers, etc.)
#3 Test Budget Management

18
Situation: you have internal or external customers with a Service Level
Agreement (SLA)
Solution: use test configured to provide the required data and provide a
daily/weekly/monthly report as needed.
#4 SLA Reporting

19
Problem: You want to avoid the “frog in a pot of cold water” effect.
Performance may degrade at a slow, but steady pace that doesn’t trip alerts
based on breakouts from patterns.
Solution: Define KPIs and set up alerts with absolute values for them, or
periodically review long-term trends.
#5 Long-term Trending

20
Example: Boiling a Frog?
Test name

21
Catchpoint Insight: To learn it is to love it
Favorite use cases:
• Identify hosts behind the load balancer
• Capture host- and load balancer-reported response times
• Capture CDN routing information
• Use Javascript to capture specialized metrics
Free Advice

22
Get the most out of your deployment by taking advantage of integrations that are available.
Free Advice

23
Our favorites:
Slack – Alert webhook API
Splunk – Use the Catchpoint Splunk add-on to pull test data for key tests into
Splunk so it can be combined with other data for display.
Single Sign-on – We use Okta to manage access to the Catchpoint portal,
simplifying administration and compliance.
Free Advice

24
Your monitoring and alerting strategy needs a holistic view of your operation,
from as many perspectives as possible.
None of these techniques, by themselves, is a silver bullet. It’s important to
expect that something not visible from one perspective may be revealed from
another, and that they can also act as sanity checks against each other.
How does it all stack up?

Any Questions?

Measure Twice, Alert Once (or Less)

Recommended

Recommended

More Related Content

Similar to Measure Twice, Alert Once (or Less)

Similar to Measure Twice, Alert Once (or Less) (20)

More from DevOps.com

More from DevOps.com (20)

Recently uploaded

Recently uploaded (20)

Measure Twice, Alert Once (or Less)