SignalFx is an advanced monitoring and alerting system for cloud applications delivered as SaaS. It provides real-time metrics, analytics, and tagging to monitor microservices architectures. Traditional monitoring approaches are noisy and reactive, while SignalFx aims to provide guided triage and correlate events using time series analytics to identify patterns and anomalies.

1. MONITORING MICROSERVICES
APRI L 2 6 , 2 0 1 7
Rich Schofield
rschofield@signalfx.com
Jamison Clouthier
jamison@signalfx.com

2. Karthik Rau, CEO
VMware, VP Products
Loudcloud/Opsware, Products
S E L E C T C U S T O M E R S I N V E S T O R S
Phillip Liu, CTO
Facebook, Lead Architect
Loudcloud/Opsware,
Chief Architect
F O U N D E R S
A LITTLE BACKGROUND
SignalFx is an advanced monitoring & alerting system for cloud apps, delivered as a SaaS solution

3. SIGNALFX HIGH LEVEL ARCHITECTURE
METADATA
1s / 5s
1m
1h
•••
MESSAGEBUS
TIME SERIES DATABASE
ANALYTICS
25
INGEST
NOTIFICATION
8d
384d
32d
1s 16m (IN MEMORY)
DATAPOINTS
MICROSERVICES

4. BENEFITS OF MICROSERVICES ARCHITECTURE
Application modules released independently
• Enables agile development per team
• Canary deployments for testing
• Containers to enable rapid automated deployment & rollback
Scale in/out with service load
• Respond quickly to increased data flow or usage
• Optimize infrastructure costs as load falls
Technology flexibility
• Easier to upgrade components or change platforms

5. MONITORING A MICROSERVICE APPLICATION
Real-time metrics and analytics
• Detect issues and trends quickly
Apply context: history, related events, service metadata
• Metrics without context are just numbers!
Tag-based monitoring for elastic/ephemeral services
• Adapt automatically to monitor services as they scale in/out
Shared, self-service access across all groups
• App developers, tech leads, support, management
• Avoid different tools for different teams

6. ISSUES WITH TRADITIONAL MONITORING
Noisy, reactive monitoring
C H A L L E N G E
• Too many alerts fire at once for a cluster-wide
problem
• Is the machine down because we scaled down
the cluster or because we had a real problem?
• Do we even care if a single node is down?
• Component-specific monitoring configurations
that require constant maintenance in
ephemeral/elastic environments
What
matters?
Where to
start?
?

7. THE MODERN MONITORING LANDSCAPE
A P M M E T R I C S L O G S
Performance Testing Pre-Flight Streaming Metrics Aggregated In-Flight Black Box Recorder Post-Flight
Luxury of TimeReal Time MattersLuxury of Time

8. LET’S TALK METRICS
Metric name
Metric value
Metric type
Timestamp
Dimensions
cpu.idle
27
gauge
1234567
host = relic47df
datacenter = sjc1
env = prod
…
Dimensions allow
you to filter,
aggregate, compare
across sources

9. METRICS IN A TIME SERIES
{
"gauge":
[{"metric":”cpu.idle",
"dimensions":
{"host":”hostname123",
"datacenter":”snc"},
"value":249}]
}
{
"gauge":
[{"metric":”cpu.idle",
"dimensions":
{"host":”hostname123",
"datacenter":”snc"},
"value":230}]
}
{
"gauge":
[{"metric":”cpu.idle",
"dimensions":
{"host":”hostname123",
"datacenter":”snc"},
"value":202}]
}
{
"gauge":
[{"metric":”cpu.idle",
"dimensions":
{"host":”hostname123",
"datacenter":”snc"},
"value":284}]
}
10:15:02 10:15:03 10:15:04 10:15:05

10. USING TIME SERIES ANALYTICS TO CORRELATE
AND IDENTIFY PATTERNS
How well load balanced is this
8-node Kafka cluster?
Compare the signal against
historical patterns and alert on
anomalous patterns
Create a signal to represent the
cluster’s load balancing
effectiveness, computed within
seconds

11. GUIDED TRIAGE
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
INFRASTRUCTURE SERVICE
PRODUCT LAUNCH
EVENT
CORRELATE EVENTS
TO MONITORING
IS THIS A PROBLEM?
ANALYSIS
1. PRODUCT LAUNCH
(EVENT RECORDED)
2. TRAFFIC SPIKE
3. TRANSIENT?
4. IMPACT EXISTING
CUSTOMERS?
5. IS IT LEVELING OUT?
6. RAM ISSUE?
7. STORAGE ISSUE?
8. BUS BACKED UP?

12. JOURNEY TO METRICS BASED MONITORING
PHASE 0 PHASE 2PHASE 1 PHASE 3
Health
checks and
logs
Small internal
metrics
system
Build out scalable,
highly-available metrics
system
Build out more
sophisticated
analytics
From individual component checks to proactive management of service-wide performance

13. M M / D D / Y Y
YOUR TITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R EDEMO

14. M M / D D / Y Y
YOUR TITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
T H A N K Y O U !
jamison@signalfx.com
rschofield@signalfx.com
S I G N U P F O R A T R I A L A T :
signalfx.com