Cassandra is the dominant data store used at Netflix and it's health is critical to many of its services. In this talk we will share details of the recent redesign of our health monitoring system and how we leveraged a reactive stream processing system to give us a real-time view our entire fleet while dramatically improving accuracy and reducing false alarms in our alerting. About the Speaker Jason Cacciatore Senior Software Engineer, Netflix Jason Cacciatore is a Senior Software Engineer at Netflix, where he's been working for the past several years. He's interested in stateful distributed systems and has a diverse background in technology. In his spare time he enjoys spending time with his wife and two sons, reading non-fiction, and watching Netflix documentaries.

1. Monitoring C* Health @ Scale
Jason Cacciatore

2. Netflix Scale
Hundreds of clusters
Thousands of nodes

3. How do we assess health ?
• Node Level
– dmesg errors
– gossip status
– thresholds of system metrics (disk usage, heap, etc)
• Cluster Level
– Rely on C*’s view of its health (nt ring)
– AWS Cache as secondary source of truth

4. Common Approach
CRON System
Job
RunnerJob
RunnerJob
RunnerJob
Runner

5. Common Scenario
Node disappears
T0
T1
T2
T3
T4

6. Problems inherent in polling
● Point-in-time snapshot, no state
● Establishing a connection to a cluster when it’s
under heavy load is problematic
● Not resilient to network hiccups, especially for
large clusters

7. A different approach
What if we had a continuous stream of
fine-grained snapshots ?

8. Mantis Streaming System
Stream processing system built on Apache Mesos
– Provides a flexible programming model
– Models computation as a distributed DAG
– Designed for high throughput, low latency
– Open Source date soon

9. Streaming micro-services
Source
Stage
Sink
Source
Stage
Sink
Source
Stage
Sink
Source - input, handles backpressure
Stage - business logic
Sink - output, handles backpressure
Source
Stage
Sink
Mantis
Job

10. Mantis Programming Model
• ReactiveX observable sequences
• Source, Stage, and Sink together form an
observable chain (which only emits data when
subscribed to)

11. Health Check using Mantis
Source
Job
Local
Ring
Agg
Global
Ring
Agg
Source
Job
Source
Job
eu-west-1us-east-1us-west-2
Local
Ring
Agg
Local
Ring
Agg

12. How much data ?
● Each node sends data every 20 seconds
● Payload size depends on cluster size
● ~6 MB/s total across east, west, and eu sent to
Local Ring Aggregators
● ~600Kbps processed by Global Aggregator

13. Local Ring Aggregator
• Stateless
• Single instance per region
• Groups data by C* cluster and scores it

14. Local Ring Aggregator (cont)
@Override
public Observable<String> call(Context context, Observable<MantisServerSentEvent> o) {
...
return
...
.filter(this::isValid)
.map(NodeRingMessage::filterByOwnership)
.buffer(config.getWindowInMillis(), TimeUnit.MILLISECONDS, 5000)
.map((nodeRingMessageList) -> new AggregatedView(nodeRingMessageList, config.getAWSClient()).score())
.flatMap((score) -> score)
.map(gson::toJson);

15. Anatomy of a Score
● Evidence - aggregate of all data points gathered
from all nodes
● AWS view - each instance in the cluster
● Cluster metadata (token to IP mapping, name, etc)

16. Global Ring Aggregator
Score
Score
Score
ScoreScore
ScoreScore
Grouped
ScoreScoreScoreScoreScoreScoreScoreScoreScoreScoreScoreScore
window

17. Global Ring Aggregator (cont)
reduce
Cluster
Health
Evaluator
SINK
Real Time
Dashboard
Score
Remediation
System
transform

18. Common Scenario revisited
T0
Cluster Health Evaluator
T1
Score
FSM
[ Start ]
T2
Score
FSM
[ Node Gone ]
T3
Score
FSM
[ Wait for Signal ]
T4
<Not tracked>
FSM
[ Wait for Signal ]
<Not tracked>
FSM
[ Done ]

19. That’s great, but...
Now the health of the fleet is encapsulated in a
single data stream, so how do we make sense of
that ?

20. Real Time Dash (Macro View)
Macro View of the fleet

21. Real Time Dash (Cluster View)

22. Real Time Dash (Perspective)

23. Benefits
● Faster detection of issues
● Greater accuracy
● Massive reduction in false positives
● Operational - hosted by Mantis infrastructure
● Separation of concerns (decouples detection from
remediation)

24. Monitoring the Monitor
● Mantis SLA
● Bytes read + written
● Incoming message count
● Sink processed + dropped counts
● CPU, memory

25. Questions ?