Cassandra is the dominant data store used at Netflix and it's health is critical to many of its services. In this talk we will share details of the recent redesign of our health monitoring system and how we leveraged a reactive stream processing system to give us a real-time view our entire fleet while dramatically improving accuracy and reducing false alarms in our alerting.
About the Speaker
Jason Cacciatore Senior Software Engineer, Netflix
Jason Cacciatore is a Senior Software Engineer at Netflix, where he's been working for the past several years. He's interested in stateful distributed systems and has a diverse background in technology. In his spare time he enjoys spending time with his wife and two sons, reading non-fiction, and watching Netflix documentaries.
3. How do we assess health ?
• Node Level
– dmesg errors
– gossip status
– thresholds of system metrics (disk usage, heap, etc)
• Cluster Level
– Rely on C*’s view of its health (nt ring)
– AWS Cache as secondary source of truth
6. Problems inherent in polling
● Point-in-time snapshot, no state
● Establishing a connection to a cluster when it’s
under heavy load is problematic
● Not resilient to network hiccups, especially for
large clusters
8. Mantis Streaming System
Stream processing system built on Apache Mesos
– Provides a flexible programming model
– Models computation as a distributed DAG
– Designed for high throughput, low latency
– Open Source date soon
10. Mantis Programming Model
• ReactiveX observable sequences
• Source, Stage, and Sink together form an
observable chain (which only emits data when
subscribed to)
11. Health Check using Mantis
Source
Job
Local
Ring
Agg
Global
Ring
Agg
Source
Job
Source
Job
eu-west-1us-east-1us-west-2
Local
Ring
Agg
Local
Ring
Agg
12. How much data ?
● Each node sends data every 20 seconds
● Payload size depends on cluster size
● ~6 MB/s total across east, west, and eu sent to
Local Ring Aggregators
● ~600Kbps processed by Global Aggregator
13. Local Ring Aggregator
• Stateless
• Single instance per region
• Groups data by C* cluster and scores it
14. Local Ring Aggregator (cont)
@Override
public Observable<String> call(Context context, Observable<MantisServerSentEvent> o) {
...
return
...
.filter(this::isValid)
.map(NodeRingMessage::filterByOwnership)
.buffer(config.getWindowInMillis(), TimeUnit.MILLISECONDS, 5000)
.map((nodeRingMessageList) -> new AggregatedView(nodeRingMessageList, config.getAWSClient()).score())
.flatMap((score) -> score)
.map(gson::toJson);
15. Anatomy of a Score
● Evidence - aggregate of all data points gathered
from all nodes
● AWS view - each instance in the cluster
● Cluster metadata (token to IP mapping, name, etc)