This talk is from Distributed Data Summit SF 2018 - http://distributeddatasummit.com/2018-sf/sessions#netflix2 Operating C* can involve a lot of required manpower, complex automation, or both. Some of this complexity comes from operational/configuration activity of the underlying kernel and hardware but much of it is operation complexity stemming from C* itself. Some examples of this complexity are restarting the database in a safe way, reliability backing up and restoring snapshots, monitoring the health of the datastore, and even ensuring eventual consistency through repair. As a result of these complexities, C* operators end up with complicated operational setups, which are expensive to build, manage and monitor. As part of this talk, we will share lessons learned in managing such complexity via our Priam sidecar including recent innovations in how our sidecar ensures the highest possible uptime and correctness of Cassandra. We then use this to motivate building in the management sidecar directly as part of C* itself (CASSANDRA-14395).

1. Looking towards
an Official C*
Sidecar
Joey Lynch and Vinay Chella
Cloud Database Engineering

2. Speakers
Joey Lynch
Senior Software Engineer
Cloud Database Engineering
Netflix
Distributed system addict and
data wrangler
Vinay Chella
Senior Software Engineer
Cloud Database Engineering
Netflix
Cassandra Specialist,
Distributed systems engineer
and creator of NdBench

3. ● Operating C*
● Operating C* with a sidecar
● State of community sidecars
● Lessons learned from Priam (Netflix’s main C* sidecar)
● Goals of C* management sidecar (CASSANDRA-14395)
Agenda

4. ● Bootstrap and data movement
● Configuration (files, jmx)
● Maintenance
● Monitoring/Metrics
● Backup/Restore
● Repair
Operating C*

5. Create a New Cluster
● Seeds
● Token assignment
Operating C*: Bootstrapping
Add/Remove/Replace
● Serial or parallel?
● Streaming?

6. Operating C*: Configuration
● Probably Have to Tune
a. cassandra.yaml
b. topology props
c. JVM options
● May Have to Tune
a. Logging
b. Incremental Backup
c. More JVM options

7. Operating C*: Lifecycle
Rolling Restarts (Upgrades)
● Semi-complex single
node procedure
● One at a time is too slow
● Token range aware
restarts?
What happens when
Cassandra dies?
Ring source @ https://v2.overleaf.com/read/zchtrzskkyjb

8. Operating C*: Maintenance
● All the Power of JMX
● … So many possibilities
a. Many work with jmxterm/jmxsh
b. Many only work with Java code
● What if you want to do it on all
nodes?

9. ● Many Metrics (good!)
● How to Collect Them?
○ JMX … no
○ Agent!
● Which agent ...
Operating C*: Monitoring

10. ● Cassandra ring health
depends on replication
● Strategies
○ Monitor replication of
keyspaces
○ Topology Aware
○ Maintenance Aware
Operating C*: Ring Health

11. Operating C*: Backup/Restore
The
Cloud
● What even do I need to backup!?
● Restore is legitimately tricky, do you practice?

12. “Eventually” Consistent
1. Partial Write
2. Read Repair
3. Hints play
… Nope not enough
4. Repair
Operating C*: Repair
N1 N2 N3
0 1 0
0 1 1
0 1 1
0 1 1
1 1 1
N4 N5 N6
0 0 0
0 0 0
0 1 0
0 1 0
1 1 1
Datacenter 1 Datacenter 2

13. Operating C* In General

14. Operating C* In General

15. Separate solutions for ...
● Bootstrap and data movement
● Maintenance
● Configuration (files, jmx)
● Monitoring/Metrics
● Backup/Restore
● Repair
So ... What is needed to use C*?

16. We need better tools!

17. Operating C* with Sidecar(s)

18. What’s a Sidecar?
Cassandra
metrics-agent
Priam
jvmkill
...
Sidecars Live Outside
Main Daemon Scope
● Often built for a specific
purpose
● Typically a different OS
process
● There isn’t going to be
“one” sidecar

19. Sidecar: Bootstrapping
Automatic Seed Management using ASGs/db
Automatic Instance Replacement
Equation+Graph from “Cassandra Availability with Virtual Nodes” by Joey Lynch and Josh Snyder

20. Sidecar: Configuration
● Hierarchy: Environment -> Cluster -> Node
● Flat namespace that is merged to provide Priam
config

21. Sidecar: Configuration
● Hierarchy
● Flat namespace that
is merged to provide
Priam config
● Functions for
defaults (e.g. based
on cpu)
prod
cass_nflx
i-08da5d...

22. Sidecar: Lifecycle
Fail
Healthcheck
Drain with
timeout
Execute Stop
Script
(systemd)

23. Sidecar: Lifecycle
Pass
Healthcheck
Execute Start
Script
(systemd)
Ensure
Health
Rolling Restarts (Upgrades)
● Cluster automation is now much easier
What happens when Cassandra dies?
● Continuous health monitoring and supervision (OOM)
● Priam + systemd + jvmkill1
== pretty good
1
https://github.com/airlift/jvmkill

24. Sidecar: Maintenance
● JMX methods on cron
● Can add arbitrary tasks like compactions, flushes, etc

25. Sidecar: Maintenance
● Sidecar provides JMX over HTTP
○ Cleanup
○ Invoke complex JMX methods
using curl
○ Many of these are better done
scheduled (e.g. repair,
compaction, flushes, etc)

26. Sidecar: Monitoring

27. Sidecar: Monitoring
Cassandra
metrics-agent
Metrics
Sink
Partition tolerant push
● Sidecar agent = scalable
● Non agent JMX metric
export … not even once
● Still a sidecar, just not
Priam

28. ● Sidecar monitoring local
node
● Can also look at the ring
○ Gossip state
● Can
○ Ask a few Priams
○ Export info to
streaming system
Sidecar: Ring Health

29. Sidecar: Backup/Restore
● Backup on cron schedule
○ Snapshot
● Parallel incrementals
● Verification
● Pluggable with code (S3, GCS, etc…)

30. Sidecar: Backup/Restore
(coming soon)
● Similar to cassandra-mirror1
● Keeps track of what files have already been uploaded
○ Unlocks minute level snapshot backups
○ Only uploads files once
● Continuous, point in time, self healing, eventually
consistent
1: https://github.com/hashbrowncipher/cassandra-mirror

31. Sidecar: Repair
(coming soon)
1: https://issues.apache.org/jira/browse/CASSANDRA-14346
● Always on
● Just works
● In Cassandra
itself? 1

32. State of Community Sidecars

33. State of Community Sidecars
Everyone builds their own

34. State of Community Sidecars

35. State of Community Tooling

36. State of Community Tooling 2

37. State of Community Tooling 3

38. State of Community Backup

39. And that’s not even all of ‘em
… but do they solve our
problems?

40. Puppet/
Chef
Terraform CRON Fabric /
SSH in a
for loop
Reaper Priam Jenkins Sensu/
Nagios
Bootstrap No Yes No Maybe No Yes Maybe No
Maintenance Yes Maybe Maybe Yes No Yes Yes No
Configuration Yes Maybe No Maybe No Yes Maybe No
Monitoring Maybe No Maybe Maybe No Yes Maybe Yes
Backup Maybe No Maybe Maybe Maybe Yes Yes No
Repair No No Sorta Yes Yes Soon Yes No
Easy to Use Yes No* Yes Yes Yes No Maybe Yes
Multi-Cloud Yes Yes Yes Yes Yes No Yes Yes
Does it Scale Yes Yes Yes Sorta Yes* Yes Sorta* Yes

41. Oops.

42. Lessons Learned
Netflix has operated Cassandra using Priam and other
sidecars for years
… We have learned a few things

43. ● Netflix OSS specific
Lessons learned from Priam

44. ● Netflix OSS specific
● Not easy to use externally
Lessons learned from Priam

45. ● Netflix OSS specific
● Not easy to use externally
● Multi-cloud?
Lessons learned from Priam

46. ● Netflix OSS specific
● Not easy to use externally
● Multi-cloud?
● ~3 different
implementations of
everything
Lessons learned from Priam

47. ● Netflix OSS specific
● Not easy to use externally
● Multi-cloud?
● ~3 different
implementations of
everything
● Version compatibility
Lessons learned from Priam

48. Lesson Learned: Control Plane
Orchestrator
Node1
Node2
Node3
1. Restart!
2. Restart!
3. Restart!
“Push Control”
● Relies on unreliable
communication (ssh < http)
● Hard to make eventually
consistent
● Hard to guarantee safety
● Very easy to implement

49. Lesson Learned: Control Plane
Orchestrator?
Node1
Node2
Node3
Restart C*
older than Y!
Desire State Store
(Cassandra, Config)
Anything to do?
Ah, time to
restart!
Is it safe?
“Pull Control”
● Write desires into state store (or config)
● Nodes coordinate through state store as needed
● Tradeoff: Hard to implement

50. Lesson Learned: Control Plane
Pull Push>>

51. 1. Easy to Use
Goals of C* management process

52. 2. Solves or eases most
common problems
Goals of C* management process

53. 3. Pluggable
Goals of C* management process

54. 4. Scalable
Goals of C* management process

55. Feedback on Scope/Design?
Management Process:
https://issues.apache.org/jira/browse/CASSANDRA-14395
Repair/Task Scheduler:
https://issues.apache.org/jira/browse/CASSANDRA-14346

56. Thank you.