Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spotify: Automating Cassandra repairs

2,521 views

Published on

Anti-entropy repairs are known to be a very peculiar maintenance operation of Cassandra clusters. They are problematic mostly because of the potential of having negative impact on the cluster's performance. Another problematic aspect is the difficulty of managing the repairs of Cassandra clusters in a careful way that would prevent the negative performance impact.

Based on the long-term pain we have been experiencing with managing repairs of nearly 100 Cassandra clusters, and being unable to find a solution that would meet our needs, we went ahead and developed an open-source tool, named Cassandra Reaper [1], for easy management of Cassandra repairs.

Cassandra Reaper is a tool that automates the management of anti-entropy repairs of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise.

I will have to cover some basics of eventual consistency mechanisms of Cassandra, after which I will be able to focus on the features of Cassandra Reaper and our six months of experience having the tool managing the repairs of our production clusters.

Published in: Technology

Spotify: Automating Cassandra repairs

  1. 1. Automating Cassandra Repairs Radovan Zvoncek zvo@spotify.com github.com/spotify/cassandra-reaper #CassandraSummit
  2. 2. About zvo
  3. 3. About zvo Likes pancakes
  4. 4. About zvo Likes pancakes Does this for the 3rd time
  5. 5. About zvo Likes pancakes Does this for the 3rd time Works at Spotify
  6. 6. Working at Spotify Is autonomous Squads responsible for their full stack Including Cassandra
  7. 7. Cassandra
  8. 8. Node’s data Cassandra
  9. 9. Replication Cassandra
  10. 10. Running Cassandra Requires many things One of them is keeping data consistent Otherwise it can get lost or reappear
  11. 11. Running Cassandra Requires many things One of them is keeping data consistent Eventually
  12. 12. Eventual consistency Cassandra
  13. 13. Eventual consistency Read Repairs Cassandra R W
  14. 14. Eventual consistency Hinted Handoff Cassandra
  15. 15. Eventual consistency Anti-entropy Repair Cassandra
  16. 16. Coordinated process Anti-entropy Repair
  17. 17. Coordinated process Four steps: Anti-entropy Repair
  18. 18. Coordinated process Four steps: 1: Hash Anti-entropy Repair # # #
  19. 19. Coordinated process Four steps: 1: Hash 2: Compare Anti-entropy Repair ###
  20. 20. Coordinated process Four steps: 1: Hash 2: Compare 3: Stream Anti-entropy Repair
  21. 21. Coordinated process Four steps: 1: Hash 2: Compare 3: Stream 4: Merge Anti-entropy Repair
  22. 22. Coordinated process Four steps: 1: Hash 2: Compare 3: Stream 4: Merge Can go wild... Anti-entropy Repair
  23. 23. Repair gone wild
  24. 24. Repair gone wild Eats a lot of disk IO ● because of hashing all the data
  25. 25. Repair gone wild Eats a lot of disk IO Saturates the network ● because of streaming a lot of data around
  26. 26. Repair gone wild Eats a lot of disk IO Saturates the network Fills up the disk ● because of receiving all replicas, possibly from all other data centers
  27. 27. Repair gone wild Eats a lot of disk IO Saturates the network Fills up the disk Causes a ton of compactions ● because of having to merge the received data
  28. 28. Repair gone wild Eats a lot of disk IO Saturates the network Fills up the disk Causes a ton of compactions … one better be careful
  29. 29. Careful repair
  30. 30. nodetool repair Careful repair All three intervals
  31. 31. Partitioner range ● nodetool repair -pr Careful repair This interval only
  32. 32. Start & end tokens ● nodetool repair -pr -st -et Careful repair A part of interval only
  33. 33. Requires splitting the ring into smaller intervals Smaller intervals mean less data Less data means fewer repairs gone wild Careful repair
  34. 34. Smaller intervals also mean more intervals More intervals mean more actual repairs Repairs need to be babysat :( Careful repair
  35. 35. The Spotify way Feature teams meant to do features Not waste time operating their C* clusters Cron-ing nodetool repair is no good ● mostly due to no feedback loop This all led to creation of the Reaper
  36. 36. The Reaper REST(ish) service Does a lot of JMX Orchestrates repairs for you
  37. 37. The reaping You: curl http://reaper/cluster --data ‘{“seedHost” : “my.cassandra.host.net”}’ The Reaper: ● Figures out cluster info (e.g. name, partitioner)
  38. 38. The reaping You: curl http://reaper/repair_run --data ‘{“clusterName”: “myCluster”}’ The Reaper: ● Prepares repair intervals
  39. 39. The reaping You: curl -X PUT http://reaper/repair_run/42 -d state=RUNNING The Reaper: ● Starts triggering repairs of repair intervals
  40. 40. Reaper’s features
  41. 41. Reaper’s features Carefulness - doesn’t kill a node ● checks for node load ● backs off after repairing an interval
  42. 42. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break ● because things break all the time
  43. 43. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes ● multiple small intervals in parallel
  44. 44. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes Scheduling - setup things only once ● regular full-ring repairs
  45. 45. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes Scheduling - setup things only once Persistency - state saved somewhere ● a bit of extra resilience
  46. 46. What we reaped First repair done 2015-01-28 1,700 repairs since then, recently 90 per week 176,000 (16%) segments failed at least once 60 repair failures
  47. 47. What we reaped
  48. 48. What we reaped
  49. 49. Reaper’s Future CASSANDRA-10070 Whatever is needed until then
  50. 50. Greatest benefit Cassandra Reaper automates a very tedious maintenance operation of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise github.com/spotify/cassandra-reaper #CassandraSummit
  51. 51. Thank you! github.com/spotify/cassandra-reaper #CassandraSummit

×