Download as PDF, PPTX
Uploaded byDataStax Academy
Spotify: Automating Cassandra repairs
The document discusses the challenges of maintaining data consistency in Cassandra clusters and introduces the Cassandra Reaper, a tool developed by Spotify for automating repairs. The Reaper orchestrates the repair process, reduces disk I/O, and enhances repair efficiency while minimizing the need for manual oversight. Key features include carefulness to avoid node failures, resilience during errors, efficient parallelism, and persistent state management.
More Related Content
![[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"](https://cdn.slidesharecdn.com/ss_thumbnails/47openinfradaykorea2018hyun-ha-180705032301-thumbnail.jpg?width=640&height=640&fit=bounds)
Spotify: Automating Cassandra repairs
- 1.
- 2.
- 3.
- 4. About zvo Likes pancakes Doesthis for the 3rd time
- 5. About zvo Likes pancakes Doesthis for the 3rd time Works at Spotify
- 6. Working at Spotify Isautonomous Squads responsible for their full stack Including Cassandra
- 7.
- 8.
- 9.
- 10. Running Cassandra Requires manythings One of them is keeping data consistent Otherwise it can get lost or reappear
- 11. Running Cassandra Requires manythings One of them is keeping data consistent Eventually
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. Coordinated process Four steps: 1:Hash Anti-entropy Repair # # #
- 19. Coordinated process Four steps: 1:Hash 2: Compare Anti-entropy Repair ###
- 20. Coordinated process Four steps: 1:Hash 2: Compare 3: Stream Anti-entropy Repair
- 21. Coordinated process Four steps: 1:Hash 2: Compare 3: Stream 4: Merge Anti-entropy Repair
- 22. Coordinated process Four steps: 1:Hash 2: Compare 3: Stream 4: Merge Can go wild... Anti-entropy Repair
- 23.
- 24. Repair gone wild Eatsa lot of disk IO ● because of hashing all the data
- 25. Repair gone wild Eatsa lot of disk IO Saturates the network ● because of streaming a lot of data around
- 26. Repair gone wild Eatsa lot of disk IO Saturates the network Fills up the disk ● because of receiving all replicas, possibly from all other data centers
- 27. Repair gone wild Eatsa lot of disk IO Saturates the network Fills up the disk Causes a ton of compactions ● because of having to merge the received data
- 28. Repair gone wild Eatsa lot of disk IO Saturates the network Fills up the disk Causes a ton of compactions … one better be careful
- 29.
- 30. nodetool repair Careful repair Allthree intervals
- 31. Partitioner range ● nodetoolrepair -pr Careful repair This interval only
- 32. Start & endtokens ● nodetool repair -pr -st -et Careful repair A part of interval only
- 33. Requires splitting thering into smaller intervals Smaller intervals mean less data Less data means fewer repairs gone wild Careful repair
- 34. Smaller intervals alsomean more intervals More intervals mean more actual repairs Repairs need to be babysat :( Careful repair
- 35. The Spotify way Featureteams meant to do features Not waste time operating their C* clusters Cron-ing nodetool repair is no good ● mostly due to no feedback loop This all led to creation of the Reaper
- 36. The Reaper REST(ish) service Doesa lot of JMX Orchestrates repairs for you
- 37. The reaping You: curl http://reaper/cluster--data ‘{“seedHost” : “my.cassandra.host.net”}’ The Reaper: ● Figures out cluster info (e.g. name, partitioner)
- 38. The reaping You: curl http://reaper/repair_run--data ‘{“clusterName”: “myCluster”}’ The Reaper: ● Prepares repair intervals
- 39. The reaping You: curl -XPUT http://reaper/repair_run/42 -d state=RUNNING The Reaper: ● Starts triggering repairs of repair intervals
- 40.
- 41. Reaper’s features Carefulness -doesn’t kill a node ● checks for node load ● backs off after repairing an interval
- 42. Reaper’s features Carefulness -doesn’t kill a node Resilience - retries when things break ● because things break all the time
- 43. Reaper’s features Carefulness -doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes ● multiple small intervals in parallel
- 44. Reaper’s features Carefulness -doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes Scheduling - setup things only once ● regular full-ring repairs
- 45. Reaper’s features Carefulness -doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes Scheduling - setup things only once Persistency - state saved somewhere ● a bit of extra resilience
- 46. What we reaped Firstrepair done 2015-01-28 1,700 repairs since then, recently 90 per week 176,000 (16%) segments failed at least once 60 repair failures
- 47.
- 48.
- 49. Reaper’s Future CASSANDRA-10070 Whatever isneeded until then
- 50. Greatest benefit Cassandra Reaperautomates a very tedious maintenance operation of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise github.com/spotify/cassandra-reaper #CassandraSummit
- 51.