Real world repairs
•
0 likes•464 views
At a millions of operation/sec entropy is inevitable. How does Netflix deal with this data inconsistency? and What it takes Netflix to repair their fleet of C* systems for keeping Entropy to it’s minimum? In this talk, Vinay Chella, Cloud Data Architect @ Netflix will shed light on Netflix's AntiEntropy strategy and problems that arise at scale. He will discuss repair strategies and how Netflix repairs petabytes of data while maintaining a strict SLA for client applications and giving seamless movie experience to its customers . Come learn about different repair strategies, tools and techniques to repair your C* clusters.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Real world repairs
Similar to Real world repairs (20)
More from Vinay Kumar Chella
Recently uploaded
Recently uploaded (20)
Real world repairs
- 1. Real World Repairs @ Netflix Vinay Chella Cassandra MVP, Cloud Data Architect
- 7. Who am I?
- 8. Vinay Chella Cloud Data Architect Cassandra MVP Cloud Database Engineering @ Netflix
- 9. Agenda • What is Repair and Why? • How do we Repair @ Netflix? • Use cases • Pain points • Alternatives to Repair • When to/ not to repair • What is missing in C*
- 10. Why is Repair Important? • Entropy is inevitable • Inconsistent data might impact your business • Data Resurrection
- 11. What is Repair
- 12. Case 1 Case 2 Repair in Netflix world • ~300 TB cluster size • ~1 TB per node • SSDs • 288 Nodes - Multi Region Cluster • Critical data, No TTLs • ~180 TB cluster size • ~3.6 TB per node • HDDs • 48 Nodes – Island cluster • No TTLs, long lived
- 13. How do we Repair @ Netflix? • Primary Range (-pr) • Subrange (-st and -et) • On every node in every dc sequentially • Parallel (-par) & SEQUENTIAL
- 14. How do we Repair @ Netflix? Cont.. • More granular Repair -Selective tables only • On schedule basis -Jenkins • Repair progress tracking
- 15. Impacts of UnRepaired C*/ Use-cases • Incorrect bookmark index • Losing the ratings • Recommendations impact • Broken recently watched
- 16. Pain Points • Over streaming • Stream timeouts • Stuck repairs • Tracking/ resuming repair • Compactions bottlenecks • Disk I/O • CPU usage • Latencies/ SLA • Wide partitions
- 17. How do we live with it?
- 18. Well, we don’t, we fix it.
- 19. Over streaming • Subrange repair -Techniques -Algorithm
- 20. Subrange Repair - Get subranges https://github.com/pauloricardomg/cassandra-list-subra nges ▪ Thrift: describe_splits_ex - Repair those ranges using https://github.com/BrianGallew/cassandra_range_repair
- 21. Regular Repair Subrange Repair Subrange Repair • Size per Node – 950 GB - Time to repair – ~11 Hours ▪ Primary Range ▪ SERIAL • Size per Node – 550 GB - Time to repair – ~24 Hours - Primary Range - SERIAL - Wide partitions • Size per Node – 950 GB - Time to repair - ~2.5 Hours - Primary Range - Threads #: 5 - Split size 64K • Size per Node – 550 GB - Time to repair - ~5.5 Hours - Primary Range - Threads # - 10 - Split size - 32K - Wide partitions
- 22. To overcome stream timeouts • YAML Tunings - streaming_socket_timeout_in_ms - stream_throughput_outbound_megabits_ per_sec - inter_dc_stream_throughput_outbound_ megabits_per_sec
- 23. To overcome resume issue • Track it -Record the progress -Record the status • Resume-ability -Resume from failed or paused range
- 24. To fix stuck repairs • Use Repair Tracking data • Timeouts for long running repair - JMX notifications • Self healable repair (Automate it !!) - Grep for repair logs - nodetool compactionstats - nodetool netstats - JMX: forceTerminateAllRepairSessions()
- 25. To avoid compaction bottlenecks • Why compactions? -Validation phase • Tune compaction settings - compaction_throughput_mb_per_sec
- 26. Disk I/O issues • SSD
- 27. High CPU Usage? • Throttled streaming - nodetool setstreamthroughput • Throttling subrange repair -Keysplit -#threads
- 28. To minimize impacts on latencies • Running Repairs during off-peak hours • Switch to SERIAL repair -Instead of PARALLEL • If Subrange -Reduce #threads -Smaller subrange
- 29. Wide partition issues • Reduce the partition size
- 30. Other ways to Repair
- 31. Other ways to Repair: Tickler • Row Tickler • All Row Tickler
- 32. When to Repair • Multi-Region clusters • Low consistent read and writes • Frequent node outages • Low read-repair chance • Flaky networks
- 33. When not to Repair • Short TTls • Highly consistent read and writes • High read repair chance settings in island clusters
- 34. What is missing in C* • Metrics and insights • Incremental Repair bug fixes • CASSANDRA-10070 - Automatic repair scheduling • Mutation-based Repairs?
- 35. Q & A