Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Cassandra London March 2016 - Lightening talk - introduction to incremental repair
Similar to Cassandra London March 2016 - Lightening talk - introduction to incremental repair (20)
More from aaronmorton
More from aaronmorton (17)
Recently uploaded
Recently uploaded (20)
Cassandra London March 2016 - Lightening talk - introduction to incremental repair
- 1. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License CASSANDRA LONDON FEBRUARY 2016 LIGHTENING TALK: INTRODUCTION TO INCREMENTAL REPAIR Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder &Team Member
- 2. AboutThe Last Pickle. Work with clients to deliver and improve Apache Cassandra based solutions. Based in New Zealand, USA, Australia, France.
- 4. Repair :(
- 5. Why Repair? It is the way to ensure consistent data on disk.
- 6. Why Repair? Hinted Handoff and (non- blocking) Read Repair are optimisations.
- 7. Why Repair? Consistency ensures Tombstones are distributed, preventing deleted data returning.
- 8. A Repair OfTwo Halves Step 1: Calculate a hash tree to describe the data. (aka Merkle Tree)
- 9. A Repair OfTwo Halves Step 2: Stream differences between nodes.
- 10. gc_grace_seconds Repair will ignore Tombstones older than gc_grace_seconds. So run repair every gc_grace_seconds.*
- 12. Why Incremental Repairs? SSTables are immutable, so repair was processing data that had not changed.
- 13. Why Incremental Repairs? Repairing less data, more frequently, reduces the impact on the cluster.
- 14. repairedAt Track repaired timestamp per SSTable. Ignore SSTables that have been repaired.
- 15. repairedAt Set after when all data in the SSTable was repaired.
- 16. repairedAt $ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 0 $ bin/nodetool repair --incremental --parallel $ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 1454500502306
- 17. Anti Compaction Splits partially repaired SSTables into Repaired and Un-repaired SSTables.
- 18. Anti Compaction $ ls -lah keyspace1/standard1 keyspace1-standard1-ka-9-Data.db keyspace1-standard1-ka-10-Data.db $ bin/nodetool repair --incremental --parallel -st 3074457345618258602 -et -9223372036854775808 $ ls -lah keyspace1/standard1 keyspace1-standard1-ka-11-Data.db keyspace1-standard1-ka-12-Data.db keyspace1-standard1-ka-13-Data.db keyspace1-standard1-ka-14-Data.db
- 19. Anti Compaction $ tools/bin/sstablemetadata keyspace1-standard1-ka-11-Data.db SSTable: ... Repaired at: 1454504607623 $ tools/bin/sstablemetadata keyspace1-standard1-ka-12-Data.db SSTable: ... Repaired at: 0 $ tools/bin/sstablemetadata keyspace1-standard1-ka-13-Data.db SSTable: ... Repaired at: 1454504607623 $ tools/bin/sstablemetadata keyspace1-standard1-ka-14-Data.db SSTable: ... Repaired at: 0
- 20. Compaction Two sets of SSTables are maintained: Repaired and Un- repaired.
- 21. Compaction Compaction Strategy runs in each SSTable set.
- 22. Migration Option 1 1. Disable compaction. 2. Run a classic full repair 3. Stop the node 4. stablerepairedset to mark all the SSTables from before step 1
- 23. Migration Option 2 bin/nodetool repair --incremental --parallel
- 25. Thanks.
- 26. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder &Team Member www.thelastpickle.com Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License