Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cassandra London March 2016 - Lightening talk - introduction to incremental repair

523 views

Published on

Five minute talk on incremental repair in Apache Cassandra, at the London Cassandra Meetup February 2016.

Published in: Technology
  • Be the first to comment

Cassandra London March 2016 - Lightening talk - introduction to incremental repair

  1. 1. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License CASSANDRA LONDON FEBRUARY 2016 LIGHTENING TALK: INTRODUCTION TO INCREMENTAL REPAIR Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder &Team Member
  2. 2. AboutThe Last Pickle. Work with clients to deliver and improve Apache Cassandra based solutions. Based in New Zealand, USA, Australia, France.
  3. 3. Repair Incremental Repair
  4. 4. Repair :(
  5. 5. Why Repair? It is the way to ensure consistent data on disk.
  6. 6. Why Repair? Hinted Handoff and (non- blocking) Read Repair are optimisations.
  7. 7. Why Repair? Consistency ensures Tombstones are distributed, preventing deleted data returning.
  8. 8. A Repair OfTwo Halves Step 1: Calculate a hash tree to describe the data. (aka Merkle Tree)
  9. 9. A Repair OfTwo Halves Step 2: Stream differences between nodes.
  10. 10. gc_grace_seconds Repair will ignore Tombstones older than gc_grace_seconds. So run repair every gc_grace_seconds.*
  11. 11. Repair Incremental Repair
  12. 12. Why Incremental Repairs? SSTables are immutable, so repair was processing data that had not changed.
  13. 13. Why Incremental Repairs? Repairing less data, more frequently, reduces the impact on the cluster.
  14. 14. repairedAt Track repaired timestamp per SSTable. Ignore SSTables that have been repaired.
  15. 15. repairedAt Set after when all data in the SSTable was repaired.
  16. 16. repairedAt $ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 0 $ bin/nodetool repair --incremental --parallel $ tools/bin/sstablemetadata keyspace1-standard1-ka-5-Data.db SSTable: ... Repaired at: 1454500502306
  17. 17. Anti Compaction Splits partially repaired SSTables into Repaired and Un-repaired SSTables.
  18. 18. Anti Compaction $ ls -lah keyspace1/standard1 keyspace1-standard1-ka-9-Data.db keyspace1-standard1-ka-10-Data.db $ bin/nodetool repair --incremental --parallel -st 3074457345618258602 -et -9223372036854775808 $ ls -lah keyspace1/standard1 keyspace1-standard1-ka-11-Data.db keyspace1-standard1-ka-12-Data.db keyspace1-standard1-ka-13-Data.db keyspace1-standard1-ka-14-Data.db
  19. 19. Anti Compaction $ tools/bin/sstablemetadata keyspace1-standard1-ka-11-Data.db SSTable: ... Repaired at: 1454504607623 $ tools/bin/sstablemetadata keyspace1-standard1-ka-12-Data.db SSTable: ... Repaired at: 0 $ tools/bin/sstablemetadata keyspace1-standard1-ka-13-Data.db SSTable: ... Repaired at: 1454504607623 $ tools/bin/sstablemetadata keyspace1-standard1-ka-14-Data.db SSTable: ... Repaired at: 0
  20. 20. Compaction Two sets of SSTables are maintained: Repaired and Un- repaired.
  21. 21. Compaction Compaction Strategy runs in each SSTable set.
  22. 22. Migration Option 1 1. Disable compaction. 2. Run a classic full repair 3. Stop the node 4. stablerepairedset to mark all the SSTables from before step 1
  23. 23. Migration Option 2 bin/nodetool repair --incremental --parallel
  24. 24. More Information http://www.datastax.com/dev/blog/more- efficient-repairs
  25. 25. Thanks.
  26. 26. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder &Team Member www.thelastpickle.com Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

×