Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[Cassandra summit Tokyo, 2015] Problem free Cassandra upgrades by Aaron Morton

730 views

Published on

Cassandra jp 2015 problem free cassandra upgrades

Published in: Technology
  • Be the first to comment

  • Be the first to like this

[Cassandra summit Tokyo, 2015] Problem free Cassandra upgrades by Aaron Morton

  1. 1. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License CASSANDRA JP 2015 PROBLEM FREE CASSANDRA UPGRADES Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder &Team Member
  2. 2. AboutThe Last Pickle. Work with clients to deliver and improve Apache Cassandra based solutions. Based in New Zealand, USA, Norway.
  3. 3. Why Upgrade? When to Upgrade Run Books Upgrading from 1.2 to 2.0
  4. 4. Why Upgrade? Bug Fixes New Features Performance Improvements
  5. 5. Why Upgrade? Because it is easy.
  6. 6. Why Upgrade? Upgrading is a series of short term node failures.
  7. 7. Why Upgrade? Cassandra is designed to maintain availability when nodes fail.
  8. 8. Why Upgrade? When to Upgrade Run Books Upgrading from 1.2 to 2.0
  9. 9. WhenTo Upgrade Upgrade before you need to upgrade.
  10. 10. WhenTo Upgrade Upgrade during business hours when people are at their best.
  11. 11. WhenTo Upgrade If you are a startup upgrade to whatever release gives you features you need. (Make sure you test.)
  12. 12. WhenTo Upgrade If you are a mature company generally avoid the first 6 revisions of a major release. (Still do testing.)
  13. 13. WhenTo Upgrade Ideally stay 1 or 2 revisions behind on the major version.
  14. 14. Why Upgrade? When to Upgrade Run Books Upgrading from 1.2 to 2.0
  15. 15. Run Books Why are we doing this? What are we doing? How will we do it?
  16. 16. Run Books Phase changes into chunks. Describe the state of the system at the end of the Phase
  17. 17. Phases Tasks that have checks and rollback. cut-and-paste actions for operators.
  18. 18. Run Book Sections Overview News Changes Tickets Configuration Schema
  19. 19. Run Book Sections Current Situation Assumptions Risks Phases
  20. 20. Why Upgrade? When to Upgrade Run Books Upgrading from 1.2 to 2.0
  21. 21. TLP Run Book A detailed Run Book for this upgrade is available at http://thelastpickle.com/blog/
  22. 22. Overview Prepare the cluster. To each node in turn: Shutdown then node. Upgrade the installation. Restart the node. Upgrade on disk files. Cleanup.
  23. 23. cassandra.yaml (There are more changes.) Setting Change reduce_cache_sizes_at Setting has been removed. reduce_cache_capacity_to Setting has been removed. flush_largest_memtables_at Setting has been removed. native_transport_min_threads Setting has been removed.
  24. 24. Schema Change Setting Change read_repair_chance The default has changed from 0.1 to 0.0. local_read_repair_chance The default has changed from 0.0 to 0.1.
  25. 25. News Java 7 Required Cassandra 1.2.9 Required Blob Formats New LCS File Size Speculative Retry Added Configurable Metrics
  26. 26. Current Situation nodetool status nodetool version nodetool gossipinfo && date
  27. 27. Assumptions Disk Space No ERRORs log messages Gossip Stable No Dropped Messages Backups Disabled Repair Disabled
  28. 28. Assumptions On Disk File Format MinimumVersion Monitoring Blob Constants Availability
  29. 29. Risks Latency Availability Incompatibility Disk Space
  30. 30. Phase 0. Check Assumptions Ensure the cluster works the way we think it does.
  31. 31. Phase 1. Prepare the cluster Snapshot Update package repositories Disable Hinted Handoff
  32. 32. Phase 2. Upgrade First Node to 2.0.14 All nodes UP Graceful shutdown Clear CommitLog Upgrade Package
  33. 33. Phase 2. Upgrade First Node to 2.0.14 Update cassandra.yaml Start Node Tail system.log watch nodetool tpstats
  34. 34. Phase 2. Upgrade First Node to 2.0.14 Rollback (if needed) Backup 2.0 SSTables. Delete all data. Restore from snapshot. Repair.
  35. 35. Phase 3. Upgrade Remaining Nodes to 2.0.14 Ensure all nodes are UP Rolling upgrade
  36. 36. Phase 4. Cleanup Enabled Hinted Handoff
  37. 37. Phase 5. Upgrade SSTables nodetool clearsnapshot -t pre-2.0 nodetool upgradesstables
  38. 38. Phase 6. Restore Support Services Enabled Backups Enabled Repair
  39. 39. Thanks.
  40. 40. Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License Aaron Morton @aaronmorton Co-Founder &Team Member www.thelastpickle.com Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

×