DataStax | Distributing the Enterprise, Safely (Thomas Valley) | Cassandra Summit 2016
•Download as PPTX, PDF•
0 likes•206 views
This document discusses testing distributed databases like Cassandra for fault tolerance. It recommends testing at scale by simulating production workloads and failure scenarios over extended periods. Critical factors to test include node performance, configuration, repair, and mean time to recovery from single node, rack, availability zone and full data center failures both within and beyond the hint window. The goal is to validate that the database can sustain workloads and recover from failures at the expected utilization levels.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to DataStax | Distributing the Enterprise, Safely (Thomas Valley) | Cassandra Summit 2016
Similar to DataStax | Distributing the Enterprise, Safely (Thomas Valley) | Cassandra Summit 2016 (20)
More from DataStax
More from DataStax (20)
Recently uploaded
Recently uploaded (20)
DataStax | Distributing the Enterprise, Safely (Thomas Valley) | Cassandra Summit 2016
- 1. Thom Valley Distributing the Enterprise, Safely
- 2. 1 Distributing the Enterprise 2 The Fault Tolerance Fallacy 3 Testing (On the Spectrum) 4 Critical Factors 5 Example Test Scenarios 2© DataStax, All Rights Reserved.
- 4. Distributing the Enterprise - Assumptions © DataStax, All Rights Reserved. 4 Can Install / Configure Have Appropriate / Adequate Hardware (?) Sound Data Model
- 5. Distributing the Enterprise © DataStax, All Rights Reserved. 5 San Francisco New York Dubai Massively Scalable Operationally Simple Always On Geographically Distributed Fault Tolerant
- 6. The Fault Tolerance Fallacy
- 7. The Fault Tolerance Fallacy © DataStax, All Rights Reserved. 7 The network is reliable Latency is zero Bandwidth is infinite The network is secure Topology doesn't change There is one administrator Transport cost is zero --Sun Microsystem Fellows…
- 8. The Fault Tolerance Fallacy © DataStax, All Rights Reserved. 8 My cluster has N copies of my data, I don’t need to backup… Cassandra / DSE is easy to scale, just add nodes… If a node fails, I will just replace it… If a DC fails, everything will just fail over… X and Y (big company) are using it, it must just work…
- 9. The Fault Tolerance Fallacy © DataStax, All Rights Reserved. 9 IF YOU CAN’T RECOVER, YOU’RE NOT FAULT TOLERANT
- 10. The Fault Tolerance Fallacy © DataStax, All Rights Reserved. 10 Very few organizations test for failure (at scale) Most configurations / circumstances are unique Anything that you haven’t proven is a potential failure point Almost everything works at a small scale The features designed to save you may cause your failure Understanding Mean Time to Recover (MTTR)
- 11. The Fault Tolerance Fallacy © DataStax, All Rights Reserved. 11 Very few organizations test for failure (at scale) Most configurations / circumstances are unique Anything that you haven’t proven is a potential failure point Almost everything works at a small scale The features designed to save you may cause your failure Understanding Mean Time to Recover (MTTR)
- 12. Testing (On the Spectrum)
- 13. Testing (On the Spectrum) © DataStax, All Rights Reserved. 13 TEST? testing acumen
- 14. Testing - Guidance © DataStax, All Rights Reserved. 14 Test at scale (nodes / data size / throughput) Test with a real data model / production compaction strategies Test with your full stack Don’t just test for speeds and feeds Test for extended time periods (days or weeks, not hours) Test failure scenarios Test recovery scenarios (understand MTTR)
- 15. Critical Factors
- 16. Critical Factors © DataStax, All Rights Reserved. 16 Node Performance / Density LAN Performance WAN Performance Bootstrap/Rebuild Window DC Rebuild Window Cluster Capacity Configuration* Hinted Handoff (?) Repair Rack / AZ Outages Rolling Restarts Upgrades
- 17. Critical Factors - Configuration © DataStax, All Rights Reserved. 17 Compaction Strategy VNODES Heap Settings Throttles
- 18. Critical Factors - Understanding © DataStax, All Rights Reserved. 18 Impact of workload on cluster / nodes Impact of failed state on cluster Impact of recovery process on cluster Mean time to recover - MTTR (node, data center)
- 19. Critical Factors - Results © DataStax, All Rights Reserved. 19 Proper node / cluster sizing Configuration validation Operational readiness (understand MTTR): Single Node Full Rack / AZ Full DC
- 20. Test Scenarios
- 21. General Guidelines © DataStax, All Rights Reserved. 21 Whenever possible, be running full workload simulation Run repair in background (where possible) Monitor application / cluster Test scenarios more than once
- 22. Single Node Failure (within hint window) © DataStax, All Rights Reserved. 22 General Process • Get workload running / monitoring • Fail node within hint window (2 hour target) • Bring node back online
- 23. Single Node Failure (within hint window) © DataStax, All Rights Reserved. 23 Things to Watch • Impact to workload during failure, replay, compaction • Load on cluster during same • Can you recover at node density / workload? • MTTR to get to normal state?
- 24. Single Node Failure (within hint window) © DataStax, All Rights Reserved. 24 Other Considerations • Hint compaction (pre 3.0) • Multi-DC with distributed workload / hint replay • Impact of latency on Multi-DC • Impact of throttling (or lack) on hint replay • What’s a sustainable hint window w/ workload?
- 25. Single Node Failure (beyond hint window) © DataStax, All Rights Reserved. 25 General Process • Get workload running / monitoring • Fail node for more than hint window • Bring node back online / rebuild node • (Optionally truncate hints if that’s your ops plan)
- 26. Single Node Failure (beyond hint window) © DataStax, All Rights Reserved. 26 Things to Watch • Impact to workload during failure, replay, compaction • Impact to workload during node rebuild • Load on cluster during same • MTTR for rebuilding node with workload running, including compaction
- 27. Single Node Failure (beyond hint window) © DataStax, All Rights Reserved. 27 Other Considerations • How does node density impact node rebuild? • How does compaction strategy impact node rebuild? • What is the impact of VNODES on streaming / compaction during rebuild?
- 28. Rack / AZ Failure (within hint window) © DataStax, All Rights Reserved. 28 General Process Same as above (single node test) but for full AZ Considerations Can cluster sustain workload? At what utilization? Can cluster sustain hint buildup? What is a sustainable hint window with workload?
- 29. Rack / AZ Failure (beyond hint window) © DataStax, All Rights Reserved. 29 General Process Same as above (single node test) but for full Rack / AZ Considerations Can cluster sustain workload? At what utilization? Can cluster sustain hint buildup? Truncate Hints? Can cluster sustain rebuilding full Rack / AZ
- 30. Full DC Failure (within hint window) © DataStax, All Rights Reserved. 30 General Process Same as above (single node test) but for full DC Considerations Can cluster sustain workload? At what utilization? Can cluster sustain hint buildup? Impact of hint replay on remote DCs? WAN Utilization? What is a sustainable hint window with workload?
- 31. Full DC Failure (beyond hint window) © DataStax, All Rights Reserved. 31 General Process Same as above (single node test) but for full DC Considerations Can cluster sustain workload? At what utilization? Can cluster sustain hint buildup? Truncate Hints? Can cluster sustain rebuilding full DC? Impact on WAN links during full DC rebuild?
- 32. Long Running Workload © DataStax, All Rights Reserved. 32 Considerations Change in response times (read and/or write) Impact of delete workloads Impact of compaction settings Changing impact of repair operations
- 33. Other Scenarios to Consider © DataStax, All Rights Reserved. 33 Flapping node Flapping network (single node) Flapping network (WAN) Over consuming WAN (Multi-DC)
- 34. Strategies © DataStax, All Rights Reserved. 34 Pick scenarios that matter to you Test in advance at as close to scale as possible Repeat tests on major changes: Infrastructure Compaction Strategies Workload (read/write mix, volume, etc.)
- 35. Thank You
Editor's Notes
- So let’s start off with some assumptions we are going to make. You have adequate hardware to meet your workload needs (or at least you think you do.) You are adept at installing and configuring Cassandra or DSE. You are able to and have designed a sound data model. There are lot’s of other presentations about data modeling…
- So why do we build distributed systems and especially use distributed databases like Cassandra? To get these specific capabilities. None of this should be a surprise to anyone. While these are fantastic reasons for using distributed systems, these capabilities don’t come without any cost. Understanding how distributed systems can fail is even more critical, because there are many connected parts that make up the powerful whole. If a component fails and you aren’t prepared for it, or don’t understand what failure looks like, or what recovery takes, you are no better of than with a single image system, in fact you may be in a worse place.
- While Distributed systems are much complex than single image systems
- Go through list… These are assumptions that I see people using Cassandra in the field make all the time The scariest part of this is, all of these are true (well, except for the first one) But they aren’t true in all circumstances, or even in many circumstances. Making blind assumptions about how something as complex as Cassandra is going to work is dangerous. What we are here to talk about today are the factors we need to be aware of in a testing plan and approaches you should be taking to ensure that Cassandra / DSE is going to provide you the benefits it promises.
- Go through list… These are assumptions that I see people using Cassandra in the field make all the time The scariest part of this is, all of these are true (well, except for the first one) But they aren’t true in all circumstances, or even in many circumstances. Making blind assumptions about how something as complex as Cassandra is going to work is dangerous. What we are here to talk about today are the factors we need to be aware of in a testing plan and approaches you should be taking to ensure that Cassandra / DSE is going to provide you the benefits it promises.
- Testing for a high write workload on STCS does not prepare you for a high read workload on LCS.
- Testing for a high write workload on STCS does not prepare you for a high read workload on LCS.
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- This is the test that almost everyone runs (at least once)
- This is the test that almost everyone runs (at least once)
- This is the test that almost everyone runs (at least once)
- This is the test that almost everyone runs (at least once)
- This is the test that almost everyone runs (at least once)
- This is the test that almost everyone runs (at least once)
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems
- While Distributed systems are much complex than single image systems