Downloaded 16 times
Uploaded byDataStax Academy
Bad Habits Die Hard
The document discusses common bad habits that can occur when working with Apache Cassandra and provides recommendations to avoid them. Specifically, it addresses issues like sliding back into a relational mindset when the data model is different, improperly benchmarking Cassandra systems, having slow client performance, and neglecting important operations tasks. The presentation provides guidance on how to approach data modeling, querying, benchmarking, driver usage, and operations management in a Cassandra-oriented way.
More Related Content
Similar to Bad Habits Die Hard
![5G Explained! A High Level Overview [Introduction]](https://cdn.slidesharecdn.com/ss_thumbnails/5gexplainedahighleveloverview-260119165306-cc137a3e-thumbnail.jpg?width=640&height=640&fit=bounds)
Bad Habits Die Hard
- 1. Bad Habits DieHard Staying on the Right Cassandra Path
- 2. 1 Sliding backinto a relational mindset 2 Benchmarking, how does this go again? 3 My client is slower than molasses in January 4 Bad operations habits 5 Q & A © 2015 DataStax, All Rights Reserved. 2
- 3. Your © 2015 DataStax,All Rights Reserved. 3 Background is Showing
- 4. © 2015 DataStax,All Rights Reserved. 4
- 5. © 2015 DataStax,All Rights Reserved. 5 Sliding back into the relational mindset
- 6. Sliding Back intothe Relational Mindset CQL is almost too reminiscent of SQL © 2015 DataStax, All Rights Reserved. 6 I’m lazy, let’s just copy the schema
- 7. © 2015 DataStax,All Rights Reserved. 7
- 8. Sliding Back intothe Relational Mindset CQL is almost too reminiscent of SQL © 2015 DataStax, All Rights Reserved. 8 Query-Driven Methodology, say it with me
- 9. Sliding Back intothe Relational Mindset Oh, wait, I forgot there’s no joins in Cassandra! © 2015 DataStax, All Rights Reserved. 9 That’s OK, we’ll just do those client side
- 10. © 2015 DataStax,All Rights Reserved. 10
- 11. Sliding Back intothe Relational Mindset Oh, wait, I forgot there’s no joins in Cassandra! © 2015 DataStax, All Rights Reserved. 11 Remember, denormalization is cheaper than joins
- 12. Sliding Back intothe Relational Mindset © 2015 DataStax, All Rights Reserved. 12 I’ve played with C* and it has indexes Great! We’ll use them then.
- 13. © 2015 DataStax,All Rights Reserved. 13 Nope
- 14. Sliding Back intothe Relational Mindset © 2015 DataStax, All Rights Reserved. 14 Querying one partition is cheaper. I’ve played with C* and it has indexes
- 15. Sweet! DataStax Enterprisehas SparkSQL! © 2015 DataStax, All Rights Reserved. 15 Great…we can use the ODBC driver just like an RDBMS! Sliding Back into the Relational Mindset
- 16. © 2015 DataStax,All Rights Reserved. 16
- 17. Sliding Back intothe Relational Mindset DataStax Enterprise has SparkSQL! © 2015 DataStax, All Rights Reserved. 17 Spark is OLAP not OLTP
- 18. © 2015 DataStax,All Rights Reserved. 18 != Remember:
- 19. © 2015 DataStax,All Rights Reserved. 19
- 20. © 2015 DataStax,All Rights Reserved. 20 Or perhaps a SAN? Benchmarking, how does this go again? Let us test on a VM!
- 21. © 2015 DataStax,All Rights Reserved. 21 No Cigar
- 22. © 2015 DataStax,All Rights Reserved. 22 Benchmarking, how does this go again? Why not select suitable hardware instead? Test on 5 nodes with RF=3For search or analytics use more RAM (128GB+)
- 23. © 2015 DataStax,All Rights Reserved. 23 Let’s try and break it with cassandra-stress! With only one client running, the cluster won’t do more than 1,000 tps? What gives? Benchmarking, how does this go again?
- 24. © 2015 DataStax,All Rights Reserved. 24
- 25. © 2015 DataStax,All Rights Reserved. 25 Let’s try and break it with cassandra-stress! Distributed Database = Distributed Clients Benchmarking, how does this go again?
- 26. © 2015 DataStax,All Rights Reserved. 26 How long should we run this for? Benchmarking, how does this go again?
- 27. © 2015 DataStax,All Rights Reserved. 27
- 28. © 2015 DataStax,All Rights Reserved. 28 How long do we run this for? Aim for days or weeks. Benchmarking, how does this go again?
- 29. © 2015 DataStax,All Rights Reserved. 29 Benchmarking, how does this go again?
- 30. © 2015 DataStax,All Rights Reserved. 30 NO MAKE IT STOP
- 31. © 2015 DataStax,All Rights Reserved. 31 Use real data volumes, not pretend ones. Benchmarking, how does this go again?
- 32. © 2015 DataStax,All Rights Reserved. 32 is slower than Molasses in January
- 33. © 2015 DataStax,All Rights Reserved. 33 Why is my Client slower than molasses in January? 1. Download DataStax Driver 2. Used unlogged BATCH statements 3. Connected Client to Load Balancer
- 34. © 2015 DataStax,All Rights Reserved. 34
- 35. © 2015 DataStax,All Rights Reserved. 35 1. Download DataStax Driver 2. Use Prepared Statements and Async Requests 3. Use the DataStax Driver to Load Balance Why is my Client slower than molasses in January?
- 36. © 2015 DataStax,All Rights Reserved. 36 OPERATIONS
- 37. © 2015 DataStax,All Rights Reserved. 37 Bad Operations Habits 1. Meh Backups 2. Meh Repairs 3. Meh Latency on 1% of our Ops 4. Let’s add a couple nodes to scale
- 38. © 2015 DataStax,All Rights Reserved. 38
- 39. © 2015 DataStax,All Rights Reserved. 39 Use OpsCenter 1. Backup service 2. Repair service 3. Capacity service 4. Proactive Alerts, External Notifications and Best Practices
- 40. © 2015 DataStax,All Rights Reserved. 40 Bad Operations Habits 1. Clock skew, you got it covered on that one node right? 2. Configured your Topology Strategy? 3. Set the right snitch? 4. Did you just use C* as a queue? 5. Checked your versions are the same?
- 41. © 2015 DataStax,All Rights Reserved. 41 NO
- 42. © 2015 DataStax,All Rights Reserved. 42 Good Operations Habits 1. Install & configure NTP on every node 2. NetworkTopology Strategy 3. GossipingPropertyFileSnitch 4. Queue? Stop misusing Cassandra 5. Upgrade strategy: rolling upgrades, node by node
- 43. © 2015 DataStax,All Rights Reserved. 43 Yes Feeling better?
- 44. • Free OnlineTraining to Up Your Game • www.academy.datastax.com • More Advanced Courses as well as Beginner • Introduction to Apache Cassandra • Cassandra Core Concepts • Operations and Performance Tuning • Data Modeling • DataStax Enterprise Search with Apache Solr • DataStax Enterprise Analytics with Apache Spark • Seek Advice • Talk to a friendly Solutions Engineer from DataStax © 2015 DataStax, All Rights Reserved. 44 Where to learn more:
- 45. © 2015 DataStax,All Rights Reserved. 45 Questions?
Editor's Notes
- #4 This would be Marc (left) and Ryan (right) Dani introduces Marc and Ryan
- #7 The white indicates the “bad idea”
- #9 Grey indicates the correct approach Yes, but it’s a distributed system, so the tables contain wide partitions that require application query planning.
- #12 Joins don’t scale in distributed systems, denormalize your data and create a table for each query, using wide partitions.
- #15 Yes, but they are for niche uses and aren’t performant with high cardinality columns. Instead, denormalize, disk is cheap, writes ARE performant.
- #18 Spark is tool for running batch or streaming analytics, it is meant for analytics processing not tight customer-facing SLAs *Predictability
- #19 Summarize the point of the section: TL;DR don’t treat Cassandra like a relational db in your application design and query planning.
- #23 How are you going to deploy your app? Cloud or hardware? Test with this in mind, don’t just use machines because you have them. Think how you can best reproduce your real world conditions. Choose options that best reflect the systems you will actually utilize and the ones that are compatible with C*. Choose commodity hardware with killer SSDs, in the cloud interrogate your hardware and ensure you’re running on machines that have decent CPU and SSDs as well. Up your memory game for search and analytics. Cassandra wasn’t built to run on one node, use at least 5 with replication.
- #26 Try 10 clients instead. Stress is a good start, but it’s for basic exploration. You should load test with real data and real clients!
- #29 1. Days, for a minimum. Performance can only be reviewed over time. 2. What happens when compactions and repairs kick in? 3.What happens when you add/remove nodes?
- #32 Don’t use meagre datasets. Exceed RAM. Put 1TB+ on each node. Beware apples to oranges comparisons. Are your settings the same?
- #36 2. Batches in Cassandra are rarely a performance optimization. Unlogged batches create lots of work for your coordinator node 3. Load balancers are a bottleneck and a single point of failure. DataStax drivers load balance for you! They also handle retries and failover.
- #38 Backing up Cassandra sounds hard Do we really have run snapshots on every node then copy the snapshots to S3? We have to run repair on every node, every 10 days? Really? Let’s start off doing it monthly and see how things go… What’s causing the high latency on those 1%? Doesn’t matter. It’s only 1%. We need to plan for future growth Let’s just purchase 5 extra nodes and hope that’s enough
- #40 Repair service – automatically keeps data consistent across a cluster. Backup service – smart and simple backup and restore management for all managed clusters. Capacity service – enables historical trend analysis and forecasts future resource needs Proactive Alerts & External Notifications Best Practices : Slow query logs