This document discusses monitoring Cassandra, including an overview of Cassandra, its internal concepts like read/write paths and compactions, and important metrics to monitor. Key metrics to monitor Cassandra's performance include read/write latency, live SSTable count, thread pool pending/completed tasks, and memtable flush count. Operations like compactions and hinted handoff replication should also be monitored. Resource usage metrics like JVM garbage collection time and memory usage are important to monitor as well. Monitoring these metrics helps detect anomalies, optimize performance, and ensure Cassandra's successful operation over the long run.

1. Presented By: Shubhrank Rastogi
Monitoring Cassandra
with an EYE

2. Agenda
01
● A little about Cassandra
● Cassandra Internal Concepts
○ Read Path
○ Write Path
○ Compactions
○ JVM GC
● Quick Q’s on Metrics, Monitoring
● Monitoring Cassandra
● Categories wise - Metrics with Visuals

3. Quick Overview - About Cassandra
● Apache Cassandra is a free and open-source, distributed, wide column store,
NoSQL database management system designed to handle large amounts of
data across many commodity servers, providing high availability with no single
point of failure.
● A Cassandra cluster has no special nodes i.e. the cluster has no masters, no slaves
or elected leaders.
● It provides various features:
○ Fault-tolerant
○ Scalable
○ Distributed
○ Tunable consistency

4. Let’s Take a PATH - WRITE PATH
● A write request for Cassandra follows the following path at node level:

5. WRITE PATH - Cont.
● A write request for Cassandra follows the following path at cluster level:

6. Let’s Take a PATH - READ PATH
● The read request follows the following path at SSTable level:

7. Compactions is necessary!
○ Why?
■to save disk space.
■to clean redundant data.
■to make response faster.
○ How cassandra stores data?
■adds data by timstamping.
■generates mulitple SSTables.
○ Why is it bad?
■Data utilization is high.
■response time is high.
○ Solution:
■remove redundant data with the help of
Tombstones
■generates ONE sstable throughout cluster.

8. JVM GC to the rescue!
● Cassandra process is run on JVM, using off-heap and in-heap
memory for its components.
● One situation that you definitely want to minimize
○ garbage collection pause aka stop-the-world event
● A pause occurs when a region of memory is full and its effects:
○ operations are suspended.
○ node can appear as down to other nodes.
○ the JVM needs to make space to continue.
○ Read/Write operations will wait, increases Read/Write
Latency.

9. Why do we need Metrics of
anything?

10. To measure something or
Probably to Monitor
something.

11. And why do we need to
Monitor something?

12. ●To make that something successful for long
run
●To detect anomalies and see future
●To optimize that something
●To improve that something
●To save some $$$$

13. “ Necessity is the mother of
Invention and to make that
Invention successful, we need
to Monitor it. ”

14. Now Let’s MONITOR Cassandra
● What we can monitor in cassandra?
○ Performance
■Write Path
■Read Path
○ Capacity (Resources)
■Server/Node
■JVM
○ Operations
■Compactions
■Eventual Consistency

15. Performance based - Metrics
● Read and Write Latency in READ and WRITE PATH
○ The recent read latency and write latency counters are important in making sure operations are happening
in a consistent manner.
○ Effected by -
■Replication factor
■Compactions (Improves READ only)
■MemTable Flush

16. Cont:
● LiveSStable count - No. of SStables for a table in all keyspaces, or you can be more generic like to a specific
keyspace and table.
○ (Read Latency, Disk Usage) is directly proportional to Live SSTable count.
○ Compaction follows a strategy and uses a large amount of CPU

17. Cont:
● ThreadPool
○ Each of the thread pools provides statistics on the number of tasks that are active, pending, and
completed.
○ Metrics:
■PendingTasks
■CompletedTasks
○ Trends on these pools for increases in the pending task column indicate when to add additional capacity.

18. ● MemTable flush count
○ Configuring memtable thresholds can improve write performance by creating SSTables.
○ it executes when the commit log space threshold or the memtable cleanup threshold has been exceeded.
○ How you tune memtable thresholds depends on your data and write load. Increase memtable thresholds
under either of these conditions:
■The write load includes a high volume of updates on a smaller set of data.
■A steady stream of continuous writes occurs.
Cont:

19. Operations based: Metrics
● Compactions
○ Metrics
■PendingTasks
■CompletedTasks
○ Monitoring compaction performance is an important aspect of knowing when to add capacity to your
cluster.
○ Compactions affects both paths READ and WRITE, indirectly or directly.

20. ● Eventual Consistency
tells us how consistent is our data replication across nodes.
○ HintedHandoff Metrics
keeps track of data which has been replicated to nodes yet.
■Hints_created: number of hinted key space created.)
○ Every data that is stored in cassandra would be replicated across the cluster based on its replication
factor.
○ When coordinator node couldn’t write to replicas node, it would store the data on it self, then re-attempt to
write to replica node if the replica node available.
Cont:

21. Resources based - Metrics
● JVM based resources
○ Metrics:
■JVM_gc_collection_seconds_count
■JVM_heap_memory_bytes_used
■JVM_memory_pool_bytes_used

22. REFERENCES:
● https://docs.datastax.com/en/ddac/doc/datastax_enterprise/ddacAdminGettingStarted.html
● https://thelastpickle.com/blog/2018/04/11/gc-tuning.html
● https://thelastpickle.com/blog/2017/03/16/compaction-nuance.html
● https://thelastpickle.com/blog/2011/04/28/Forces-of-Write-and-Read.html
● https://www.datadoghq.com/blog/how-to-monitor-cassandra-performance-metrics/
● https://medium.com/prismapp/how-do-we-monitor-cassandra-cluster-3cb4c9a2c162
● https://www.datastax.com/dev/blog/modern-hinted-handoff
● https://www.metricly.com/how-to-monitor-cassandra/

23. Thank You :)
ANY Questions?