This document provides guidance on diagnosing problems in Cassandra production systems. It outlines various preventative monitoring measures that can be used to monitor systems such as OpsCenter, Munin, Nagios, and Graphite. When problems occur, it recommends narrowing down the issue by examining areas like consistency, repair, queries, compaction, and system metrics. Specific tools are presented for analyzing compaction, system utilities, histograms, query tracing, Java garbage collection, and profiling garbage collection issues.

1. Diagnosing Problems in Production
Jon Haddad, Technical Evangelist, Datastax
Blake Eggleston, Software Developer, Datastax
©2013 DataStax Confidential. Do not distribute without consent.
1

2. Preventative Measures
• Opscenter
• Metrics Integration
• Munin
• Monit
• Nagios / Icinga
• Graphite / Statsd (application level)
• Variety of 3rd party monitoring services

3. Narrow Down the Problem
• Weird consistency issues - NTP?
• Problems with Streaming / Repair - version conflicts
• Slow queries
• Compaction
• Histograms
• Tracing
• Nodes flapping / failing
• Dig into system metrics
• JVM GC issues

4. Compaction
• Compaction merges SSTables
• Too much compaction?
• Opscenter provides insight into compaction cluster wide
• nodetool
• compactionhistory
• getcompactionthroughput
• Leveled vs Size Tiered
• Leveled on SSD + Read Heavy
• Size tiered on Spinning rust
• Size tiered is great for write heavy time series workloads

5. System Utilities
• iostat
• disk level statistics
• htop
• process overview
• iftop & netstat
• network utilities
• dstat
• all the above in 1 tool
• strace
• …for the hardcore

6. Histograms
• proxyhistograms
• High level read and write times
• Includes network latency
• cfhistograms <keyspace> <table>
• reports stats for single table on a single
node
• Used to identify tables with
performance problems

7. Query Tracing

8. JVM GC Overview
• What is garbage collection?
• Manual vs automatic memory management
• Generational garbage collection (ParNew & CMS)
• New Generation
• Old Generation

9. New Generation
• New objects are created in the new gen
• Minor GC
• Occurs when new gen fills up
• Stop the world
• Dead objects are removed
• Live objects are promoted into old gen
• Removing objects is fast, promoting objects is slow

10. Old Generation
• Objects are promoted to new gen from old gen
• Major GC
• Old generations fills up some percentage.
• Mostly concurrent
• 2 short stop the world pauses
• Full GC
• Occurs when old gen fills up or objects can’t be promoted
• Stop the world
• Collects all generations
• These are bad!

11. GC Profiling
• Opscenter gc stats
• Look for correlations between gc spikes
and read/write latency
• Cassandra GC Logging
• Can be activated in cassandra-env.sh
• jstat
• prints gc activity

12. GC Profiling
• What to look out for:
• Long, multi-second pauses
• Caused by Full GCs. Old gen is filling up faster than the concurrent GC can keep up with
it. Typically means garbage is being promoted out of the new gen too soon
• Long minor GC
• Many of the objects in the new gen are being promoted to the old gen.
• Most commonly caused by new gen being too big
• Sometimes caused by objects being promoted prematurely

13. ©2013 DataStax Confidential. Do not distribute without consent. 13