This document discusses changes made to Cassandra's storage format and key cache to support large partitions. It describes how IndexedEntry objects were restructured to reduce heap pressure during reads and writes of large partitions. Testing showed the changes allowed reading and writing partitions over 8GB in size while constraining the heap. However, issues with direct memory usage and GC loops were discovered and addressed with off-heap memory pools.

1. Storage format and key cache changes to
support large partitions
ROBERT STUPP, DATASTAX
SOLUTION ARCHITECT, COMMITTER

2. Read Path recap
1. Bloom filter
2. Index Summary
3. Primary Index
4. Data File
RowIndexEntry
◦ points to partition in data file
◦ Only for partitions <64kB
IndexedEntryextends RowIndexEntry
◦ points to partition in data file
◦ Only for partitions >64kB
◦ Contains one IndexInfo object per 64kB

3. IndexedEntry
IndexedEntry extends RowIndexEntry
DeletionTime
ArrayList
IndexInfo ß per 64kB
DeletionTime
BufferClustering
Kind
ByteBuffer[]
ByteBuffer
byte[]
…
BufferClustering
Kind
ByteBuffer[]
ByteBuffer
…
Approximation on
16 byte clustering-value:
•1MB : 3kB / >200 objects
•4MB : 11kB / > 800 objects
•64MB : 180kB / > 13k objects
•512MB : 1.4MB / > 106k objects

4. Issues with IndexedEntry

5. Issues with IndexedEntry
•IndexedEntry object tree built during flush/compaction
•IndexedEntry object tree constructed for every read
•Huge number of objects
à GC, GC, GC, …
Nested object structure – harder for garbage collection
•Evicts “legit” entries from the key cache on reads
àmore disk I/O

6. Initial approach
•IndexInfo never kept on heap
•Read from disk when needed
•Causes non-negligible performance degration w/ trades-workload

7. Current approach
•IndexInfo kept on heap,
if serialized size of IndexedEntry
< column_index_cache_size_in_kb
•Otherwise always read from disk when needed
•Write Path (flush, compaction) similar:
• IndexInfo kept on heap (for key cache)
if < column_index_cache_size_in_kb
• Otherwise serialized to a buffer (not kept as an object)

8. Read patterns
•Binary search
•IndexInfo objects revisistedby the same ”consumer”
•Sequential reads (not using index)

9. What does it buy?
•Less heap pressure during reads
•Less heap pressure during flushes/compactions
•Tested functionality (write, read, full compaction) with 8GB partitions in
a utest (w/ tiny heap)
•Local node load test w/ 280MB partitions
•GCE (5 node n1-standard-8) cluster load test w/ 770MB partitions
• Cluster constrainted by disk I/O + net I/O

10. Large partitions considerations
•Depending on workload
• Increase commitlog_segment_size_in_mb +
commitlog_total_space_in_mb
• Increase concurrent_compactors (default of 2 might be a bottleneck)
and/or compaction_throughput
•Key cache can hold “tons” of partition information
•Repairs still take time
(don’t seem to be negatively influenced by the patch)
•Compactions & flushes cause less heap pressure
•Recommendations on max amount of data per node still applies IMO
•Large partitions does not mean large CQL rows
(native protocol resp buffer)

11. Findings,
Suggesstions,
Improvements…
… FOR DISCUSSION TOMORROW

12. Biggest issue during tests:
Endless CMS-GC loops

13. Biggest issue during tests:
Endless CMS-GC loops
Reading large partitions results in large responses (duh!)
Concurrent large responses lead to direct-memory-OOM
à Causes “endless” CMS-GC loop
à Dead node
(thank you, ByteBuffer!)
•Solution part #1 : separate off-heap memory pool in Netty (4.0.37 + 4.1.1)
•Solution part #2 : separate off-heap memory pool in C*
An overloaded 2.2 node recovers from this. (Less direct memory usage – that
simple?)

14. Issues during local tests :
ME!
“How the hell do I setup monitoring?”
•Graphite + Whisper
à python dependency hell
à too complicated
•Prometheus
à really nice!
à had to write a “native” exporter
(https://github.com/snazy/prometheus-metrics-exporter)
•Grafana
à cool !

15. Gatling
•GatlingCql – initially by Mikhail Stepura
https://github.com/gatling-cql/GatlingCql
Gatling 2.2.1 + C*-driver 3.0.2
Gatling itself emits metrics during
Contributions welcome !

16. Findings / default configs
•Change default of
concurrent_reads/writes/couter_writes/mv_writes
from 32 to # of CPUs
•Change default of native_transport_threads from 128 to 2*
CPUs

17. Issues during local tests -
Resource consumers?
•No ”lightweight” metrics to measure CPU and heap consumption of
thread groups
•Solution option #1 : integrate in our pools (would miss some threads)
•Solution option #2 : use prometheus-metrics-exporter
(https://github.com/snazy/prometheus-metrics-exporter)+ C* patch