Operations and tuning for Cassandra involve: 1) Ensuring a good data model first before trying to optimize operations, as a bad model cannot be fixed by operations alone. 2) Sizing for latency and operations depends on factors like CPU, memory, disk type and replication factor, with SSDs offering much faster performance than mechanical disks. 3) Various tuning techniques are described like disabling access time, warming the buffer cache, using SSDs, adjusting read ahead and schedulers on SSDs, choosing appropriate compaction strategies, and Cassandra heap size settings.

1. Operations & Tuning

2. Data Model First
Operations Can’t Fix a Bad Data Model

3. Sizing for Latency
CPU
Memory
Disk

4. Sizing for Latency
CPU
Buffer Cache
Disk
JVM

5. Sizing for Operations
(C * Nodes * Cores) / Replication Factor = Ops/Second
C is a hardware constant!
• 3000 AWS Non-SSD!
• 4000 Commodity SAS Hardware!
• 12000 Commodity SSD Hardware

6. Sizing for Latency
• 30-130 microseconds in memory!
• 100 microseconds to 12 milliseconds for SSD!
• 9 milliseconds to 60 milliseconds for mechanical disks

7. Shared Storage
DO NOT USE SHARED
STORAGE!!!!!

8. Disable Access Time
/dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2

9. Warm The Buffer Cache
➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null ;

10. Disable NUMA Zone Reclaim
➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode

11. Use SSD
25% more expensive!
10x Faster

12. Tuning SSD: Read Ahead
➜ ~ sudo blockdev --report
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 6442450944 /dev/xvda
rw 256 512 4096 1920 6144000000 /dev/xvda1
rw 8 512 4096 0 343568023552 /dev/xvdb
rw 256 512 4096 0 343568023552 /dev/xvdc
➜ ~ sudo blockdev --setra 8 /dev/xvdb

13. Tuning SSD: Scheduler
➜ ~ echo 0 > /sys/block/xvdb/queue/rotational
➜ ~ echo noop > /sys/block/xvdb/queue/scheduler

14. Compaction
Mechanical Disk = Size Tiered!
SSD = Leveled Compaction

15. TTL
TTL Compaction!
https://issues.apache.org/jira/browse/CASSANDRA-5228!
!
Default TTL!
https://issues.apache.org/jira/browse/CASSANDRA-3974

16. Cassandra Heap Settings
8G MAX_HEAP !
2G NEW_HEAP

17. Solr/Cassandra Heap Settings
14G MAX_HEAP !
2G NEW_HEAP

18. Hadoop/Cassandra Heap Settings
10G MAX_HEAP !
2G NEW_HEAP

19. Kernel
Modern Kernel = 30% performance boost

20. Monitoring

21. nodetool cfhistograms
Offset SSTables Write Latency Read Latency Row Size Column Count
1 3579 0 0 0 0
2 0 0 0 0 0
. . .
35 0 0 0 0 0
42 0 0 27 0 0
50 0 0 187 0 0
60 0 10 460 0 0
72 0 200 689 0 0
86 0 663 552 0 0
103 0 796 367 0 0
124 0 297 736 0 0
149 0 265 243 0 0
179 0 460 263 0 0
. . .
25109160 0 0 0 0 0

22. nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 15 0 0
RequestResponseStage 0 0 0 0 0
MutationStage 0 0 3674 0 0
ReadRepairStage 0 0 0 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemoryMeter 0 0 1 0 0
MemtablePostFlusher 0 0 267 0 0
FlushWriter 0 0 9 0 1
MiscStage 0 0 0 0 0
commitlog_archiver 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
HintedHandoff 0 0 0 0 0
!
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0

23. Warning Signs
• Pending compactions > 15!
• Parnew collections > 1 second!
• CMS collections > 5 minutes!
• Disk utilization above 40% or 70%!
• CFHistograms write throughput > 150 μs!
• CFHistograms read throughput > 50 ms!
• Really large rows in CFHistograms!
• Anything blocked in TPStats

24. Tools