Operations & Tuning
Data Model First
Operations Can’t Fix a Bad Data Model
Sizing for Latency
CPU
Memory
Disk
Sizing for Latency
CPU
Buffer Cache
Disk
JVM
Sizing for Operations
(C * Nodes * Cores) / Replication Factor = Ops/Second
C is a hardware constant!
• 3000 AWS Non-SSD!
...
Sizing for Latency
• 30-130 microseconds in memory!
• 100 microseconds to 12 milliseconds for SSD!
• 9 milliseconds to 60 ...
Shared Storage
DO NOT USE SHARED
STORAGE!!!!!
Disable Access Time
/dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2
Warm The Buffer Cache
➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null ;
Disable NUMA Zone Reclaim
➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode
Use SSD
25% more expensive!
10x Faster
Tuning SSD: Read Ahead
➜ ~ sudo blockdev --report 	
RO RA SSZ BSZ StartSec Size Device	
rw 256 512 4096 0 6442450944 /dev/...
Tuning SSD: Scheduler
➜ ~ echo 0 > /sys/block/xvdb/queue/rotational	
➜ ~ echo noop > /sys/block/xvdb/queue/scheduler
Compaction
Mechanical Disk = Size Tiered!
SSD = Leveled Compaction
TTL
TTL Compaction!
https://issues.apache.org/jira/browse/CASSANDRA-5228!
!
Default TTL!
https://issues.apache.org/jira/br...
Cassandra Heap Settings
8G MAX_HEAP !
2G NEW_HEAP
Solr/Cassandra Heap Settings
14G MAX_HEAP !
2G NEW_HEAP
Hadoop/Cassandra Heap Settings
10G MAX_HEAP !
2G NEW_HEAP
Kernel
Modern Kernel = 30% performance boost
Monitoring
nodetool cfhistograms
Offset SSTables Write Latency Read Latency Row Size Column Count
1 3579 0 0 0 0
2 0 0 0 0 0
. . .
35...
nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 15 0 0
RequestResponseStage 0 0...
Warning Signs
• Pending compactions > 15!
• Parnew collections > 1 second!
• CMS collections > 5 minutes!
• Disk utilizati...
Tools
Upcoming SlideShare
Loading in …5
×

Cassandra Day SV 2014: Basic Operations with Apache Cassandra

2,564 views

Published on

Matt Stump, Solutions Architect at DataStax talks basic operations with Apache Cassandra.

Published in: Technology

Cassandra Day SV 2014: Basic Operations with Apache Cassandra

  1. 1. Operations & Tuning
  2. 2. Data Model First Operations Can’t Fix a Bad Data Model
  3. 3. Sizing for Latency CPU Memory Disk
  4. 4. Sizing for Latency CPU Buffer Cache Disk JVM
  5. 5. Sizing for Operations (C * Nodes * Cores) / Replication Factor = Ops/Second C is a hardware constant! • 3000 AWS Non-SSD! • 4000 Commodity SAS Hardware! • 12000 Commodity SSD Hardware
  6. 6. Sizing for Latency • 30-130 microseconds in memory! • 100 microseconds to 12 milliseconds for SSD! • 9 milliseconds to 60 milliseconds for mechanical disks
  7. 7. Shared Storage DO NOT USE SHARED STORAGE!!!!!
  8. 8. Disable Access Time /dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2
  9. 9. Warm The Buffer Cache ➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null ;
  10. 10. Disable NUMA Zone Reclaim ➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode
  11. 11. Use SSD 25% more expensive! 10x Faster
  12. 12. Tuning SSD: Read Ahead ➜ ~ sudo blockdev --report RO RA SSZ BSZ StartSec Size Device rw 256 512 4096 0 6442450944 /dev/xvda rw 256 512 4096 1920 6144000000 /dev/xvda1 rw 8 512 4096 0 343568023552 /dev/xvdb rw 256 512 4096 0 343568023552 /dev/xvdc ➜ ~ sudo blockdev --setra 8 /dev/xvdb
  13. 13. Tuning SSD: Scheduler ➜ ~ echo 0 > /sys/block/xvdb/queue/rotational ➜ ~ echo noop > /sys/block/xvdb/queue/scheduler
  14. 14. Compaction Mechanical Disk = Size Tiered! SSD = Leveled Compaction
  15. 15. TTL TTL Compaction! https://issues.apache.org/jira/browse/CASSANDRA-5228! ! Default TTL! https://issues.apache.org/jira/browse/CASSANDRA-3974
  16. 16. Cassandra Heap Settings 8G MAX_HEAP ! 2G NEW_HEAP
  17. 17. Solr/Cassandra Heap Settings 14G MAX_HEAP ! 2G NEW_HEAP
  18. 18. Hadoop/Cassandra Heap Settings 10G MAX_HEAP ! 2G NEW_HEAP
  19. 19. Kernel Modern Kernel = 30% performance boost
  20. 20. Monitoring
  21. 21. nodetool cfhistograms Offset SSTables Write Latency Read Latency Row Size Column Count 1 3579 0 0 0 0 2 0 0 0 0 0 . . . 35 0 0 0 0 0 42 0 0 27 0 0 50 0 0 187 0 0 60 0 10 460 0 0 72 0 200 689 0 0 86 0 663 552 0 0 103 0 796 367 0 0 124 0 297 736 0 0 149 0 265 243 0 0 179 0 460 263 0 0 . . . 25109160 0 0 0 0 0
  22. 22. nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 15 0 0 RequestResponseStage 0 0 0 0 0 MutationStage 0 0 3674 0 0 ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemoryMeter 0 0 1 0 0 MemtablePostFlusher 0 0 267 0 0 FlushWriter 0 0 9 0 1 MiscStage 0 0 0 0 0 commitlog_archiver 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 0 0 0 ! Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0
  23. 23. Warning Signs • Pending compactions > 15! • Parnew collections > 1 second! • CMS collections > 5 minutes! • Disk utilization above 40% or 70%! • CFHistograms write throughput > 150 μs! • CFHistograms read throughput > 50 ms! • Really large rows in CFHistograms! • Anything blocked in TPStats
  24. 24. Tools

×