Cassandra Day SV 2014: Basic Operations with Apache Cassandra

  • 923 views
Uploaded on

Matt Stump, Solutions Architect at DataStax talks basic operations with Apache Cassandra.

Matt Stump, Solutions Architect at DataStax talks basic operations with Apache Cassandra.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
923
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
67
Comments
0
Likes
8

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Operations & Tuning
  • 2. Data Model First Operations Can’t Fix a Bad Data Model
  • 3. Sizing for Latency CPU Memory Disk
  • 4. Sizing for Latency CPU Buffer Cache Disk JVM
  • 5. Sizing for Operations (C * Nodes * Cores) / Replication Factor = Ops/Second C is a hardware constant! • 3000 AWS Non-SSD! • 4000 Commodity SAS Hardware! • 12000 Commodity SSD Hardware
  • 6. Sizing for Latency • 30-130 microseconds in memory! • 100 microseconds to 12 milliseconds for SSD! • 9 milliseconds to 60 milliseconds for mechanical disks
  • 7. Shared Storage DO NOT USE SHARED STORAGE!!!!!
  • 8. Disable Access Time /dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2
  • 9. Warm The Buffer Cache ➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null ;
  • 10. Disable NUMA Zone Reclaim ➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode
  • 11. Use SSD 25% more expensive! 10x Faster
  • 12. Tuning SSD: Read Ahead ➜ ~ sudo blockdev --report RO RA SSZ BSZ StartSec Size Device rw 256 512 4096 0 6442450944 /dev/xvda rw 256 512 4096 1920 6144000000 /dev/xvda1 rw 8 512 4096 0 343568023552 /dev/xvdb rw 256 512 4096 0 343568023552 /dev/xvdc ➜ ~ sudo blockdev --setra 8 /dev/xvdb
  • 13. Tuning SSD: Scheduler ➜ ~ echo 0 > /sys/block/xvdb/queue/rotational ➜ ~ echo noop > /sys/block/xvdb/queue/scheduler
  • 14. Compaction Mechanical Disk = Size Tiered! SSD = Leveled Compaction
  • 15. TTL TTL Compaction! https://issues.apache.org/jira/browse/CASSANDRA-5228! ! Default TTL! https://issues.apache.org/jira/browse/CASSANDRA-3974
  • 16. Cassandra Heap Settings 8G MAX_HEAP ! 2G NEW_HEAP
  • 17. Solr/Cassandra Heap Settings 14G MAX_HEAP ! 2G NEW_HEAP
  • 18. Hadoop/Cassandra Heap Settings 10G MAX_HEAP ! 2G NEW_HEAP
  • 19. Kernel Modern Kernel = 30% performance boost
  • 20. Monitoring
  • 21. nodetool cfhistograms Offset SSTables Write Latency Read Latency Row Size Column Count 1 3579 0 0 0 0 2 0 0 0 0 0 . . . 35 0 0 0 0 0 42 0 0 27 0 0 50 0 0 187 0 0 60 0 10 460 0 0 72 0 200 689 0 0 86 0 663 552 0 0 103 0 796 367 0 0 124 0 297 736 0 0 149 0 265 243 0 0 179 0 460 263 0 0 . . . 25109160 0 0 0 0 0
  • 22. nodetool tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 15 0 0 RequestResponseStage 0 0 0 0 0 MutationStage 0 0 3674 0 0 ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemoryMeter 0 0 1 0 0 MemtablePostFlusher 0 0 267 0 0 FlushWriter 0 0 9 0 1 MiscStage 0 0 0 0 0 commitlog_archiver 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 0 0 0 ! Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0
  • 23. Warning Signs • Pending compactions > 15! • Parnew collections > 1 second! • CMS collections > 5 minutes! • Disk utilization above 40% or 70%! • CFHistograms write throughput > 150 μs! • CFHistograms read throughput > 50 ms! • Really large rows in CFHistograms! • Anything blocked in TPStats
  • 24. Tools