• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
C* Summit EU 2013: Cassandra on Flash: Performance & Efficiency Lessons Learned
 

C* Summit EU 2013: Cassandra on Flash: Performance & Efficiency Lessons Learned

on

  • 1,334 views

Speaker: Matt Kennedy, Solution Architect: Big Data at Fusion.io ...

Speaker: Matt Kennedy, Solution Architect: Big Data at Fusion.io
YouTube: http://www.youtube.com/watch?v=xu_4TAQlY2U&list=PLqcm6qE9lgKLoYaakl3YwIWP4hmGsHm5e&index=21
Flash Memory technology, deployed as server-side PCIe or solid state disks (SSDs), is emerging as a critical tool for performance and efficiency in data centers of all scales. This presentation will discuss how the use of Flash impacts Cassandra deployments in terms of configuration, DRAM requirements and performance expectations. Ideas on leveraging C*'s cutting-edge data-center awareness to blend flash and disk storage nodes for cost and workload efficiency will also be shared. Flash media itself will be examined from a physical perspective to understand endurance issues. Data on write amplification under bulk-load and operational workload conditions will be presented to explain the impact to Flash of C*'s Log Structured Merge Tree architecture and the associated compactions. Finally, we will examine strategies to make Cassandra more Flash-aware using both conventional techniques as well as emerging Non-volatile memory (NVM) programming capabilities. Lessons learned from real-world customer deployments will be shared to complete this presentation.

Statistics

Views

Total Views
1,334
Views on SlideShare
1,322
Embed Views
12

Actions

Likes
1
Downloads
10
Comments
0

3 Embeds 12

https://twitter.com 9
http://23.253.69.203 2
http://localhost 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • D’oh! Indeed.
  • Tuning: turn off compaction throttling

C* Summit EU 2013: Cassandra on Flash: Performance & Efficiency Lessons Learned C* Summit EU 2013: Cassandra on Flash: Performance & Efficiency Lessons Learned Presentation Transcript

  • Matt Kennedy (@mattmorefaster) October 17, 2013 #CassandraEU — Copyright © 2013 Fusion-io, Inc. All rights reserved. Cassandra: No Moving Parts Cassandra on Flash Memory
  • What is this talk about? ▸ Efficiency • Definition: noun 1. The state or quality of being efficient. ▸ Efficient • Definition: adjective 1. (especially of a system of machine) achieving maximum productivity with minimum wasted effort or expense 2 #CassandraEU
  • Flash vs Disk Cost Efficiency ▸ Capacity 3TB ▸ IOPS 150 200,000 ▸ Cost per IOP 3 4TB $$$$ ¢¢¢¢ #CassandraEU
  • What is flash? 4 #CassandraEU
  • NAND Flash Memory Source Line Word Line Control Gate Bit Line Float Gate N P N Flash is a persistent memory technology invented by Dr. Fujio Masuoka at Toshiba in 1980. 5 #CassandraEU
  • Consumer Volume Drives Economics 6 #CassandraEU
  • Flash in Servers 7 #CassandraEU
  • Direct Cut Through Architecture FUSION DIRECT APPROACH LEGACY APPROACH Host CPU App OS DRAM DRAM SAS SC Super Capacitors App PCIe RAID Controller OS PCIe Host CPU Data path Controller NAND Goal of every I/O operation to move data to/from DRAM and flash. 8 #CassandraEU
  • 9 #CassandraEU
  • Cassandra I/O - Writes http://www.datastax.com/docs/1.2/dml/about_writes 10 #CassandraEU
  • Cassandra I/O - Reads http://www.datastax.com/docs/1.2/dml/about_reads 11 #CassandraEU
  • DRAM Dictates Cassandra Scaling ▸ Key Design Principle: ▸ Working Set < DRAM 12 #CassandraEU
  • Cost of DRAM Modules 1600 $$$$$$ 1400 1200 DOLLARS 1000 800 600 $$$ 400 200 $ $$ 0 4GB 13 #CassandraEU 8GB 16GB 32GB
  • When do we scale out? ▸ A typical server… CPU Cores: 32 with HT Memory: 128 GB …is your working set > 128GB? 14 #CassandraEU
  • Is there a better way? ▸ With NoSQL Databases, we tend to scale out for DRAM Combined Resources CPU Cores: 192 Memory: 768 GB • Low CPU utilization • High Utility cost 15 #CassandraEU 15
  • Flash Offers A New Architectural Choice CPU Cache DRAM Server-based Flash Disk Drives Milliseconds 10-3 16 #CassandraEU Microseconds 10-6 Nanoseconds 10-9
  • How can we use flash in Cassandra? 17 #CassandraEU
  • Four Deployment Options 1. All Flash 2. Data Placement (CASSANDRA-2749) 3. Use Logical Data Centers 4. Cache Layer 18 #CassandraEU
  • Cassandra with All-Flash Storage Step 1: Mount ioMemory at /var/lib/cassandra Step 2: 19 #CassandraEU
  • Data Placement ▸ https://issues.apache.org/jira/browse/CASSANDRA-2749 • Thanks Marcus! ▸ Takes advantage of filesystem hierarchy ▸ Use mount points to pin Keyspaces or Column Families to flash: • /var/lib/cassandra/data/{Keyspace}/{CF} ▸ Use flash for high performance needs, disk for capacity needs 20 #CassandraEU
  • Data Centers for Storage Control Cassandra cluster DC1 (Interactive requests) HIGH MEDIUM 21 #CassandraEU DC2 (Hadoop MR Jobs) P E R F O R M AN C E C APAC I T Y / N O D E DC3 (High density replicas) LOW HIGH
  • Flash Caching ▸ Use Flash to cache blocks from spinning disk • Larger cheaper caches than DRAM • Helps stabilize performance during compaction ▸ Open-Source & Commercial options: • Flashcache: FB developed write-through/back/around cache ▸ Kernel patch ▸ https://github.com/facebook/flashcache/ • bcache: write-through/back/around cache ▸ Kernel patch ▸ http://bcache.evilpiepirate.org/ • Fusion ioTurbine: write-through, commercially supported 22 #CassandraEU
  • The Numbers 23 #CassandraEU
  • YCSB Testing Setup 150 million 1KB records, RF=3: ~ 120GB SSTables/node YCSB Load Generator x4 x1 Workloads use uniform random key selection instead of Zipfian. 24 #CassandraEU 10GB 16-cores 24GB DRAM
  • 10 750 1490 2230 2970 3710 4450 5190 5930 6670 7410 8150 8890 9630 10370 11110 11850 12590 13330 14070 14810 15550 16290 17030 17770 18510 19250 19990 20730 21470 22210 22950 23690 24430 25170 25910 26650 27390 28131 28871 29611 30351 31091 31831 32571 33311 34051 34791 35531 YCSB MIXED OPS/SEC 50/50 R/W Uniform distribution 10hrs 70000 60000 25 50000 40000 30000 20000 10000 Update Latency Average: 511 µs 95th Pctl:1 ms 99th Pctl: 2 ms #CassandraEU Read Latency Average: 7.0 ms 95th Pctl: 18 ms 99th Pctl: 42 ms 0 mixed ops/sec
  • 95/5 R/W Uniform distribution 80000 70000 MIXED OPS/SEC 60000 50000 # threads 10000 99th pctl 1.4/0.22 ms 2/0 ms 5/0 ms 3.1/0.19 ms 7/0 ms 13/0 ms 300 20000 95th pctl 200 30000 Avg Lat. 75 40000 4.4/2.2 ms 11/0 ms 19/0 ms 75 threads 26 #CassandraEU 200 threads 300 threads 690 670 650 630 610 590 570 550 530 510 490 470 450 430 410 390 370 350 330 310 290 270 250 230 210 190 170 150 130 110 90 70 50 30 10 0
  • Consolidation 27 #CassandraEU
  • http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html 28 #CassandraEU
  • Real-World Cassandra on Fusion • 3-4x consolidation factor • 3-6x reduction in latency • 2.2x ROI 29 #CassandraEU
  • Efficiency: Performance or Consolidation? Cassandra @ ~100,000 ops/sec (mixed workload) x 4 x 4 x 4 x 4 x 4 x 4 x 4 x 4 vs. x 4 x 4 Memory/Disk ioMemory http://www.fusionio.com/white-papers/accelerate-cassandra-without-the-cluster-crawl/ 30 #CassandraEU
  • Thank You @mattmorefaster fusionio.com | S A M E P L A N E T. D I F F E R E N T W O R L D .
  • Cassandra: ioDrive2 vs 10 disk RAID-0 32 #Cassandra13 Novemb
  • 50/50 R/W Uniform distribution 120000 YCSB MIXED OPS/SEC 100000 80000 60000 Read Latency Average: 8.2 ms 95th Pctl: 20 ms 99th Pctl: 62 ms Update Latency Average: 311 µs 95th Pctl:0 ms 99th Pctl: 1 ms 40000 20000 mixed ops/sec 33 #Cassandra13 Novemb 550 530 510 490 470 450 430 410 390 370 350 330 310 290 270 250 230 210 190 170 150 130 110 90 70 50 30 10 0
  • 10 70 130 190 250 310 370 430 490 550 610 670 730 790 850 910 970 1030 1090 1150 1210 1270 1330 1390 1450 1510 1570 1630 1690 1750 1810 1870 1930 1990 2050 2110 2170 2230 2290 2350 2410 2470 2530 2590 2650 2710 2770 2830 YCSB INSERTS YCSB: Bulk Load (CL=ALL) 70000 60000 50000 34 40000 30000 20000 Avg Latency: 0.9 ms 95th Percentile: 1 ms 99th Percentile: 4 ms 10000 0 inserts/sec #CassandraEU