Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla
•
2 likes•1,181 views
Kenshoo is a leader in digital marketing with very heavy data usage. Learn about their big data challenges, the tools that they use, and their experience evaluating Scylla.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla
Similar to Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla (20)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
Scylla Summit 2016: Why Kenshoo is about to displace Cassandra with Scylla
- 1. Cassandra & Scylla at Kenshoo
- 2. About Me • Wrote “Basic” code when I was a kid • 17 years in the internet industry • Big data fanatic for the last 6 years • Big data team leader At Kenshoo • Our team is Big data DBA • Programming: ETL & Administration tools
- 3. Kenshoo • 10 years, Tel-Aviv based Startup • Industry Leader in Digital Marketing • 500+ employees • Heavy data shop
- 6. Cassandra at Kenshoo • Datastax Community • 5 clusters • 70 nodes • Biggest cluster, ver 1.2.19 ▪ 40 physical nodes ▪ 4TB compressed to 1TB per node ▪ 14 Billion records ▪ 1500 bytes values, IOPS: 5K avg, 30K burst ▪ Processing clicks and conversion for user behavior
- 7. Cassandra Cost • Cassandra is great at writes (At first) • Postponing the “cost” to background processes (no free meals) ▪ Compact ▪ Repair ▪ Cleanup ▪ Add / Remove node
- 8. War & Peace Knowing your Cassandra
- 9. Day to day • Requires a lot of in depth knowledge • Lack of documentation • Tuning per application • Lot’s of custom maintenance scripts • Don’t run repair , only rebuild
- 10. • Compaction didn’t complete during maintenance window • Leveled CS is a ColumnFamily cluster wide configuration • Found a jmx “hack” for a per server config Migration to Leveled Compaction Strategy • It took us a month to manually switch • Needed to reconfigure each server after restart
- 11. GC Hell • Lot’s of GC under load • GC causing 95th percentile performance ▪ One problematic node affects the cluster performance • Full restart to the cluster each week • Change rpc server type to hsha • Tuning is black magic, takes days to see effects
- 12. Maintenance is delicate • Need to wait between adding and removing nodes for cluster to rebalance • Turn off a node thrift service • Running partial cleanup • Tuning of params ▪ compaction_throughput_mb_per_sec ▪ concurrent_compactors
- 15. Lab setup • i2.8xlarge • Cassandra 2.1.15 ▪ compaction_throughput_mb_per_sec = 0 (16) ▪ stream_throughput_outbound_megabits_per_sec = 10000 (200) ▪ inter_dc_stream_throughput_outbound_megabits_per_sec = 10000 (200) • Scylla 1.3 ▪ out of the box
- 16. GC at Scylla
- 17. Repair • 3 x i2.8xlarge • RF 3 • 72GB data per node • 10M rows ▪ 5 columns ▪ 1500 bytes value • Delete all the data from one node
- 18. Repair results
- 19. Cleanup at Scylla • 4 x i2.8xlarge • RF 3 • 270GB data per node • 50M rows ▪ 5 columns ▪ 1500 bytes value • Decommission a node and join it back
- 20. Cleanup results
- 21. Compact at Scylla • 3 x i2.8xlarge • RF 3 • 30 minute stress • 10M rows ▪ 5 columns ▪ 1500 bytes value
- 23. Latency & Throughput • 3 x i2.8xlarge • RF 3 • 72GB data per node • 10M rows ▪ 5 columns ▪ 1.5K value ▪ 3 writes, 2 reads mixed • 4 x cassandra-stress nodes ▪ 30 minutes
- 31. • Scylla has lower cost ▪ Compaction, repair & cleanup are much more efficient ▪ Consistent in latency under much higher load • Moving forward ▪ Integrate it in an inner production monitoring system Conclusion
- 32. Q&A