Scylla Summit 2018: Keeping Your Latency SLAs No Matter What!
•Download as PPTX, PDF•
1 like•1,610 views
As a real time Big Data database, there are few things more important than keeping latencies low and bounded. Scylla has been delivering great tail latencies from our day one, but the job of making them better never ends and there is always more to do. In this talk we will explore some of the changes made to Scylla in the past few releases to help keep latencies down.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Scylla Summit 2018: Keeping Your Latency SLAs No Matter What!
Similar to Scylla Summit 2018: Keeping Your Latency SLAs No Matter What! (20)
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
Scylla Summit 2018: Keeping Your Latency SLAs No Matter What!
- 1. Keeping your latency SLA No matter what! Glauber Costa VP Field Engineering ScyllaDB
- 2. Presenter bio Glauber Costa is VP of Field Engineering at ScyllaDB. He shares his time between the engineering department working on upcoming Scylla features and helping customers succeed. Before ScyllaDB, Glauber worked with Virtualization in the Linux Kernel for 10 years, with contributions into the Xen and KVM Hypervisors and all sorts of guest functionality and containers.
- 3. Where do latencies come from?
- 4. Where do latencies come from?
- 5. Storage I/O ▪ 4kB reads ▪ 150+ concurrent requests ▪ 300𝜇s avg latency
- 6. Storage I/O ▪ 128kB reads ▪ ~30 concurrent requests ▪ 2ms avg latency
- 7. Storage I/O ▪ AWS i3.metal write bandwidth: 6GB/s
- 8. Storage I/O ▪ AWS i3.metal write bandwidth: 6GB/s ▪ AWS i3.metal read bandwidth: 15GB/s
- 9. Storage I/O 128kB write128kB write128kB write 12ms worth of latency Storage
- 10. Storage I/O 512-byte read 128kB write128kB write128kB write 12ms worth of latency Storage at least 12ms!
- 11. Scylla 2.2 vs Scylla 2.3 Scylla 2.2 --max-io-requests=150 Scylla 2.3 read_iops: 9001 read_bandwidth: 160MB write_iops: 9723 write_bandwidth: 162MB
- 12. Scylla 2.2 vs Scylla 2.3 (AWS EBS, reads vs compaction) Scylla 2.2 Scylla 2.3 average 8.4ms 6.0ms 40% p50 5.6ms 3.1ms 80% p95 15ms 8ms 85% p99 67ms 63ms 6.3%
- 13. CPU Scheduling read write read Compaction CPU SSTable write
- 14. CPU Scheduling read write read Compaction CPU CPU Compaction SSTable write SSTable write read write readread write read
- 15. CPU Scheduling ▪ Preemption every x 𝜇s - the task quota ▪ Expected latency increase when n more classes run: • n * task_quota • Scylla 2.3 task quota: 500 𝜇s
- 16. Which tasks to run? 100 shares 100 shares
- 17. Which tasks to run? 100 shares 50 shares
- 18. Which tasks to run?
- 19. Workload changes, Scylla adapts
- 20. Scylla 2.1, no controller
- 21. Scylla 2.2, compaction controller
- 22. Comparative results Scylla 2.1 Scylla 2.2 Writes/s 354K 418K +18%
- 23. CPU Scheduling ▪ Preemption every x 𝜇s - the task quota ▪ Expected latency increase when n more classes run: • n * task_quota • Scylla 2.3 task quota: 500 𝜇s
- 24. CPU Scheduling ▪ Preemption every x 𝜇s - the task quota ▪ Expected latency increase when n more classes run: • n * task_quota • Scylla 2.3 task quota: 500 𝜇s ▪ Failure to preempt: task quota violation
- 25. Task quota violations ▪ Large memory allocations ▪ Memory allocator itself ▪ The Cache ▪ I/O Subsystem ▪ SSTable writing ▪ Bloom Filter generation ▪ The Linux Kernel
- 26. Scylla 2.2 vs Scylla 2.3 (99.9th percentile write latency) Scylla 2.2 - 4ms to 6ms Scylla 2.3 - < 4ms
- 27. It doesn’t end there
- 28. We need to preempt more ▪ Preemption uses a thread that sets a flag • Source of context switches ▪ Every preemption does I/O • It is a good thing, but I/O goes to the kernel ▪ New polling API merged to the Linux kernel • No context switches for I/O! Can poll a lot more often
- 29. We need to preempt more - Linux 4.19 Completion queue shared between kernel and user Disk Write Completion Disk Read Completion Socket is readable Disk Read Completion Socket is readable Database dequeues event notifications from shared ring without a system call Kernel appends event notifications from interrupts SCYLLA KERNEL
- 30. Shard-aware driver - Scylla 2.3 There is no network hop, both CPUs are in the same system. But worst case, it adds a task quota to request processing. CPU that owns connection CPU that owns the token Token
- 31. Shard-aware driver - Scylla 2.3 Driver opens connections to all CPUs, and learns which tokens belong to each CPU. Request is sent directly to owner. CPU that owns the token Token
- 32. Summary ▪ Keeping latencies bounded and predictable is important ▪ Latency-inducing events really come from everywhere - from the Linux kernel itself to the most surprising places in Scylla ▪ We keep investing time year after year to make sure our latencies are even more predictable
- 33. Thank You Any Questions ? Please stay in touch glauber@scylladb.com @glcst