There’s a popular misconception about I/O that (modern) SSDs are easy to deal with; they work pretty much like RAM but use a “legacy” submit-complete API. And other than keeping in mind a disk’s possible peak performance and maybe maintaining priorities of different IO streams there’s not much to care about. This is not quite the case – SSDs do show non-linear behavior and understanding the disk’s real abilities is crucial when it comes to squeezing as much performance from it as possible. Diskplorer is an open-source disk latency/bandwidth exploring toolset. By using Linux fio under the hood it runs a battery of measurements to discover performance characteristics for a specific hardware configuration, giving you an at-a-glance view of how server storage I/O will behave under load. ScyllaDB CTO Avi Kivity will share an interesting approach to measuring disk behavior under load, give a walkthrough of Diskplorer and explain how it’s used. With the elaborated model of a disk at hand, it becomes possible to build latency-oriented I/O scheduling that cherry-picks requests from the incoming queue keeping the disk load perfectly Balanced. ScyllaDB engineer Pavel Emelyanov will also present the scheduling algorithm developed for the Seastar framework and share results achieved using it.

1. Understanding
Storage I/O Under
Load
Avi Kivity, CTO
Pavel Emelyanov, Principal Engineer

2. Presenter
2
Avi Kivity
● Original maintainer of Linux KVM - Kernel-based
Virtual Machine
● Co-maintainer of Seastar, ScyllaDB
● Co-founder of ScyllaDB

3. Presenter
3
Pavel Emelyanov
● Ex Linux kernel hacker
● Founder of the CRIU project
● Co-maintainer of Seastar & ScyllaDB

4. 4
+ For distributed, data-intensive apps that require high
performance and low latency
+ 400+ users worldwide
+ Results
+ Comcast: Reduced P99 latencies by 95%
+ FireEye: 1500% improvement in throughput
+ Discord: Reduced C* nodes from ~140 to 6
+ iFood: 9X cost reduction vs. DynamoDB
+ Open Source, Enterprise and Cloud options
+ Fully compatible with Apache Cassandra and Amazon
DynamoDB
About ScyllaDB
1ms <1ms
10ms
1M
10M
ScyllaDB Universe of 400+ Users

5. 400+ Companies Use ScyllaDB
Seamless experiences
across content + devices
Fast computation of flight
pricing
Corporate fleet
management
Real-time analytics
2,000,000 SKU -commerce
management
Real-time location tracking
for friends/family
Video recommendation
management
IoT for industrial
machines
Synchronize browser
properties for millions
Threat intelligence service
using JanusGraph
Real time fraud detection
across 6M
transactions/day
Uber scale, mission critical
chat & messaging app
5
Network security threat
detection
Power ~50M X1 DVRs with
billions of reqs/day
Precision healthcare via
Edison AI
Inventory hub for retail
operations
Property listings and
updates
Unified ML feature store
across the business
Cryptocurrency exchange
app
Geography-based
recommendations
Distributed storage for
distributed ledger tech
Global operations- Avon,
Body Shop + more
Predictable performance
for on sale surges
GPS-based exercise
tracking

6. Part I
Mixed Workloads
6

7. Current SSDs are Amazing
7

8. SSDs are Amazing
+ 6.4 GB/s read
+ 3.3 GB/s write
+ 1M read IOPS
+ 200k write IOPS
+ Often, several disks in a single server!
8

9. SSDs are Amazing, but not Magic
+ 6.4 GB/s read
+ OR 3.3 GB/s write
+ OR 1M read IOPS
+ OR 200k write IOPS
+ Or some kind of mix
+ But what kind of mix?!
9

10. + Online transaction processing (OLTP)
+ There’s a real user at the other end
+ Maintenance workloads
+ Scaling out
+ Compaction
+ Backup
+ Analytics (OLAP)
+ Want to soak up free bandwidth, but not under a tight deadline
+ Multi-tenancy
+ Several OLTP and OLAP workloads on the same disk/data
Why mixed workloads?
10

11. Prevent one workload from dominating
and hurting another workload
The Challenge
11

12. Part II
Understanding Disks
12

13. Introducing Diskplorer
+ Tool to test disks at a variety of mixed workloads
+ Open source: https://github.com/scylladb/diskplorer
+ Python, fio, matplotlib
+ Fancy graphs
+ Hours of fun!
13

14. AWS i3en.3xlarge
14

15. AWS i3.3xlarge
15

16. AWS im4gn.4xlarge
16

17. GCP n2-standard-16, 8 load SSDs
17

18. GCP n2-standard-8, 2TB Persistent Disk
18

19. Clunky hard disk
19

20. Part III
Working with Disks
20

21. 21
+ Equation for the “safety area” can be complex
+ Bandwidth and IOPS are not instantly measurable
Disk math troubles

22. Disk math (step 1)
22

23. 23
+ Equation for the “safety area” can be complex
+ Bandwidth and IOPS are not instantly measurable
Disk math troubles

24. Token bucket algo (and its math)
24

25. Disk math (step 2)
25

26. Scheduler algo
26

27. Scheduler safety area
27

28. 28
+ runtime – total CPU time consumed
+ virtual runtimes – total time normalized by entity “weight”
+ algorithm selects entity with smallest virtual runtime
+ long inactive entity “forgets” idle virtual runtime
CPU scheduler classes

29. Classes and shares
29

30. Scheduler real work
30

31. United States
2445 Faber St, Suite #200
Palo Alto, CA USA 94303
Israel
Maskit 4
Herzliya, Israel 4673304
www.scylladb.com
@scylladb
Thank You!
avi@scylladb.com
@avikivity