There’s a popular misconception about I/O that (modern) SSDs are easy to deal with; they work pretty much like RAM but use a “legacy” submit-complete API. And other than keeping in mind a disk’s possible peak performance and maybe maintaining priorities of different IO streams there’s not much to care about. This is not quite the case – SSDs do show non-linear behavior and understanding the disk’s real abilities is crucial when it comes to squeezing as much performance from it as possible.
Diskplorer is an open-source disk latency/bandwidth exploring toolset. By using Linux fio under the hood it runs a battery of measurements to discover performance characteristics for a specific hardware configuration, giving you an at-a-glance view of how server storage I/O will behave under load.
ScyllaDB CTO Avi Kivity will share an interesting approach to measuring disk behavior under load, give a walkthrough of Diskplorer and explain how it’s used.
With the elaborated model of a disk at hand, it becomes possible to build latency-oriented I/O scheduling that cherry-picks requests from the incoming queue keeping the disk load perfectly Balanced.
ScyllaDB engineer Pavel Emelyanov will also present the scheduling algorithm developed for the Seastar framework and share results achieved using it.
2. Presenter
2
Avi Kivity
● Original maintainer of Linux KVM - Kernel-based
Virtual Machine
● Co-maintainer of Seastar, ScyllaDB
● Co-founder of ScyllaDB
4. 4
+ For distributed, data-intensive apps that require high
performance and low latency
+ 400+ users worldwide
+ Results
+ Comcast: Reduced P99 latencies by 95%
+ FireEye: 1500% improvement in throughput
+ Discord: Reduced C* nodes from ~140 to 6
+ iFood: 9X cost reduction vs. DynamoDB
+ Open Source, Enterprise and Cloud options
+ Fully compatible with Apache Cassandra and Amazon
DynamoDB
About ScyllaDB
1ms <1ms
10ms
1M
10M
ScyllaDB Universe of 400+ Users
5. 400+ Companies Use ScyllaDB
Seamless experiences
across content + devices
Fast computation of flight
pricing
Corporate fleet
management
Real-time analytics
2,000,000 SKU -commerce
management
Real-time location tracking
for friends/family
Video recommendation
management
IoT for industrial
machines
Synchronize browser
properties for millions
Threat intelligence service
using JanusGraph
Real time fraud detection
across 6M
transactions/day
Uber scale, mission critical
chat & messaging app
5
Network security threat
detection
Power ~50M X1 DVRs with
billions of reqs/day
Precision healthcare via
Edison AI
Inventory hub for retail
operations
Property listings and
updates
Unified ML feature store
across the business
Cryptocurrency exchange
app
Geography-based
recommendations
Distributed storage for
distributed ledger tech
Global operations- Avon,
Body Shop + more
Predictable performance
for on sale surges
GPS-based exercise
tracking
8. SSDs are Amazing
+ 6.4 GB/s read
+ 3.3 GB/s write
+ 1M read IOPS
+ 200k write IOPS
+ Often, several disks in a single server!
8
9. SSDs are Amazing, but not Magic
+ 6.4 GB/s read
+ OR 3.3 GB/s write
+ OR 1M read IOPS
+ OR 200k write IOPS
+ Or some kind of mix
+ But what kind of mix?!
9
10. + Online transaction processing (OLTP)
+ There’s a real user at the other end
+ Maintenance workloads
+ Scaling out
+ Compaction
+ Backup
+ Analytics (OLAP)
+ Want to soak up free bandwidth, but not under a tight deadline
+ Multi-tenancy
+ Several OLTP and OLAP workloads on the same disk/data
Why mixed workloads?
10
11. Prevent one workload from dominating
and hurting another workload
The Challenge
11
13. Introducing Diskplorer
+ Tool to test disks at a variety of mixed workloads
+ Open source: https://github.com/scylladb/diskplorer
+ Python, fio, matplotlib
+ Fancy graphs
+ Hours of fun!
13
28. 28
+ runtime – total CPU time consumed
+ virtual runtimes – total time normalized by entity “weight”
+ algorithm selects entity with smallest virtual runtime
+ long inactive entity “forgets” idle virtual runtime
CPU scheduler classes
31. United States
2445 Faber St, Suite #200
Palo Alto, CA USA 94303
Israel
Maskit 4
Herzliya, Israel 4673304
www.scylladb.com
@scylladb
Thank You!
avi@scylladb.com
@avikivity