Meeting a tight P99 latency goal is hard, it's harder when running multiple workloads with a mix of real time sensitive and analytical workloads. In this presentation, I will cover the Scylla schedulers and controllers and demonstrate how they guarantee a good level of resource isolation.

1. Brought to you by
How to Meet Your P99 Goal
While Overcommitting
Another Workload
Shlomi Livne
VP of R&D at

2. Building Applications
(scaling the DB Part)

7. The solutions are:
● Increase DB Size (and start sharding if you are really not lucky)
● Create read replicas
● Create DB caches
Scaling the DB

8. NIGHTMARE
● Are we DOOMED to create more complex application logic to use db caches ?
● Are we DOOMED to create additional replicas to isolate workloads ?
● Are we DOOMED to overpay and underuse
○ Total resource capacity is broken up into pools (replicas) - applications use their specific poll -
they can’t utilize other resources even if they are not in use.

9. Workloads Management

10. Workloads Management
■ Defining for each workload
● Latency (timeout)
● Throughput - RPS
● Type (overload)
■ Defining workload priority so DB can prioritize workloads relative one to the
other
■ VISIBILITY !!!!
● We want to see that this is adhered
● We want to see when it gets broken and why

11. An Example
Priority Type Latency Timeout Throughput
App 1 1000 Interactive 2 ms 3 ms 100K RPS
Analytics 10 Batch 0.5 seconds 1 seconds 1000K RPS

12. Workload
Priority: Set by “business”
● Relative priority of workload compared to others - affecting resource allocation and scheduling
Type: Interactive vs Batch (OLTP vs OLAP)
● Interactive (ex. user requests on the web) - unbounded concurrency (spam attacks)
● Batch (ex. Analytics , reports) - bounded concurrency
Latency: Desired response time in 99% / 95 % / Average
Timeout: time after which the request should be discarded
Throughput: capped concurrency - after which workload overload must be applied
Priority Type Latency Timeout Throughput
App 1 1000 Online 2 ms 3ms 100K RPS

13. It’s Complex
■ Key ability that is required is isolation (without providing dedicated resource
provisioning)
■ With correct isolation - accounting can be done to provide prioritization
■ Isolation needs to be for all resources
● A bad query can “eat up” memory / disk / cpu / network bandwidth (any or all)

14. “Legacy” Databases

15. Oracle QoS

16. IBM DB2 Workload Manager

17. Scylla’s Approach

18. Schedulers Basics

19. Schedulers Basics
100 shares
100 shares

20. Schedulers Basics
100 shares
50 shares

21. Scylla’s Workload Prioritization
● Fine grain scheduling and accounting for disk and cpu tasks
● Allowing administrator to define service levels
CREATE SERVICE LEVEL app1 with
shares=1000 and
timeout=3ms and
workload_type='interactive';

22. P99 Latency
non-isolated (30ms)
isolated (6.7 ms)

23. Workload Management
“State of the Union”

24. SQL No SQL
Mysql Postgres Oracle IBM DB2 MongoDB DynamoDB Cassandra Datastax Scylla
Enterprise
Priority No No(*) Yes(*) Yes(*) No No No No Yes
Type No No No No No No No No Yes
Latency No No Yes Yes No No No No No
Timeout
(server side)
Yes Yes Yes Yes No No No No Yes
Throughput Partial
(2)
No No Yes No External No No No
Resource
Limitation
Partial Partial Yes Yes No No No No No
Visibility Partial Partial Yes Partial Yes No No No Partial

25. Summary
■ We are not DOOMED

26. Summary
■ We are not DOOMED
■ It is complex: “Throwing more money at it” — can help to a certain degree

27. Summary
■ We are not DOOMED
■ It is complex: “Throwing more money at it” — can help to a certain degree
■ Scylla and others are working toward that (we need your feedback)

28. Brought to you by
Shlomi Livne
shlomi@scylladb.com