ScyllaDB uses a shared-nothing architecture where data is split into partitions across nodes and shards. The "hot partition problem" can occur when a partition becomes overloaded, impacting other nearby partitions. To address this, ScyllaDB implements per-partition rate limiting which counts operations and rejects some to keep the rate under a defined limit. Exceptions were initially making rejections expensive, but this was addressed by avoiding exceptions or implementing missing exception inspection capabilities. In benchmarks, rate limiting restored goodput and provided more stable performance under timeouts.

1. Brought to you by
Retaining Goodput with
Query Rate Limiting
Piotr Dulikowski
Senior Software Engineer at ScyllaDB

2. Motivation

3. Data distribution in ScyllaDB - in a nutshell
■ A ScyllaDB cluster consists of multiple nodes
● Each node is divided into shards (CPU core + part of RAM)
● Shards within a node handle separate data (shared-nothing architecture)
■ Data is split into partitions
● Consists of rows with the same partition key
● Each partition has a subset of nodes called replicas, responsible for storing the partition
● Requests can be handled from any node/shard, but the coordinator has to contact replicas

4. The “hot partition” problem
Each partition has limited computing resources assigned to it, and it’s easy to
exhaust them if the workload becomes too unbalanced.
Partitions whose replicas intersect with hot partition’s replicas will be affected,
too.

5. Choose appropriate schema
■ Keep in mind how your expected workload looks like
● Hot partitions may appear due to badly chosen schema
● ScyllaDB won’t fix those issues for you - schema is your responsibility

6. It’s not always about bad schema
It makes sense to optimize your schema for the common case. What about the
“uncommon case”?
You can always encounter:
■ Malicious/misbehaving users
■ Parts of your system going awry due to bugs
The system does not have to satisfy these requests, but they should not affect the
whole system too much.

7. How to retain goodput?
■ Requests will start piling up on overloaded shards
● When latency exceeds the request timeout, most of the work will be wasted
■ We can reject some requests early
● Accept only as much as we can comfortably handle
● Rejecting some requests early leaves more resources for handling the remaining
ones

8. Per-partition rate limiting
A maximum read/write rate can be
set for a table.
ScyllaDB will reject some operations
in an effort to keep the rate of
successful requests under the limit.
ALTER TABLE ks.tbl
WITH per_partition_rate_limit = {
'max_writes_per_second': 100,
'max_reads_per_second': 200
};

9. How it works

10. Single node case
■ Count partition (token) hits
● On operation, increase the partition’s counter by 1
● Every second, divide all counters by 2 (rounding towards 0)
● Assuming constant rate of X ops/s, counter’s value eventually oscillates between X and 2X
■ The counters fit into a statically-allocated hashmap
● We only keep non-zero counters
● Exponential decay keeps their number low
■ Reject some requests to hot partitions
● Based on the counter value, we can estimate request rate
● Reject with probability such that the accepted requests’ rate is at the limit

11. General case
■ Coordinator cannot reject in general case
● Every shard in the cluster can coordinate
● Can’t reliably estimate request rate only based on the local coordinator’s counters
■ Replicas count the operations and decide whether to reject
● Coordinator chooses a random value and sends it to replicas
● Replicas calculate cutoff threshold based on the counter and reject if random value > cutoff
● Replicas will usually agree on the decision as their counter values should be similar

12. Special case - coordinator is a replica
If the coordinator is a replica, it can decide whether reject or not by itself.
■ No communication is done with other replicas in case of reject
● Rejection is even cheaper
● The operation is either done in full, or fully rejected - no risk of wasted work
● Slightly undercounts operations and a bit more can be accepted than the limit
■ This case happens most of the time with shard-aware drivers
● We recommend you to use them anyway!

13. Rejecting is more costly than accepting?
During development, at some point it appeared that rejected operations were even
more expensive than the accepted ones.
It turned out that it was an issue with a language feature: C++ exceptions.

14. Problems with exceptions

15. What’s wrong with C++ exceptions?
■ People have mixed feelings about exceptions
● They are a part of the language, and they are used in the standard library
● …but they have some undesirable properties, e.g. hard-to-predict performance
■ We are using exceptions in Scylla
● Leads to more idiomatic code, and our framework supports them well
● They aren’t a big problem, as long as you aren’t throwing them in large volumes
■ Throwing exceptions can be slow
● It involves acquiring a global mutex which is not scalable
● We worked around it, but had to disable caching which made throwing scalable, but slow
● https://github.com/scylladb/seastar/blob/master/src/core/exception_hacks.cc

16. Exceptions in Seastar
Seastar gives us flow control
constructs that do not use throwing
underneath.
Exceptions can be stored in
std::exception_ptr and passed
around without throwing.
Problem is, the exception inside the
std::exception_ptr must be rethrown
in order to access it.
future<> do_thing() {
return really_do_thing().finally([] {
std::cout << "Did the thingn"
});
}
future<> really_do_thing() {
if (fail_flag) {
return make_exception_future<>(
std::runtime_error("oh no!"));
} else {
return make_ready_future<>();
}
}

17. Approach 1: avoid them
Use boost::result to return the result
(contains success or exception).
Use a custom container that allows
inspecting the exception.
Results in portable code, but very
tedious to convert existing code.
future<result<>> do_thing() {
return really_do_thing().then(
[] (result<> res) -> result<> {
if (res) {
// handle success
} else {
// handle failure
}
}
);
}

18. Approach 2: implement missing parts ourselves
Introduce an “exception_ptr
inspector” function and replace
existing try..catch blocks in a
straightforward way.
Make sure that for other things we
use the existing tools.
Non-portable code, but much less
work!
std::exception_ptr ep = get_exception();
if (auto* ex
= try_catch<std::logic_error>(ep)) {
// ...
} else if (auto* ex
= try_catch<std::runtime_error>(ep)) {
// ...
} else {
// ...
}
Based on the C++ proposal:
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1066r1.html

19. Results

20. Benchmark - goodput restored after enabling rate limit

21. Benchmark - more stable goodput under timeouts

22. Brought to you by
Piotr Dulikowski
piodul@scylladb.com