Retaining Goodput with Query Rate Limiting

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Retaining Goodput
with Query Rate
Limiting
Piotr Dulikowski, Senior Software Engineer

Piotr Dulikowski
■ Holds BA and MSc in Computer Science from the
University of Warsaw
■ Involved in development of several...

Motivation

■ A ScyllaDB cluster consists of multiple nodes
■ Each node is divided into shards (CPU core + part of RAM)
■ Shards withi...

Each partition has limited computing resources assigned to it, and it’s easy to
exhaust them if the workload becomes too u...

■ Keep in mind how your expected workload looks like
■ Hot partitions may appear due to badly chosen schema
■ ScyllaDB won...

It makes sense to optimize your schema for the common case. What about the
“uncommon case”?
You can always encounter:
■ Ma...

■ Requests will start piling up on overloaded shards
■ When latency exceeds the request timeout, most of the work will be
...

A maximum read/write rate can be
set for a table.
ScyllaDB will reject some operations
in an effort to keep the rate of
su...

How it Works

■ A shard tracks “hit count” for tuples of
(token, table name, operation type)
■ Every second all counters are halved
■ As...

Coordinator increments the counter relevant
to the operation and chooses to reject with
some probability.
■ If the operati...

Coordinator lets replicas decide whether to
accept or reject.
■ Coordinator chooses a random value
and sends it to replica...

On write, all live replicas participate. All
replica counters are updated and all replicas
have a good estimate of the req...

Problems with
Exceptions

■ People have mixed feelings about exceptions
■ They are a part of the language, and they are used in the standard library...

Seastar gives us ﬂow control
constructs that do not use throwing
underneath.
Exceptions can be stored in
std::exception_pt...

Use boost::result to return the result
(contains success or exception).
Use a custom container that allows
inspecting the ...

Introduce an “exception_ptr
inspector” function and replace
existing try..catch blocks in a
straightforward way.
Make sure...

Results

Goodput Restored after Enabling Rate Limit
Highlighted: all shards loaded uniformly

Goodput Restored after Enabling Rate Limit
Highlighted: one shard overloaded, no per-partition rate limiting

Goodput Restored after Enabling Rate Limit
Highlighted: one shard overloaded, with per-partition rate limiting

More Stable Goodput under Timeouts
Yellow - before improvements, Green - after improvements
Before
Before
After
After

piodul
Thank You
Stay in Touch
Piotr Dulikowski
piodul@scylladb.com

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1054718 views
Storytelling For The Web: Integrate... by Chiara Aliotta 970719 views
Artificial Intelligence, Data and C... by OECD Directorate ... 905165 views
How to Leverage AI to Boost Employe... by SocialHRCamp 382887 views
2024 State of Marketing Report – by... by Marius Sescu 216911 views
Everything You Need To Know About C... by Expeed Software 234552 views

View on Slideshare

1 of 25 Ad

View on Slideshare

1 of 25 Ad