Reworking the Neon IO stack: Rust+tokio+io_uring+O_DIRECT by Christ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

A ScyllaDB Community
Reworking the Neon IO stack:
Rust+tokio+io_uring+O_DIRECT
Christian Schwarz
Member of Technical Staff

Christian Schwarz
Member of Technical Staff
■ Interest in foundational systems software:
databases, ﬁlesystems, OS, hyperv...

Neon Architecture

Separation Of
Storage &
Compute
Compute = vanilla*
Postgres inside
resizable virtual machines
*
: where vanilla PG accesse...

Pageserver
Like a ﬁlesystem:
serve 8 KiB page images to PG
Unlike a ﬁlesystem:
materialize the page images
from the WAL!
M...

Pageserver
LSN
Page
#2 a
init
page #2
insert
tuple into
page #2
#3 b
init
page #3
insert
tuple into
page #3
c
insert
tuple...

Pageserver
Custom-built temporal key-value store.
Organizes WAL records by the 8k page
number they affect.
Key = (Page,LSN...

Pageserver
Layer map
Layer ﬁle
Layer ﬁle: (~SSTables) groups
kv pairs; separation of keys
and values; packed btree
index t...

GetPage@LSN
= fold(walredo, [image, rec, ..., rec])
Collect everything on a “page
vertical” from request LSN until we
ﬁnd ...

DRAM vs IOPs
Sequential traversal is unacceptable if some % reads
need to do IO: latency ~ O(#records)
Where can we spend ...

Caching Bigger Picture
OLTP: index lookup +
point query
OLAP-like /
sequential-scans
User Workloads
(simpliﬁed!)
Pageserve...

How We Got There

Early Days
March
2021
Aug
2021
March
2022
July
2022
ﬁrst
commit
ﬁrst version of
our custom
storage format
second format
re...

Enter Async Rust
Pageserver stored all layers locally & was
running out of NVMe capacity.
Solution: evict cold layers, re-...

Executor Stalls
Ramp up tenant density, enabled by the
work we just did.
Symptom: clients see tput/latency knee,
but serve...

The Plan

The Plan
Step 1
Stick with buffered IO (too
much implicit dependence).
But: use io_uring to do the
buffered IO async ⇒ we ...

How we use io_uring to avoid executor stalls
Constraints: no forking, no switch other
runtime, no sidecar thread/runtime. ...

How we use io_uring to avoid executor stalls
Use a piece of shared state as sqe user_data to model in-ﬂight op.
submission...

How we use io_uring to avoid executor stalls
completion & wake-up of blocking submissions
💡 The fd that represents the io_...

Performance
Working Set Throughput Bottleneck
Fits in Kernel Page Cache 5x (7x if no fairness yield) CPU (⇒ more eﬃcient)
...

Step 2
Get caching and IO under full
userspace control using
O_DIRECT.
The Plan
Step 1
Stick with buffered IO (too
much im...

Why not use Buffered IO / Kernel Page Cache?
Double caching overheads between userspace cache &
kernel (DRAM, memcpy)
No p...

O_DIRECT
So, we made Pageserver ready for O_DIRECT for all reads and writes.
Execution: deliver incrementally over ~1 year...

Conclusion
“Cache in DRAM” vs “Spend IOPS” now under full userspace control.
tokio-epoll-uring drives all our IOs, using i...

Thank you! Let’s connect.
Christian Schwarz
christian.schwarz@databricks.com
@problame
https://cschwarz.com

Reworking the Neon IO stack: Rust+tokio+io_uring+O_DIRECT by Christian Schwarz

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1045526 views
Storytelling For The Web: Integrate... by Chiara Aliotta 958884 views
Artificial Intelligence, Data and C... by OECD Directorate ... 886438 views
How to Leverage AI to Boost Employe... by SocialHRCamp 373696 views
2024 State of Marketing Report – by... by Marius Sescu 212106 views
Everything You Need To Know About C... by Expeed Software 231238 views

View on Slideshare

1 of 27 Ad

View on Slideshare

1 of 27 Ad