Pavel Emelyanov, Principal Engineer at ScyllaDB Botond Denes, C++ Developer at ScyllaDB What performance-minded engineers need to know. Hear from Pavel Emelyanov and Botond Dénes on the impact of database internals – specifically, what to look for if you need latency and/or throughput improvements.

1. Database Internals
Pavel Emelyanov & Botond Dénes

2. Hardware (and kernel) matters
■ CPU
■ Memory
■ Disk I/O
■ Networking
2

3. CPU
3

4. Share nothing
4
■ Cross-cores locking costs
○ Cache lines
○ CPU ticks
■ Amdahl’s law

5. Processes/Threads → Fibers
5
■ Context switch time matters
■ Enforced preemption switch also matters
■ Locking synchronization is a must

6. Quirks
6
■ Execution stages
■ Branch (mis)prediction

7. Memory
7

8. Allocation
8
■ Fragmentation
■ Slab-like allocation
■ Log-structured allocation

9. Cache control
9
■ Let kernel or application do it?
■ Caching can exist at different levels
○ Raw IO blocks
○ Decoded objects

10. Disk I/O
10

11. Types of I/O
11
■ Buffered reads/writes
■ Memory mapped IO
■ Direct IO
■ Asynchronous direct IO
○ IOUring

12. FS vs Disk
12
■ Filesystem adds a level of manageability
■ Extra IO operations
○ Blocks allocation
○ Inodes metadata
○ Journal

13. How moderns SSDs work
13
■ Random reads – YES
■ Random writes – NO
○ Blocks vs Pages
○ Internal parallelism
○ FTL and background operations
■ Sustained vs Burst throughput
■ Mixed workload handling

14. Networking
14

15. Tribute to ANK
15
■ Linux kernel TCP/IP had been kicking arses
since day 0
■ Coupled with epoll/AIO
■ Can be tuned for both
○ RPC messages ping pong
○ Data streaming
■ System calls switches still matters

16. DPDK
16
■ Removes extra switches
■ Works better in poll mode

17. IRQ binding
17
■ NIC IRQ processing times
■ NIC Soft-IRQ processing times
■ Binding IRQs (and Soft-IRQs) to specific
cores

18. ScyllaDB Internals
18

19. Predictably Low
Latencies
19

20. CPU
20
■ Thread-per core architecture
■ User-space scheduler

21. Memory
21
■ Row-cache
■ Page-cache
■ Full control over allocations

22. Disk I/O
22
■ Direct I/O
■ User-space scheduler