Gleb Natapov presents on using in-memory storage in ScyllaDB to provide faster access times compared to disk-based storage. He discusses how ScyllaDB uses SSTables that are stored on disk by default, which can incur latency penalties during cache misses. Natapov introduces new in-memory and mirror file types that allow storing SSTable contents directly in RAM to avoid disk access latency. He demonstrates how to configure ScyllaDB to reserve RAM for in-memory storage and to create tables specifying the in-memory option to gain predictably low latency performance.

1. In Memory
When Fast Storage Is Not fast Enough
Gleb Natapov
Developer At ScyllaDB

2. Presenter bio
I am a long-time open source developer who previously
worked on the open source routing suite “Zebra”, the
OpenMPI HPC library, KVM hypervisor for Linux, the OSv
unikernel, and now Scylla.

3. Motivation

4. Scylla Single Node Storage Model

5. Cache Miss

6. SSTable Format
One SSTable consists of multiple files:
la-1-big-CRC.db
la-1-big-Data.db
la-1-big-Digest.sha1
la-1-big-Filter.db
la-1-big-Index.db
la-1-big-Scylla.db
la-1-big-Statistics.db
la-1-big-Summary.db
la-1-big-TOC.txt

7. SStable Format (Cont.)

8. SStable Read
At least two storage accesses per SSTable

9. Storage Speed
Device Latency
Register 1 Cycle
Cache 2-10ns
DRAM 100-200ns
NVMe 10-100us
SATA SSD 400us
Hard Disk 10ms

10. Cache Miss Price
With Average 2 SSTables Per Cache Miss on NVMe
~200us
Not too bad?

11. Cache Miss Price (In Real World)
▪ SSD becomes slower over time
▪ fstrim is running in the background
▪ Many parallel requests cause tail latency to grow
Real latency may be much bigger:

12. IOPING
No other IO:
99 requests completed in 8.54 ms, 396 KiB read, 11.6 k iops, 45.3 MiB/s
generated 100 requests in 990.3 ms, 400 KiB, 100 iops, 403.9 KiB/s
min/avg/max/mdev = 59.6 us / 86.3 us / 157.8 us / 27.2 us
Read/Write fio benchmark:
99 requests completed in 34.2 ms, 396 KiB read, 2.90 k iops, 11.3 MiB/s
generated 100 requests in 990.3 ms, 400 KiB, 100 iops, 403.9 KiB/s
min/avg/max/mdev = 73.0 us / 345.2 us / 5.74 ms / 694.3 us
fstrim:
99 requests completed in 300.3 ms, 396 KiB read, 329 iops, 1.29 MiB/s
generated 100 requests in 1.24 s, 400 KiB, 80 iops, 323.5 KiB/s
min/avg/max/mdev = 62.2 us / 3.03 ms / 83.4 ms / 14.5 ms

13. In-Memory vs On-Disk

14. Implementation

15. New “In-Memory” File Type
Store its content in RAM

16. New “Mirror” File Type
▪ Writes to both regular and in-memory file, reads from memory only:
Mirror file
Disk file
memory
file
Mirror file
Disk file
memory
file
write path read path
▪ Read everything into memory when opened.

17. Configuration

18. RAM Reservation
Command line:
--in-memory-storage-size-mb
Scylla.yaml:
in_memory_storage_size_mb:

19. Table Creation
CREATE TABLE ks.cf (
key blob PRIMARY KEY,
"C0" blob,
) WITH compression = {}
AND read_repair_chance = '0'
AND speculative_retry = 'ALWAYS'
AND in_memory = 'true'
AND compaction = {'class':'InMemoryCompactionStrategy'};

20. InMemoryCompactionStrategy
Compacts aggressively until only one SSTable is left.

21. Use In-Memory
For Predictably Low Latency

22. Thank You
Any Questions ?
Please stay in touch
gleb@scylladb.com