Different applications need different performance guarantees. Some applications need fastest possible ingest of a dataset (a hare). Other applications need write rate guarantees for every single record (a tortoise). The choice between the two becomes critical as CPU cores, memory size, and disk throughput decrease to save money on cloud virtual machines / AWS instances. This presentation demonstrates how to convert RocksDB's natural "hare mode" into "tortoise mode" for timing sensitive applications and/or lower cost, lower capability hardware. RocksDB's stalls and stops are explained.

1. Copyright © ArangoDB GmbH, 2017
Are you a Tortoise
or a Hare?
By Matthew Von-Maszewski
Annual RocksDB Meetup 2017

2. Copyright © ArangoDB GmbH, 2017
Understanding Google’s leveldb and derivatives:
Everything is about target environment
• Google leveldb: cell phone
• Facebook RocksDB: high end servers

3. Copyright © ArangoDB GmbH, 2017
High end server ingest
3
32 cores, 128G RAM, 1 FusionIO PCI drive card

4. Copyright © ArangoDB GmbH, 2017
High end server ingest versus AWS like server
4

5. Copyright © ArangoDB GmbH, 2017
Flow of write operations
5
Write()
Skip list (write buffer)
Recovery log
Immutable memory
(old write buffer)
Level-0 .sst (overlapping)
Level-1 .sst (sorted/overlapping)
Level-2 .sst (sorted/overlapping)
Level-3 .sst (sorted)
Level-4 .sst (sorted)
Level-5 .sst (sorted)
Level-6 .sst (sorted)
MANIFEST

6. Copyright © ArangoDB GmbH, 2017
Hare in cactus patch … (AWS like server)
6

7. Copyright © ArangoDB GmbH, 2017
… versus Tortoise in cactus patch
7

8. Copyright © ArangoDB GmbH, 2017
‣ Algorithm 1: pacing via dynamic smoothing algorithm
‣ Algorithm 2: prioritization via operating system
scheduler priorities
‣ Key capability: self tuning to current environment
https://github.com/arangodb/arangodb/blob/devel/arangod/RocksDBEngine/RocksDBThrottle.cpp
https://github.com/arangodb/arangodb/blob/devel/arangod/RocksDBEngine/RocksDBThrottle.h
External, two part fix
8

9. Copyright © ArangoDB GmbH, 2017
Algorithm 1: Pacing
9
arangodb::RocksDBThrottle
( rocksdb::EventListener )
rocksdb::WriteController

10. Copyright © ArangoDB GmbH, 2017 10
With cap_sys_nice capability
‣ 20 application threads
‣ 21 rocksdb flush threads
‣ 22 rocksdb compactions for level 0
‣ 23 rocksdb compactions all other
Without cap_sys_nice
‣ 20 application threads
‣ 21 rocksdb flush threads
‣ --
‣ 23 rocksdb compactions
Algorithm 2: Prioritization