Isolating Streaming Ingest and Queries Using RocksDB
"In a real-time analytics architecture, streaming data ingestion, from a source like Kafka, and query serving run on the same compute unit, so that queries can reflect newly ingested data. These two distinct competing functions invariably contend for the available compute resources, which makes it difficult to handle situations where there are unexpected bursts of either streaming ingestion or queries that can slow down the system. We will examine common approaches to the problem of compute contention, such as scaling, replication, and querying from shared storage, and discuss their tradeoffs and how they remain incomplete solutions. In this talk, we will present a real-time analytics architecture we implemented in the Rockset database, based on RocksDB, that effectively isolates streaming data ingestion from query serving. RocksDB is a popular log-structured merge-tree storage engine that writes to an in-memory memtable and periodically flushes to disk. Core to our architecture is the separation of compute and storage. This allows multiple RocksDB instances to query from the same shared storage. We use cloud object storage to ensure durability and use SSD as a shared hot storage tier for low-latency reads. On the compute side, we designed our query processing engine to be completely separate from all the modules that perform data ingestion. For fresh data to be available to multiple compute units, it is essential that the in-memory state of the ingester's RocksDB memtable be replicated to other RocksDB instances. We built a RocksDB memtable replicator that propagates changes to remote instances in single-digit milliseconds. This architecture enables compute isolation so that real-time streaming ingestion does not interfere with queries, while still allowing the most recent data to be queried."
Recommended
Recommended
More Related Content
Similar to Isolating Streaming Ingest and Queries Using RocksDB
Similar to Isolating Streaming Ingest and Queries Using RocksDB (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Isolating Streaming Ingest and Queries Using RocksDB
- 2. 2 Apps ? Streaming records + deltas Search queries Vector search Real-time analytics SQL
- 3. 3 Apps Streaming records + deltas Search queries Vector search Real-time analytics 1. Sharding → scalability
- 4. Sharding choices and tradeoffs 4 Value- dependent mapping? Data + indexes together? no yes yes no ✅ Opportunities for larger read I/Os ❌ Coordination overheads balloon as ingest latency drops Can documents change? yes no ❌ Unable to support most search and analytics apps Doc sharding ❗Small read I/Os ✅ Efficient streaming ingest ✅ Consistent indexes
- 5. 5 Apps Streaming records + deltas Search queries Vector search Real-time analytics 1. Doc sharding → scalability + streaming ingest
- 6. 6 2. Isolation between ingest and query work
- 7. 7 2. Post-ingest replication → isolation + elasticity App B App A Leader Follower
- 8. 8 3. Disaggregated storage → efficiency + elasticity App B App A
- 9. What technology for disaggregated storage? 9 Cold (AWS S3) Hot (EBS or NVMe) 👍 Cheapest $/GB 👍 Highly durable 👍 Built-in RPC API 👎 High/unpredictable latency 👎 Expensive $/IOPS 👍 Cheapest $/IOPS 👍 Low latency 👍+👎 Build/run your own RPC service 👎 More expensive $/GB
- 10. Cloud-native search + analytics 1. Doc sharding with indexes – Converged indexing → Scalability → Streaming ingest 2. Post-ingest replication – Compute:compute separation → Isolation → Compute elasticity 3. Disaggregated hot storage – Compute:storage separation → High disk utilization → Compute elasticity → Storage elasticity 10
- 12. Shared hot storage 1 Rockset shard ≈ 1 RocksDB instance Data stream Rockset stores data for each shard in RocksDB Writes are flushed to storage after the memtable is full SSD Query Ingest App Memtable 12
- 13. Document:RocksDB-key mapping is M:N 13
- 14. Shared hot storage Ingest turns logical update into physical deltas Data stream Ingesting one document applies a delta to many RocksDB keys Deltas are merged lazily by RocksDB SSD Query Ingest App Memtable 14
- 15. Shared hot storage Fine-grained RocksDB replication Data stream App Leader/follower replication makes fresh data available in all RocksDB instances ● Replication stream sends data and metadata changes ● Applying memtable updates takes 6× to 10× less CPU than ingest ● Followers don’t run compaction SSD Optional Query Ingest Query App Memtable Memtable 15
- 16. Shared hot storage Compute:compute separation for vector search Data stream App IVF ANN (inverted file approximate nearest neighbor) ● Leader periodically builds Voronoi cell decomposition of vector space ● Cell membership can be updated in real-time ● Follower queries are executed the same as an inverted index lookup SSD IVF partitioning Ingest Query Memtable Memtable 16 IVF cells
- 18. RocksDB is a log-structured merge tree (LSM) 1. Writes are buffered in RAM in a “memtable” 2. Many megabytes of values are written to a new file at once 3. Files are immutable, and store keys in sorted order 4. “Compaction” creates new files by merging several old files, making things more sorted and removing duplicates Big async writes Disk Memory 18
- 19. Rockset uses indexes to accelerate queries Query fragment Cost-based optimizer Column scan Index lookup Filter Fetch other cols in matching rows ● Large reads ● Bandwidth limited ● Small reads ● Latency limited ● IOPS limited 19
- 20. Big writes Small reads Write to S3 Read from SSD 20
- 21. Shared hot storage Write to S3 + read from SSD Data stream App ● (Big) writes go to AWS S3 ○ High BW ○ Ensures durability ● Reads go to 1-copy SSD ○ Low latency ○ High IOPS ○ Efficient small reads ○ High space utilization SSD Query Ingest Query App S3 Memtable Memtable 21
- 22. Challenge: Cache miss to S3 can be 1,000× slower ● Cold misses ○ Synchronous prefetch on file creation ○ Prefetch from periodic S3 list ● Capacity misses ○ Cluster auto-scaling ● Software restart for upgrades ○ Dual-head serving during rolls ● Cluster resizing ○ Double-reading with rendezvous hashing ● Failure recovery ○ Whole-cluster recovery with rendezvous hashing 22
- 23. IT HAS BEEN DAYS SINCE THE LAST CACHE MISS 6 Rockset hot storage is a near-perfect S3 cache ● Cold misses ○ Synchronous prefetch on file creation ○ Prefetch from periodic S3 list ● Capacity misses ○ Cluster auto-scaling ● Software restart for upgrades ○ Dual-head serving during rolls ● Cluster resizing ○ Second-chance reads with recent old configs ● Failure recovery ○ Whole-cluster recovery with rendezvous hashing 99.9999% cache hit rate https://rockset.com/blog/separate-compute-storage-rocksdb/ 23
- 24. Rockset ● Real-time indexing ● Cloud-native efficiency ● Full-featured SQL 24