This document discusses the components and architecture of InfluxDB IOx for replication, durability, and subscriptions. It describes the write buffer, how writes are routed and distributed across shards, replication between buffers to ensure durability, and how subscriptions are handled for querying data.

1. Paul Dix
InfluxData – CTO & co-founder
paul@influxdata.com
@pauldix
Replication, durability and
subscriptions in InfluxDB IOx

2. InfluxDB IOx components
• Write Buffer
• Write segment persistence
• Subscriptions
• In-memory database
• Parquet persistence
• Catalog
• Configuration, partition summaries

4. Routing Layer
• Parse LP
• Split into shards
• Split into partitions
• Write one Entry per
shard

5. Possible Destinations
• Single server or
multiple servers
• Write Buffer
• In-memory DB

6. The Write Buffer
• Subscriptions
• Persistence
• Replication

7. The Write Buffer
• Router to Buffer

8. The Write Buffer
• Buffer assigns
clock value &
writer id
(unique)
• Makes request
to peers
• Note the
request from
Router to Buffer
still open

9. The Write Buffer
• Buffer 2 takes
the write,
updates its
clock to max
• Buffer 2
responds with
its clock value
• Buffer 1
updates its
clock to the
max of the two

10. The Write Buffer
• Buffer 1 sends
success
response to
routing layer
• Routing layer
responds to the
client (after all
shards succeed)

11. The Write Buffer
• Continues to
retry

12. The Write Buffer
• Consistency
checks

13. The Write Buffer
• Ask peers for
anything
missing
• Logical clock +
entry id
• High water mark

14. The Write Buffer
• Persistence to
object storage
• Entries in
segments
• Time or size

15. Subscriptions
• Everything
• Table (or regex)
• Predicate
• Push or Pull
• Or poll object
storage for
segments

16. Subscriptions
• Scale query tier

17. Catalog & Parquet
• Persist in larger
blocks

19. Thank You