Paul Dix
InfluxData – CTO & co-founder
@pauldix
InfluxDB IOx
overview and
updates

© 2021 InfluxData. All rights reserved. 10
| Project Goals
• Time series & events
– Metrics (sensor or server)
– Tracing (or any event, really)
– Logs (semi-structured events)
• Dashboards/monitoring AND Analytics
• Unlimited cardinality
• Decoupled compute & storage
• Bulk import/export
• Replication & Subscriptions
• Embeddable scripting
• Broader ecosystem integration

© 2021 InfluxData. All rights reserved. 15
| Operational Environments
• Public Clouds (AWS, GCP, Azure)
• Kubernetes
• Other data center
• Edge devices

© 2021 InfluxData. All rights reserved. 20
| InfluxDB IOx Highlights
• In-memory columnar database
• Object store for persistence
• Supports SQL natively
• Flux & InfluxQL support through API

© 2021 InfluxData. All rights reserved. 21

© 2021 InfluxData. All rights reserved. 22

© 2021 InfluxData. All rights reserved. 27
| arrow-rs
• RecordBatch & compute kernels
• Parquet reader/writer
• Arrow Flight RPC layer
• DataFusion SQL parser, planner, executor

© 2021 InfluxData. All rights reserved. 28
&

How data is organized
telegrafdb
2021-05-14T10:00 2021-05-14T11:00 2021-05-14T12:00
cpu mem cpu mem
0 1 0 1
Database (bucket)
Partition
Table (measurement)
Chunk

© 2021 InfluxData. All rights reserved. 35
| Mapping InfluxDB to IOx & SQL
• Bucket = database
• Measurement = table
• Tag = column (dictionary encoded)
• Field = column
• Time = column

© 2021 InfluxData. All rights reserved. 36

© 2021 InfluxData. All rights reserved. 37

The big disk in the sky

© 2021 InfluxData. All rights reserved. 39
| Backing Object Stores
• Memory
• Locally attached disk
• S3
• Azure blob store
• Google Cloud Storage (objects)
• Anything compatible

© 2020 InfluxData. All rights reserved.
40
2815898179/
my_db/
data/
2020-01/
0/
sensors.parquet
stocks.parquet
1/
sensors.parquet
2020-02/
0/
sensors.parquet
1/
sensors.parquet

© 2020 InfluxData. All rights reserved.
41
2815898179/
my_db/
data/
2020-01/
0/
sensors.parquet
stocks.parquet
1/
sensors.parquet
2020-02/
0/
sensors.parquet
1/
sensors.parquet
Writer ID

© 2020 InfluxData. All rights reserved.
42
2815898179/
my_db/
data/
2020-01/
0/
sensors.parquet
stocks.parquet
1/
sensors.parquet
2020-02/
0/
sensors.parquet
1/
sensors.parquet
Database

© 2020 InfluxData. All rights reserved.
43
2815898179/
my_db/
data/
2020-01/
0/
sensors.parquet
stocks.parquet
1/
sensors.parquet
2020-02/
0/
sensors.parquet
1/
sensors.parquet
Partition

© 2020 InfluxData. All rights reserved.
44
2815898179/
my_db/
data/
2020-01/
0/
sensors.parquet
stocks.parquet
1/
sensors.parquet
2020-02/
0/
sensors.parquet
1/
sensors.parquet
Chunk

© 2020 InfluxData. All rights reserved.
45
2815898179/
my_db/
data/
2020-01/
0/
sensors.parquet
stocks.parquet
1/
sensors.parquet
2020-02/
0/
sensors.parquet
1/
sensors.parquet
Table

Indexing?

© 2021 InfluxData. All rights reserved. 51
| IOx Components
• Write Buffer
• MUB (Mutable Buffer)
• RUB (Read Buffer)
• Catalog

© 2021 InfluxData. All rights reserved. 52
Query Engine
SQL Frontend
Flux Frontend
InfluxQL
Frontend
Mutable
Buffer
Read Buffer
Object
Storage
Reader
| IOx Read Path

© 2021 InfluxData. All rights reserved. 53
Query Engine
SQL Frontend
Flux Frontend
InfluxQL
Frontend
Mutable
Buffer
Read Buffer
Object
Storage
Reader
In-memory
| IOx Read Path

© 2021 InfluxData. All rights reserved. 59
| RUB techniques
• Dictionary Encoding
• RLE
• Byte-trimming (ints and natural floats)
• Frame of reference encoding
• Handling nulls

© 2021 InfluxData. All rights reserved. 60

© 2021 InfluxData. All rights reserved. 61

© 2021 InfluxData. All rights reserved. 62

© 2021 InfluxData. All rights reserved. 63

API

© 2021 InfluxData. All rights reserved. 68
| Write
• InfluxDB 2.x compatible (HTTP & Line Protocol)
• Protobuf
• /write
• gRPC service definition
• Bulk via Parquet
• IOx direct
• Object Store

© 2021 InfluxData. All rights reserved. 71
| Read
• HTTP GET
/iox/api/v1/databases/:name/query?q=..&format=..
• SQL query
• Format: pretty, csv, json
• Arrow Flight

© 2021 InfluxData. All rights reserved. 77
| Manage
• gRPC
• CLI
• Create database
• Manage lifecycle rules
• Persist to object store
• Evict from memory
• Load from object store (soon)
• Subscriptions (soon)

© 2021 InfluxData. All rights reserved. 83
| Road to Release
• Dogfooding in production (in process)
• Optimization
• Memory footprint
• Query execution
• Recovery from object store
• Available in InfluxDB Cloud (Q3)
• Migration tooling
• Docs, Docker builds and Cargo crate (later)

© 2021 InfluxData. All rights reserved. 84
| Find out more
• #influxdb-iox in community Slack
• Tech talk every 2nd Wednesday at 8:00 AM PT (15:00 GMT)
• github.com/influxdata/influxdb_iox

Q&A