Successfully reported this slideshow.

Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays EMEA 2021

1

Share

Loading in …3
×
1 of 36
1 of 36

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Paul Dix [InfluxData] | InfluxDays Opening Keynote | InfluxDays EMEA 2021

  1. 1. Paul Dix InfluxData – CTO & co-founder @pauldix InfluxDB IOx overview and updates
  2. 2. © 2021 InfluxData. All rights reserved. 10 | Project Goals • Time series & events – Metrics (sensor or server) – Tracing (or any event, really) – Logs (semi-structured events) • Dashboards/monitoring AND Analytics • Unlimited cardinality • Decoupled compute & storage • Bulk import/export • Replication & Subscriptions • Embeddable scripting • Broader ecosystem integration
  3. 3. © 2021 InfluxData. All rights reserved. 15 | Operational Environments • Public Clouds (AWS, GCP, Azure) • Kubernetes • Other data center • Edge devices
  4. 4. © 2021 InfluxData. All rights reserved. 20 | InfluxDB IOx Highlights • In-memory columnar database • Object store for persistence • Supports SQL natively • Flux & InfluxQL support through API
  5. 5. © 2021 InfluxData. All rights reserved. 21
  6. 6. © 2021 InfluxData. All rights reserved. 22
  7. 7. © 2021 InfluxData. All rights reserved. 27 | arrow-rs • RecordBatch & compute kernels • Parquet reader/writer • Arrow Flight RPC layer • DataFusion SQL parser, planner, executor
  8. 8. © 2021 InfluxData. All rights reserved. 28 &
  9. 9. How data is organized telegrafdb 2021-05-14T10:00 2021-05-14T11:00 2021-05-14T12:00 cpu mem cpu mem 0 1 0 1 Database (bucket) Partition Table (measurement) Chunk
  10. 10. © 2021 InfluxData. All rights reserved. 35 | Mapping InfluxDB to IOx & SQL • Bucket = database • Measurement = table • Tag = column (dictionary encoded) • Field = column • Time = column
  11. 11. © 2021 InfluxData. All rights reserved. 36
  12. 12. © 2021 InfluxData. All rights reserved. 37
  13. 13. The big disk in the sky
  14. 14. © 2021 InfluxData. All rights reserved. 39 | Backing Object Stores • Memory • Locally attached disk • S3 • Azure blob store • Google Cloud Storage (objects) • Anything compatible
  15. 15. © 2020 InfluxData. All rights reserved. 40 2815898179/ my_db/ data/ 2020-01/ 0/ sensors.parquet stocks.parquet 1/ sensors.parquet 2020-02/ 0/ sensors.parquet 1/ sensors.parquet
  16. 16. © 2020 InfluxData. All rights reserved. 41 2815898179/ my_db/ data/ 2020-01/ 0/ sensors.parquet stocks.parquet 1/ sensors.parquet 2020-02/ 0/ sensors.parquet 1/ sensors.parquet Writer ID
  17. 17. © 2020 InfluxData. All rights reserved. 42 2815898179/ my_db/ data/ 2020-01/ 0/ sensors.parquet stocks.parquet 1/ sensors.parquet 2020-02/ 0/ sensors.parquet 1/ sensors.parquet Database
  18. 18. © 2020 InfluxData. All rights reserved. 43 2815898179/ my_db/ data/ 2020-01/ 0/ sensors.parquet stocks.parquet 1/ sensors.parquet 2020-02/ 0/ sensors.parquet 1/ sensors.parquet Partition
  19. 19. © 2020 InfluxData. All rights reserved. 44 2815898179/ my_db/ data/ 2020-01/ 0/ sensors.parquet stocks.parquet 1/ sensors.parquet 2020-02/ 0/ sensors.parquet 1/ sensors.parquet Chunk
  20. 20. © 2020 InfluxData. All rights reserved. 45 2815898179/ my_db/ data/ 2020-01/ 0/ sensors.parquet stocks.parquet 1/ sensors.parquet 2020-02/ 0/ sensors.parquet 1/ sensors.parquet Table
  21. 21. Indexing?
  22. 22. © 2021 InfluxData. All rights reserved. 51 | IOx Components • Write Buffer • MUB (Mutable Buffer) • RUB (Read Buffer) • Catalog
  23. 23. © 2021 InfluxData. All rights reserved. 52 Query Engine SQL Frontend Flux Frontend InfluxQL Frontend Mutable Buffer Read Buffer Object Storage Reader | IOx Read Path
  24. 24. © 2021 InfluxData. All rights reserved. 53 Query Engine SQL Frontend Flux Frontend InfluxQL Frontend Mutable Buffer Read Buffer Object Storage Reader In-memory | IOx Read Path
  25. 25. © 2021 InfluxData. All rights reserved. 59 | RUB techniques • Dictionary Encoding • RLE • Byte-trimming (ints and natural floats) • Frame of reference encoding • Handling nulls
  26. 26. © 2021 InfluxData. All rights reserved. 60
  27. 27. © 2021 InfluxData. All rights reserved. 61
  28. 28. © 2021 InfluxData. All rights reserved. 62
  29. 29. © 2021 InfluxData. All rights reserved. 63
  30. 30. API
  31. 31. © 2021 InfluxData. All rights reserved. 68 | Write • InfluxDB 2.x compatible (HTTP & Line Protocol) • Protobuf • /write • gRPC service definition • Bulk via Parquet • IOx direct • Object Store
  32. 32. © 2021 InfluxData. All rights reserved. 71 | Read • HTTP GET /iox/api/v1/databases/:name/query?q=..&format=.. • SQL query • Format: pretty, csv, json • Arrow Flight
  33. 33. © 2021 InfluxData. All rights reserved. 77 | Manage • gRPC • CLI • Create database • Manage lifecycle rules • Persist to object store • Evict from memory • Load from object store (soon) • Subscriptions (soon)
  34. 34. © 2021 InfluxData. All rights reserved. 83 | Road to Release • Dogfooding in production (in process) • Optimization • Memory footprint • Query execution • Recovery from object store • Available in InfluxDB Cloud (Q3) • Migration tooling • Docs, Docker builds and Cargo crate (later)
  35. 35. © 2021 InfluxData. All rights reserved. 84 | Find out more • #influxdb-iox in community Slack • Tech talk every 2nd Wednesday at 8:00 AM PT (15:00 GMT) • github.com/influxdata/influxdb_iox
  36. 36. Q&A

×