Successfully reported this slideshow.
Your SlideShare is downloading. ×

Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry | InfluxDays NA 2021

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 25 Ad

More Related Content

Slideshows for you (20)

Similar to Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry | InfluxDays NA 2021 (20)

Advertisement

More from InfluxData (20)

Recently uploaded (20)

Advertisement

Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry | InfluxDays NA 2021

  1. 1. Observability with InfluxDB/IOx & OpenTelemetry Jacob Marble, InfluxData
  2. 2. © 2021  InfluxData Inc. All Rights Reserved. About Me
  3. 3. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. InfluxDB/TSM: Definition ● Columnar Series-oriented storage engine ○ Like a log-structured merge-tree ■ Key is roughly concat(tags, field key, timestamp) ■ Value is field value ○ “Column per series” ○ Leverages mmap() syscall ● Series index ○ Index to data by concat(tags, field key) ● Introduced in 2015 ○ Superseded engine based on BoltDB ● Implemented in Golang
  4. 4. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. InfluxDB/TSM: Good The TSM storage engine is good at: ● Storage capacity: hundreds of GBs ● Storage cardinality: millions of series ● Query limits: selective, analytic ● Strong use cases: IoT, metrics, events*
  5. 5. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. InfluxDB/IOx: Definition ● Distributed columnar storage engine ○ Parquet ■ Columns are typed data columns ■ Time is a column type ■ Tags, fields: semantic property of columns ○ Arrow, Flight ■ In-memory, network ○ Federated storage, write, query ■ Collection, storage, query ● No series index ○ Partition keys ○ Dictionary encoding ● Under development since 2020 ● Implemented in Rust
  6. 6. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. InfluxDB/IOx: Better The IOx storage engine is good at: ● Storage capacity: unlimited ● Storage cardinality: unlimited ● Query limits: network protocols ● Strong use cases: IoT, metrics, events, logs, traces, ...
  7. 7. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry: Definition All observability signals in one standard. ✅ Tracing ✅ Logs ✅ Metrics The project has wide industry support. ● Top-tier cloud vendors and APM SaaS are committed. ● SDKs for ~10 languages. ● Second most active CNCF project in 2020. ○ Most active was Kubernetes.
  8. 8. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Open Source Observability: Today metrics logs traces instrumentation Prometheus printf() Jaeger collection Prometheus Expo Logfmt, JSON Thrift or gRPC delivery Pull via HTTP Push via stdout Push via UDP pipeline Telegraf Fluentd Jaeger storage InfluxDB MongoDB Elasticsearch query Grafana Graylog Jaeger
  9. 9. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Open Source Observability: Today application application application application application application application application Telegraf Fluentd Jaeger InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger SDK SDK Prom. Expo Logfmt Jaeger Thrift SDK
  10. 10. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry: Tomorrow metrics logs traces instrumentation OpenTelemetry SDK collection OTLP delivery Push via gRPC pipeline OpenTelemetry Collector storage ? query
  11. 11. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry: Tomorrow application application application application application application application application OTel SDK ? OTLP OTel Collector
  12. 12. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry SDK: Versatile application application application application application application application application OTel SDK Telegraf Fluentd Jaeger Prom. Expo Logfmt Jaeger Thrift InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger
  13. 13. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Versatile OTel Collector InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger application application application application application application application application SDK SDK SDK Prom. Expo Logfmt Jaeger Thrift
  14. 14. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry Migration: Incremental InfluxDB MongoDB Elasticsearch Grafana Graylog Jaeger application application application application application application application application OTel SDK OTLP OTel Collector
  15. 15. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry SDK: Span ID Everywhere application application application application application application application application span := newSpan(parent) defer span.Finish() log.Warn("error...") OTel SDK span ID: 3849 log record spanID: 3849 OTel Collector
  16. 16. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Temporal Aggregation OpenTelemetry Collector - bar = avg(15m) metric: foo gauge 10s metric: bar gauge 15m
  17. 17. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Spatial Aggregation metric: foo labels: k8s_node metric: bar labels: ∅ OpenTelemetry Collector - bar = sum()
  18. 18. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Traces→Metrics span: foo duration: 456ms metric: bar histogram OpenTelemetry Collector - if (span = foo) then bar = hist(duration)
  19. 19. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry Collector: Tail Sampling span: foo duration: 456ms trace sampled OpenTelemetry Collector - if (span = foo && duration > 100ms) then sample trace
  20. 20. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry: Vendor Neutral application application application application application application application application OTel SDK ? OTel Collector OTLP
  21. 21. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry: SaaS APM application application application application application application application application OTel Collector OTel SDK $aa$ Very Fancy UI OTLP OTLP
  22. 22. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. OpenTelemetry: Open Source End-to-End application application application application application application application application OTel SDK Something People Want Object Storage InfluxDB/IOx OTel Collector OTLP OTLP
  23. 23. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. InfluxData 💜 OpenTelemetry: Status ● InfluxData common schema for observability ○ github.com/influxdata/influxdb-observability ● OTel Collector sends Line Protocol ● OTel Collector receives Line Protocol ● Telegraf sends OTLP ● Telegraf receives OTLP ● Jaeger UI queries IOx (prototype)
  24. 24. © 2021  InfluxData Inc. All Rights Reserved. Questions
  25. 25. © 2021  InfluxData Inc. All Rights Reserved. Thank you Jacob Marble, InfluxData jacobmarble@influxdata.com

×