Successfully reported this slideshow.
Your SlideShare is downloading. ×

Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays NA 2021

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 24 Ad

Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays NA 2021

Download to read offline

Telegraf, InfluxDB’s native data collector, supports nearly 300 inputs and outputs. Using Telegraf, you can send data from a variety of systems, apps, databases and services in the appropriate format to InfluxDB. This session covers Telegraf plugins and InfluxDB client libraries as well as early access to release candidates. Alan Pope and Sebastian Spaink will highlight recent updates to Telegraf — including the new JSON and XML parsers!

Telegraf, InfluxDB’s native data collector, supports nearly 300 inputs and outputs. Using Telegraf, you can send data from a variety of systems, apps, databases and services in the appropriate format to InfluxDB. This session covers Telegraf plugins and InfluxDB client libraries as well as early access to release candidates. Alan Pope and Sebastian Spaink will highlight recent updates to Telegraf — including the new JSON and XML parsers!

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays NA 2021 (20)

Advertisement

More from InfluxData (20)

Recently uploaded (20)

Advertisement

Alan Pope, Sebastian Spaink [InfluxData] | Data Collection 101 | InfluxDays NA 2021

  1. 1. Alan Pope | Engineering Manager, InfluxData Sebastian Spaink | Software Engineer, InfluxData Data Collection 101
  2. 2. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Talk outline • Introduction to Telegraf • What’s new with Telegraf? • New parser demos • Tiger bot and more! • Final words
  3. 3. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Writing to InfluxDB Several ways to write and query InfluxDB v2 • HTTP API • Client library • Telegraf
  4. 4. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Telegraf The plugin-driven server agent for collecting & reporting metrics • Over 300 plugins • Current release: v1.20.2 • Get artifacts here: https:/ /github.com/influxdata/telegraf/releases/tag/v1.20.2
  5. 5. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. How does Telegraf work? • Component called “agent” controlling main functionality • Extended functionality with plugins • Configured using TOML
  6. 6. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. The Crew & Community
  7. 7. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. New input plugins • OpenTelemetry • Receives traces, metrics and logs from OpenTelemetry clients and agents via gRPC • https://github.com/influxdata/telegraf/tree/master/plugins/inputs/opentelemetry • Elasticsearch Query • Queries endpoints to obtain metrics from data stored in an Elasticsearch cluster • https://github.com/influxdata/telegraf/tree/master/plugins/inputs/elasticsearch_query
  8. 8. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. New output plugins • Generic SQL Output Plugin • Saves Telegraf metric data to an SQL database • Multiple supported SQL databases • https://github.com/influxdata/telegraf/tree/master/plugins/outputs/sql • Azure Data Explorer Output Plugin • Azure Data Explorer is a distributed, columnar store, purpose-built for any type of logs, metrics and time series data • https://github.com/influxdata/telegraf/tree/master/plugins/outputs/azure_data_explorer
  9. 9. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. New Parser Demo Time! • json_v2 parser • A new and improved JSON parser • Still uses gjson path expressions • xpath parser • The first XML parser • Uses xpath path expressions • ISS (International space station) current location • http://api.open-notify.org/iss-now.json
  10. 10. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. ISS location JSON { "iss_position": { "longitude": "116.2486", "latitude": "-37.0576" }, "timestamp": 1633645318, "message": "success" }
  11. 11. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. What is GJSON ? ● GJSON Path: a text syntax that describes a search pattern to retrieve values from JSON ● Easily expressed as a series of components separated by a . character ● Given: {"example": { "test": 1 }} ● GJSON Path: example.test ● Returned value: 1 • Source: https://github.com/tidwall/gjson • Documentation: https://github.com/tidwall/gjson/blob/master/SYNTAX.md • Try it yourself: https://gjson.dev/
  12. 12. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. json_v2 config (v1.20) [[inputs.http]] interval = "1m" urls = ["http://api.open-notify.org/iss-now.json"] data_format = "json_v2" [[inputs.http.json_v2]] [[inputs.http.json_v2.object]] path = "iss_position" timestamp_path="timestamp" timestamp_format="unix" [inputs.http.json_v2.object.fields] longitude = "float" latitude = "float"
  13. 13. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. json_v2 config option #2 (v1.21) [[inputs.http]] interval = "1m" urls = ["http://api.open-notify.org/iss-now.json"] data_format = "json_v2" [[inputs.http.json_v2]] [[inputs.http.json_v2.object]] path = "@this" timestamp_path="timestamp" timestamp_format="unix" [[inputs.http.json_v2.object.field]] path = "iss_position.longitude" type = "float" [[inputs.http.json_v2.object.field]] path = "iss_position.latitude" type = "float"
  14. 14. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. ISS JSON converted to XML <?xml version="1.0" encoding="UTF-8" ?> <root> <iss_position> <longitude>116.2486</longitude> <latitude>-37.0576</latitude> </iss_position> <timestamp>1633645318</timestamp> <message>success</message> </root>
  15. 15. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. What is XPath? Recommended blog to learn more ⇒ Scan QR code XPath: path expressions to select nodes or node-sets in an XML document ● Given: <root> <example> 1 </example> </root> ● GJSON Path: /root/example ● Returned value: 1
  16. 16. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. xml config (v1.20) [[inputs.file]] files = ["input.xml"] data_format = "xml" [[inputs.file.xpath]] [inputs.file.xpath.fields] longitude = "number(/root/iss_position/longitude)" latitude = "number(/root/iss_position/latitude)"
  17. 17. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved.
  18. 18. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved.
  19. 19. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Community Shoutout • Thomas Casteleyn (hipska) • Paweł Żak (zak-pawel) • Sven Rebhan (srebhan) … (your name here?)
  20. 20. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Telegraf Developer Improvements • conventional commit messages • e.g. feat: or fix: • linter • Using: https://github.com/golangci/golangci-lint • improved CI times • 40 minutes -> 12 minutes
  21. 21. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Telegraf Tiger Bot • Updates to telegraf tiger bot • Checks CLA • Manages labels • Shares artifacts
  22. 22. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Telegraf Bot shares artifacts
  23. 23. © 2021  InfluxData Inc. All Rights Reserved. © 2021  InfluxData Inc. All Rights Reserved. Get involved! • Slack: https:/ /influxdata.com/slack • Forum: https:/ /community.influxdata.com/c/6 • Issues: https:/ /github.com/influxdata/telegraf/issues
  24. 24. © 2021  InfluxData Inc. All Rights Reserved. Thank you!

×