In-Depth Look at Telegraf Data Collection and Parsing with JSON and XML Examples

Alan Pope | Engineering Manager,
InﬂuxData
Sebastian Spaink | Software Engineer,
InﬂuxData
Data Collection 101

© 2021 InﬂuxData Inc. All Rights Reserved.
Talk outline
• Introduction to Telegraf
• What’s new with Telegraf?
• New parser demos
• Tiger bot and more!
• Final words

Writing to InﬂuxDB
Several ways to write and query InﬂuxDB v2
• HTTP API
• Client library
• Telegraf

Telegraf
The plugin-driven server agent for collecting & reporting metrics
• Over 300 plugins
• Current release: v1.20.2
• Get artifacts here:
https:/
/github.com/inﬂuxdata/telegraf/releases/tag/v1.20.2

How does Telegraf work?
• Component called “agent” controlling main functionality
• Extended functionality with plugins
• Conﬁgured using TOML

The Crew & Community

New input plugins
• OpenTelemetry
• Receives traces, metrics and logs from OpenTelemetry clients and agents via
gRPC
• https://github.com/influxdata/telegraf/tree/master/plugins/inputs/opentelemetry
• Elasticsearch Query
• Queries endpoints to obtain metrics from data stored in an Elasticsearch cluster
• https://github.com/influxdata/telegraf/tree/master/plugins/inputs/elasticsearch_query

New output plugins
• Generic SQL Output Plugin
• Saves Telegraf metric data to an SQL database
• Multiple supported SQL databases
• https://github.com/influxdata/telegraf/tree/master/plugins/outputs/sql
• Azure Data Explorer Output Plugin
• Azure Data Explorer is a distributed, columnar store, purpose-built for any type of
logs, metrics and time series data
• https://github.com/influxdata/telegraf/tree/master/plugins/outputs/azure_data_explorer

New Parser Demo Time!
• json_v2 parser
• A new and improved JSON parser
• Still uses gjson path expressions
• xpath parser
• The first XML parser
• Uses xpath path expressions
• ISS (International space station) current location
• http://api.open-notify.org/iss-now.json

ISS location JSON
{
"iss_position": {
"longitude": "116.2486",
"latitude": "-37.0576"
},
"timestamp": 1633645318,
"message": "success"
}

What is GJSON ?
● GJSON Path: a text syntax that describes a search pattern to retrieve values from JSON
● Easily expressed as a series of components separated by a . character
● Given: {"example": { "test": 1 }}
● GJSON Path: example.test
● Returned value: 1
• Source: https://github.com/tidwall/gjson
• Documentation: https://github.com/tidwall/gjson/blob/master/SYNTAX.md
• Try it yourself: https://gjson.dev/

json_v2 config (v1.20)
[[inputs.http]]
interval = "1m"
urls = ["http://api.open-notify.org/iss-now.json"]
data_format = "json_v2"
[[inputs.http.json_v2]]
[[inputs.http.json_v2.object]]
path = "iss_position"
timestamp_path="timestamp"
timestamp_format="unix"
[inputs.http.json_v2.object.fields]
longitude = "float"
latitude = "float"

json_v2 config option #2 (v1.21)
[[inputs.http]]
interval = "1m"
urls = ["http://api.open-notify.org/iss-now.json"]
data_format = "json_v2"
[[inputs.http.json_v2]]
[[inputs.http.json_v2.object]]
path = "@this"
timestamp_path="timestamp"
timestamp_format="unix"
[[inputs.http.json_v2.object.field]]
path = "iss_position.longitude"
type = "float"
[[inputs.http.json_v2.object.field]]
path = "iss_position.latitude"
type = "float"

ISS JSON converted to XML
<?xml version="1.0" encoding="UTF-8" ?>
<root>
<iss_position>
<longitude>116.2486</longitude>
<latitude>-37.0576</latitude>
</iss_position>
<timestamp>1633645318</timestamp>
<message>success</message>
</root>

What is XPath?
Recommended blog to learn more ⇒
Scan QR code
XPath: path expressions to select nodes or node-sets in an XML document
● Given:
<root>
<example>
1
</example>
</root>
● GJSON Path: /root/example
● Returned value: 1

xml config (v1.20)
[[inputs.file]]
files = ["input.xml"]
data_format = "xml"
[[inputs.file.xpath]]
[inputs.file.xpath.fields]
longitude =
"number(/root/iss_position/longitude)"
latitude = "number(/root/iss_position/latitude)"

Community Shoutout
• Thomas Casteleyn (hipska)
• Paweł Żak (zak-pawel)
• Sven Rebhan (srebhan)
… (your name here?)

Telegraf Developer Improvements
• conventional commit messages
• e.g. feat: or fix:
• linter
• Using: https://github.com/golangci/golangci-lint
• improved CI times
• 40 minutes -> 12 minutes

Telegraf Tiger Bot
• Updates to telegraf tiger bot
• Checks CLA
• Manages labels
• Shares artifacts

Telegraf Bot shares artifacts

Get involved!
• Slack:
https:/
/influxdata.com/slack
• Forum:
https:/
/community.influxdata.com/c/6
• Issues:
https:/
/github.com/influxdata/telegraf/issues

Thank you!

In-Depth Look at Telegraf Data Collection and Parsing with JSON and XML Examples

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to In-Depth Look at Telegraf Data Collection and Parsing with JSON and XML Examples

Similar to In-Depth Look at Telegraf Data Collection and Parsing with JSON and XML Examples (20)

More from InfluxData

More from InfluxData (20)

Recently uploaded

Recently uploaded (20)

In-Depth Look at Telegraf Data Collection and Parsing with JSON and XML Examples