Getting Started: Intro to Telegraf - July 2021

1. Samantha Wang Product Manager, InﬂuxData Getting Started: Intro to Telegraf

4. © 2021 InﬂuxData. All rights reserved. 4 What is InﬂuxDB? Characteristics of the data – Time-stamped – Generated in regular (metric) and irregular (event) time periods – Huge volumes – Real time and time sensitive

9. © 2021 InﬂuxData. All rights reserved. 9 9 Current Data Acquisition - All Methods Agent-based Push (aka Telegraf) Agentless Pull (aka Scrapers) Client Libraries Native/Ecosystem ● 200+ Telegraf plugins ● Regular cadence of releases ● Why use it? ○ No code ○ Large community ○ Lightweight but powerful ○ Customizable ● Prometheus scraper (OSS only) ● Flux prometheus.from ● Flux csv.from(url) ● Why use them? ○ Get data in quickly ○ Doesn’t require agent downloads on monitoring device ● 7 Libraries: Python, C#, Java, GO, Javascript/Node.js, Ruby, PHP ● Handles batching, chunking, setting right headers, etc. ● Why use them? ○ Easy way to get started ○ Need libraries when building custom applications ● Source system speaks line protocol ● Examples: JMeter, NiFi, Vector, FluentD ● Inﬂux CLI CSV Import ● Why use them? ○ Know what you want to monitor, quick and easy integration

10. © 2021 InﬂuxData. All rights reserved. 10 Telegraf provides the beneﬁts of… ● Low/No code ● Robust scheduler ● High-speed ingestion ● Full-streaming support ● Metric routing ● Flexible parsing, formatting, serializing ● Customizable and Extensible (ExecD plugins, Starlark) Instead of… ● Writing long data scraping scripts ● Worrying about unreliable data collection ● Trouble scaling your data collection ● Resulting in messy data ● Having a lot of unnecessary data in your database

11. © 2021 InﬂuxData. All rights reserved. 11 11 Telegraf: Agent for Collecting Metrics & Events Plugin-driven server agent for collecting and reporting metrics • Written in Go • Single Binary, No external dependencies • Minimal memory footprint • Optimized for writing to InﬂuxDB • Optimized for streaming data Input plugins to source a variety of metrics • directly from the system it’s running on • pull metrics from third party APIs • listen for metrics via a statsd and Kafka consumer services

12. © 2021 InfluxData. All rights reserved. 12 12 Telegraf: Agent for Collecting Metrics & Events Output plugins to send metrics to a variety of other datastores, services, and message queues • InfluxDB, Graphite, OpenTSDB, Datadog, and many others • Writes to multiple outputs at the same time Metrics routing and conversion Aggregation stage for InfluxDB

13. © 2021 InfluxData. All rights reserved. 13 13 Telegraf Agent – Core of Telegraf functionality providing the following – Robust scheduler – Adjustments for clock-drift – Adjustments for job scheduling issues that may occur – In-memory metric buffers – Metric tracking with flow back-pressure in plugins like Kafka – Full-streaming support – Metric routing: name & field pass & drop – Flexible parsing, formatting, serializing

15. © 2021 InﬂuxData. All rights reserved. 15 15 Telegraf Plugin Ecosystem Input Plugins ● collect metrics from system, services, or third party APIs Output Plugins ● write data to various destinations Processors ● transform, decorate, and/or ﬁlter metrics Aggregators ● create aggregate metrics (e.g. mean, min, max, quantiles, etc.)

17. © 2021 InfluxData. All rights reserved. 17 17 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_que ue bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb

18. © 2021 InfluxData. All rights reserved. 18 18 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetr y kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap

19. © 2021 InfluxData. All rights reserved. 19 19 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensibl e powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper

20. © 2021 InﬂuxData. All rights reserved. 20 20 Processor Plugins clone converter date dedup defaults enum execd ﬁlepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot

22. © 2021 InfluxData. All rights reserved. 22 22 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monito ring

23. © 2021 InfluxData. All rights reserved. 23 23 Telegraf CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL Process - transform - decorate - filter Aggregate - mean - min,max - count - variance - stddev InfluxDB File Kafka CloudWatch CloudWatch Input Output One Telegraf, Multiple Plugins

24. © 2021 InﬂuxData. All rights reserved. 24 24 Can’t ﬁnd the plugin you need? Telegraf is 100% open source with a strong community of contributors It’s easy to write your own Telegraf plugin! 1) Follow the contribution guide for Go 2) Write your plugin in any language and run it externally with ExecD

25. © 2021 InﬂuxData. All rights reserved. 25 25 Popular Plugins Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (consumer) Prometheus (/metrics) SNMP Exec/Execd AMQP (mq metadata) StatsD Redis Nginx HAproxy Jolokia2

31. © 2021 InfluxData. All rights reserved. 31 31 The Telegraf config file needs to be specified for Telegraf agent to operate properly. It contains setup for the agent, global tags, and enabled outputs (through commenting out or removing unnecessary lines) What’s in a configuration file?

32. © 2021 InﬂuxData. All rights reserved. 32 32 Set and Test Your Conﬁguration $ telegraf --input-filter cpu:system --output-filter influxdb --processor-filter starlark --aggregator-filter valuecounter config > telegraf.conf WRITE DESIRED CONFIG FILE RUN CONFIG FILE TEST $ telegraf --config telegraf.conf --test VERIFY YOUR CONFIG WORKS

34. © 2021 InﬂuxData. All rights reserved. 34 34 ● Work with the open-source community or submit upgrades or enhancements for existing plugins ● Use ExecD to write a plugin in Go or a language of your choice You can extend Telegraf by:

35. © 2021 InﬂuxData. All rights reserved. 35 35 ● Plugin runs in its own process ● Requires line protocol ● Avoid the need for review by Telegraf team ● Supports the same API as an internal plugin ● Can use for non-GO plugins ● Can use for licensed software plugins ● Can use for any type of plugin (input, output, processor, aggregator) External plugins via ExecD

38. © 2021 InﬂuxData. All rights reserved. 38 38 Customer Quotes “Telegraf is like a swiss army knife for connecting various MQTT sources and OPC UA sources.” — Fr. Ant. Niedermayr “Our next-generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement the four R’s: routability, retention, resilience, and redundancy.” — Wayfair

40. © 2021 InfluxData. All rights reserved. 40 Get involved with Telegraf • Telegraf GitHub: github.com/influxdata/telegraf • Community Slack: influxdata.com/slack #telegraf #telegraf-dev • Community Website: community.influxdata.com

42. Thank You

43. © 2021 InfluxData. All rights reserved. 43 43 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_queu e bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb

44. © 2021 InfluxData. All rights reserved. 44 44 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetry kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap

45. © 2021 InfluxData. All rights reserved. 45 45 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensibl e powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper

46. © 2021 InﬂuxData. All rights reserved. 46 46 Processor Plugins clone converter date dedup defaults enum execd ﬁlepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot

48. © 2021 InfluxData. All rights reserved. 48 48 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monitor ing

Getting Started: Intro to Telegraf - July 2021

You are reading a preview.

Check these out next

Getting Started: Intro to Telegraf - July 2021

More Related Content

Slideshows for you (20)

Similar to Getting Started: Intro to Telegraf - July 2021 (20)

More from InfluxData (20)

Recently uploaded (20)