How to Use Telegraf and Its Plugin Ecosystem

1. Jessica Ingrassellino Engineering Manager, InfluxData Samantha Wang Product Manager, InfluxData December 2020 How to Use Telegraf and Its Plugin Ecosystem

7. © 2020 InfluxData. All rights reserved.7 © 2020 InfluxData. All rights reserved.7 Current Data Acquisition - All Methods Agent-based Push (aka Telegraf) Agentless Pull (aka Scrapers) Client Libraries Native/Ecosystem ● 200+ Telegraf plugins ● Regular cadence of releases ● Why use it? ○ No code ○ Large community ○ Lightweight but powerful ○ Customizable ● Prometheus scraper (OSS only) ● Flux prometheus.from ● Flux csv.from(url) ● Why use them? ○ Get data in quickly ○ Doesn’t require agent downloads on monitoring device ● 7 Libraries: Python, C#, Java, GO, Javascript/Node.js, Ruby, PHP ● Handles batching, chunking, setting right headers, etc. ● Why use them? ○ Easy way to get started ○ Need libraries when building custom applications ● Source system speaks line protocol ● Examples: JMeter, NiFi, Vector, FluentD ● Influx CLI CSV Import ● Why use them? ○ Know what you want to monitor, quick and easy integration

8. © 2020 InfluxData. All rights reserved.8 © 2020 InfluxData. All rights reserved.8 Telegraf: Agent for Collecting Metrics & Events Plugin-driven server agent for collecting and reporting metrics • Written in Go • Single Binary, No external dependencies • Minimal memory footprint • Optimized for writing to InfluxDB • Optimized for streaming data Input plugins to source a variety of metrics • directly from the system it’s running on • pull metrics from third party APIs • listen for metrics via a statsd and Kafka consumer services

9. © 2020 InfluxData. All rights reserved.9 © 2020 InfluxData. All rights reserved.9 Telegraf: Agent for Collecting Metrics & Events Output plugins to send metrics to a variety of other datastores, services, and message queues • InfluxDB, Graphite, OpenTSDB, Datadog, and many others • Writes to multiple outputs at the same time Metrics routing and conversion Aggregation stage for InfluxDB

10. © 2020 InfluxData. All rights reserved.10 © 2020 InfluxData. All rights reserved.10 Telegraf Agent – Core of Telegraf functionality providing the following – Robust scheduler – Adjustments for clock-drift – Adjustments for job scheduling issues that may occur – In-memory metric buffers – Metric tracking with flow back-pressure in plugins like Kafka – Full-streaming support – Metric routing: name & field pass & drop – Flexible parsing, formatting, serializing

12. © 2020 InfluxData. All rights reserved.12 © 2020 InfluxData. All rights reserved.12 Telegraf Plugin Ecosystem Input Plugins ● collect metrics from system, services, or third party APIs Output Plugins ● write data to various destinations Processors ● transform, decorate, and/or filter metrics Aggregators ● create aggregate metrics (e.g. mean, min, max, quantiles, etc.)

14. © 2020 InfluxData. All rights reserved.14 © 2020 InfluxData. All rights reserved.14 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_que ue bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb

15. © 2020 InfluxData. All rights reserved.15 © 2020 InfluxData. All rights reserved.15 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetr y kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap

16. © 2020 InfluxData. All rights reserved.16 © 2020 InfluxData. All rights reserved.16 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensib le powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper

17. © 2020 InfluxData. All rights reserved.17 © 2020 InfluxData. All rights reserved.17 Processor Plugins clone converter date dedup defaults enum execd filepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot

19. © 2020 InfluxData. All rights reserved.19 © 2020 InfluxData. All rights reserved.19 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monito ring

20. © 2020 InfluxData. All rights reserved.20 © 2020 InfluxData. All rights reserved.20 Telegraf CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL Process - transform - decorate - filter Aggregate - mean - min,max - count - variance - stddev InfluxDB File Kafka CloudWatch CloudWatch Input Output One Telegraf, Multiple Plugins

21. © 2020 InfluxData. All rights reserved.21 © 2020 InfluxData. All rights reserved.21 Can’t find the plugin you need? Telegraf is 100% open source with a strong community of contributors It’s easy to write your own Telegraf plugin! 1) Follow the contribution guide for Go 2) Write your plugin in any language and run it externally with ExecD

22. © 2020 InfluxData. All rights reserved.22 © 2020 InfluxData. All rights reserved.22 Popular Plugins Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (consumer) Prometheus (/metrics) SNMP Exec/Execd AMQP (mq metadata) StatsD Redis Nginx HAproxy Jolokia2

25. © 2020 InfluxData. All rights reserved.25 © 2020 InfluxData. All rights reserved.25 Telegraf Configuration Snippets Include many plugins per Telegraf - multiple inputs, outputs, processors, aggregators

26. © 2020 InfluxData. All rights reserved.26 © 2020 InfluxData. All rights reserved.26 The Telegraf config file needs to be specified for Telegraf agent to operate properly. It contains setup for the agent, global tags, and enabled outputs (through commenting out or removing unnecessary lines) What’s in a configuration file?

27. © 2020 InfluxData. All rights reserved.27 © 2020 InfluxData. All rights reserved.27 Set and Test Your Configuration $ telegraf --input-filter cpu:system --output-filter influxdb --processor-filter starlark --aggregator-filter valuecounter config > telegraf.conf WRITE DESIRED CONFIG FILE RUN CONFIG FILE TEST $ telegraf --config telegraf.conf --test VERIFY YOUR CONFIG WORKS

29. © 2020 InfluxData. All rights reserved.29 © 2020 InfluxData. All rights reserved.29 ● Work with the open-source community or submit upgrades or enhancements for existing plugins ● Use ExecD to write a plugin in Go or a language of your choice You can extend Telegraf by:

30. © 2020 InfluxData. All rights reserved.30 © 2020 InfluxData. All rights reserved.30 ● Plugin runs in its own process ● Requires line protocol ● Avoid the need for review by Telegraf team ● Supports the same API as an internal plugin ● Can use for non-GO plugins ● Can use for licensed software plugins ● Can use for any type of plugin (input, output, processor, aggregator) External plugins via ExecD

32. © 2020 InfluxData. All rights reserved.32 © 2020 InfluxData. All rights reserved.32 [[processors.execd]] command = ["python", "./processors/forecasting.py"] Sample ExecD Config: Input, Output, Processor

33. © 2020 InfluxData. All rights reserved.33 © 2020 InfluxData. All rights reserved.33 ExecD: To Shim, or not to Shim shim Use this when building an external plugin using Go no shim Use this when building a plugin in other languages (Python, Ruby, etc)

34. © 2020 InfluxData. All rights reserved.34 © 2020 InfluxData. All rights reserved.34 Customer Quotes “Telegraf is like a swiss army knife for connecting various MQTT sources and OPC UA sources.” — Fr. Ant. Niedermayr “Our next-generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement the four R’s: routability, retention, resilience, and redundancy.” — Wayfair

36. Thank You

37. © 2020 InfluxData. All rights reserved.37 © 2020 InfluxData. All rights reserved.37 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_queu e bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb

38. © 2020 InfluxData. All rights reserved.38 © 2020 InfluxData. All rights reserved.38 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetry kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap

39. © 2020 InfluxData. All rights reserved.39 © 2020 InfluxData. All rights reserved.39 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensibl e powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper

40. © 2020 InfluxData. All rights reserved.40 © 2020 InfluxData. All rights reserved.40 Processor Plugins clone converter date dedup defaults enum execd filepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot

42. © 2020 InfluxData. All rights reserved.42 © 2020 InfluxData. All rights reserved.42 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monitor ing

How to Use Telegraf and Its Plugin Ecosystem

You are reading a preview.

How to Use Telegraf and Its Plugin Ecosystem

More Related Content

Slideshows for you

Similar to How to Use Telegraf and Its Plugin Ecosystem

More from InfluxData

Featured

Related Books

Related Audiobooks