Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Write your own telegraf plugin


Published on

Telegraf is a plugin-driven server agent for collecting & reporting metrics and there are many plugins already written to source data from a variety of services and systems. However, there may be instances where you need to write your own plugin to source data from your particular systems. In this session, Noah will provide you with the steps on how to write your own Telelgraf plugin. This will require an understanding of the Go programming language.

Published in: Internet

Write your own telegraf plugin

  1. 1. Noah Crowley, Developer Advocate Getting Started Series Write your Own Telegraf plugin
  2. 2. © 2017 InfluxData. All rights reserved.2 © 2017 InfluxData. All rights reserved.2 ✓ Telegraf overview ✓ Examples of Telegraf plugins ✓ Telegraf plugin architecture ✓ How to a write a Telegraf plugin (demo) What we will be covering
  3. 3. © 2017 InfluxData. All rights reserved.3 © 2017 InfluxData. All rights reserved.3 Telegraf Overview
  4. 4. © 2018 InfluxData. All rights reserved.4 InfluxData Open Source Projects: TELEGRAF!
  5. 5. © 2018 InfluxData. All rights reserved.5 InfluxData Open Source Projects: TELEGRAF!
  6. 6. © 2018 InfluxData. All rights reserved.6 Telegraf Telegraf is an Open Source plugin-driven server agent for collecting & reporting metrics: “Push” or “Pull”. • Used to batch metrics from multiple sources together in order to reduce the number of atomic write requests • Large collection of Plugins: – Inputs – Outputs – Aggregators – Processors • Can act as an: – Agent – Collector – Ingest Pipeline
  7. 7. © 2018 InfluxData. All rights reserved.7 Telegraf continued • Metrics ingestion from the host system (cpu, I/O, network) – Common programs (Hadoop, Postgres, Redis, etc.) – Third party APIs (Mailchimp, CloudWatch, Google Analytics, etc.) • Output plugins to send metrics to a variety of other datastores, services, and message queues, including – InfluxDB, Graphite, OpenTSDB, Datadog, Librato, Kafka, MQTT, NSQ, Prometheus, and many others • Over 140 plugins!
  8. 8. © 2018 InfluxData. All rights reserved.8 Telegraf Benefits • Although if we start using it then it’s kind of the de-facto tool anyway… • Minimal memory footprint • Tagging of metrics • Easy contribution of functionality • Strong Community contributions
  9. 9. © 2017 InfluxData. All rights reserved.9 © 2017 InfluxData. All rights reserved.9 Examples of Telegraf plugins
  10. 10. © 2017 InfluxData. All rights reserved.10 Some Telegraf plugins statsd Listens for data in statsd format and outputs it to any configured output apache Collects data from the /server-status?auto endpoint and stores it postgresql Collects performance metrics about postgres; uses data from the built in pg_stat_database and pg_stat_bgwriter views win_perf_counters Collects performance data from Windows machines
  11. 11. © 2017 InfluxData. All rights reserved.11 Some Telegraf plugins prometheus_client Exposes all metrics on /metrics to be polled by a Prometheus server nginx Read Nginx's basic status (ngx_http_stub_status_module) histogram Creates histograms containing the counts of field values within a range kubernetes Experimental; Talks to the kubelet api using the /stats/summary endpoint
  12. 12. © 2018 InfluxData. All rights reserved.12 How do I install it? Download binaries at then: Telegraf normally runs as a Linux service $ telegraf -sample-config -input-filter haproxy -output-filter influxdb > telegraf.conf $ telegraf -config telegraf.conf $ telegraf -sample-config -input-filter haproxy -output-filter influxdb > /etc/telegraf/telegraf.conf $ service telegraf start
  13. 13. © 2017 InfluxData. All rights reserved.13 © 2017 InfluxData. All rights reserved.13 Telegraf plugin architecture
  14. 14. © 2018 InfluxData. All rights reserved.14 The Open Source Collector Agent: Telegraf
  15. 15. © 2018 InfluxData. All rights reserved.15 Input interface The plugin must conform to the telegraf.Input interface: type Input interface { // SampleConfig returns the default configuration of the Input SampleConfig() string // Description returns a one-sentence description on the Input Description() string // Gather takes in an accumulator and adds the metrics that the Input // gathers. This is called every "interval" Gather(Accumulator) error }
  16. 16. © 2017 InfluxData. All rights reserved.16 © 2017 InfluxData. All rights reserved.16 Let’s write a plugin
  17. 17. © 2017 InfluxData. All rights reserved.17 What do you need? • Go 1.8+ installed and configured on your computer (1.9+ recommended) • A desire to write a Telegraf plugin • What is our plugin going to do? • It will emit a trigonometric pattern called `trig`
  18. 18. © 2018 InfluxData. All rights reserved.18 Getting Started This will create a new Telegraf binary at $GOPATH/bin/telegraf $ go get $ cd $GOPATH/ $ git checkout -b mySweetPlugin $ make
  19. 19. © 2018 InfluxData. All rights reserved.19 Make your files and add boilerplate $ cd plugins/inputs $ mkdir trig $ touch trig/trig.go # Add boilerplate $ cat trig/trig.go
  20. 20. © 2018 InfluxData. All rights reserved.20 Boilerplate $ cat trig/trig.go package trig import ( "" "" ) type Trig struct { } func (s *Trig) SampleConfig() string { return "" } func (s *Trig) Description() string { return "" } func (s *Trig) Gather(acc telegraf.Accumulator) error { return nil } func init() { inputs.Add("trig", func() telegraf.Input { return &Trig{} }) }
  21. 21. © 2018 InfluxData. All rights reserved.21 Import your plugin In the file -> telegraf/plugins/inputs/all/all.go Add: This imports your plugin to the main telegraf package and ensures that it can run _ ""
  22. 22. © 2018 InfluxData. All rights reserved.22 Add properties to your struct There must be a property on the struct to hold any configuration variables Any other variables should be unexported. x will hold state of the plugin between collection intervals, being updated each one. type Trig struct { x float64 Amplitude float64 }
  23. 23. © 2018 InfluxData. All rights reserved.23 Add any configuration variables you need Telegraf dynamically constructs configuration files by aggregating the configurations for all of it’s plugins. This allows you to easily generate configuration files with only certain plugins enabled by just passing a couple of CLI flags. var TrigConfig = ` ## Set the amplitude amplitude = 10.0 ` func (s *Trig) SampleConfig() string { return TrigConfig }
  24. 24. © 2018 InfluxData. All rights reserved.24 Add a simple description of your plugin This description is included above the plugin configuration and should briefly describe what your plugin does func (s *Trig) Description() string { return "Inserts sine and cosine waves for demonstration purposes" }
  25. 25. © 2018 InfluxData. All rights reserved.25 Test it! $ make $ telegraf -sample-config -input-filter trig -output-filter influxdb -debug … # # INPUTS: # # Inserts sine and cosine waves for demonstration purposes [[inputs.trig]] ## Set the amplitude amplitude = 10.0
  26. 26. © 2017 InfluxData. All rights reserved.26 The Gather (telegraf.Accumulator) Function • The Gather function is the heart of the plugin and is run every time telegraf collects metrics. • This number is configurable via the [agent]interval=10s configuration option in the top level config in telegraf. • Plugins can also be written to collect at different intervals. • The important function is the telegraf.Accumulator.AddFields(measurement, tags, fields). This defines a new point in influxdb with the timestamp being the collection time.
  27. 27. © 2018 InfluxData. All rights reserved.27 Write your Gather function! func (s *Trig) Gather(acc telegraf.Accumulator) error { sinner := math.Sin((s.x*math.Pi)/5.0) * s.Amplitude cosinner := math.Cos((s.x*math.Pi)/5.0) * s.Amplitude fields := make(map[string]interface{}) fields["sine"] = sinner fields["cosine"] = cosinner tags := make(map[string]string) s.x += 1.0 acc.AddFields("trig", fields, tags) return nil }
  28. 28. © 2018 InfluxData. All rights reserved.28 Add starting state Starting state can be defined in the struct you pass to the input.Add() function. For our example we need to initialize x This function packages up the whole plugin (all the methods are defined on the Trig{} struct) and gives it to the Telegraf agent. The agent iterates through all of the plugins that are enabled every collection and calls their gather func init() { inputs.Add("trig", func() telegraf.Input { return &Trig{x: 0.0} }) }
  29. 29. © 2018 InfluxData. All rights reserved.29 Compile and test your plugin $ make $ telegraf -sample-config -input-filter trig -output-filter influxdb >> telegraf.conf.test $ telegraf -config telegraf.conf.test -debug 2016/03/24 11:50:26 Starting Telegraf (version 1.3.0-223-gf543dbb) 2016/03/24 11:50:26 Loaded outputs: influxdb 2016/03/24 11:50:26 Loaded inputs: trig 2016/03/24 11:50:26 Tags enabled: host=XXXXXX.local 2016/03/24 11:50:26 Agent Config: Interval:10s, Debug:false, Quiet:false, Hostname:"XXXXXX.local", Flush Interval:10s 2016/03/24 11:50:30 Gathered metrics, (10s interval), from 1 inputs in 237.28µs
  30. 30. © 2018 InfluxData. All rights reserved.30 Graph it with Chronograf!
  31. 31. © 2018 InfluxData. All rights reserved.31 Write some tests... package trig import ( "math" "testing" "" ) func TestTrig(t *testing.T) { s := &Trig{ Amplitude: 10.0, } for i := 0.0; i < 10.0; i++ { var acc testutil.Accumulator sine := math.Sin((i*math.Pi)/5.0) * s.Amplitude cosine := math.Cos((i*math.Pi)/5.0) * s.Amplitude s.Gather(&acc) fields := make(map[string]interface{}) fields["sine"] = sine fields["cosine"] = cosine acc.AssertContainsFields(t, "trig", fields) } }
  32. 32. © 2018 InfluxData. All rights reserved.32 How to submit a plugin Must include: • a file, • a LICENSE file, • and a sample of the output/input format
  33. 33. @noahcrowley