Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Build a Telegraf Plugin by Noah Crowley

1,507 views

Published on

Telegraf is a plugin-driven server agent for collecting & reporting metrics and there are many plugins already written to source data from a variety of services and systems. However, there may be instances where you need to write your own plugin to source data from your particular systems. In this InfluxDays NYC 2019 session, Noah Crowley will provide you with the steps on how to write your own Telegraf plugin. Writing your own Telegraf plugin will require an understanding of the Go programming language.

Published in: Technology
  • Login to see the comments

How to Build a Telegraf Plugin by Noah Crowley

  1. 1. Noah Crowley / Developer Advocate How to Build a Telegraf Plugin
  2. 2. © 2018 InfluxData. All rights reserved. What we’ll be covering ✔ Telegraf overview ✔ Examples of Telegraf plugins ✔ Telegraf plugin architecture ✔ How to a write a Telegraf plugin
  3. 3. Telegraf Overview
  4. 4. © 2018 InfluxData. All rights reserved.
  5. 5. © 2018 InfluxData. All rights reserved. Telegraf Overview • Telegraf is an open source, plugin-driven agent for collecting and reporting metrics • “Push” or “Pull” • Large collection of plugins: 192 (as of 3/13/2019) • Inputs: 149 • Outputs: 30 • Aggregators: 9 • Processors: 4
  6. 6. © 2018 InfluxData. All rights reserved. Telegraf Overview • Can act as: • Agent • Collector • Part of an Ingest Pipeline
  7. 7. © 2018 InfluxData. All rights reserved. Telegraf Plugins • Outputs • Allows you to send metrics to a variety of datastores • Plugins for both InfluxDB 1.x and 2.0 • Supports Kafka, MQTT, NSQ, OpenMetrics, and more • Aggregators and Processors allow you to manipulate data as it flows through Telegraf • Transform tags, convert types, calculate histograms
  8. 8. © 2018 InfluxData. All rights reserved. Telegraf Input Plugins • Metrics ingestion from host system: CPU, I/O, Network… • Common Applications like NGINX, Postgres, Redis… • Third Party APIs: Mailchimp, Cloudwatch, Google Analytics… • General-Purpose Protocols: HTTP, Socket, MQTT… • …and more!
  9. 9. © 2018 InfluxData. All rights reserved. Telegraf Benefits • Minimal memory footprint • Tagging of metrics • Batching of metrics to reduce the number of atomic writes • Easy contribution of functionality thanks to Go • Static Binary • Strong Community Contributions
  10. 10. Examples of Plugins
  11. 11. © 2018 InfluxData. All rights reserved. statsd • Listens for incoming data • Acts as a statsd instance • Outputs to any configured output
  12. 12. © 2018 InfluxData. All rights reserved. postgresql • Collects performance data • Uses data from built-in views: • pg_stat_database • pg_stat_bgwriter
  13. 13. © 2018 InfluxData. All rights reserved. apache • Connects over HTTP • Reads data from: • /server-status?auto
  14. 14. © 2018 InfluxData. All rights reserved. win_perf_counters • Collects performance data from Windows machines
  15. 15. © 2018 InfluxData. All rights reserved. Input Plugin Considerations • Where does the data exist? • How can it be collected? • What shape is the data?
  16. 16. © 2018 InfluxData. All rights reserved. prometheus_client • Output plugin • Exposes metrics in Prometheus (now OpenMetrics) format.
  17. 17. © 2018 InfluxData. All rights reserved. mqtt • Output half of the MQTT plugins • Allows you to write messages to an MQTT broker.
  18. 18. © 2018 InfluxData. All rights reserved. Output Plugin Considerations • How should the data be shaped? • How is the data output? • Where is the data output to?
  19. 19. Plugin Architecture
  20. 20. © 2018 InfluxData. All rights reserved. Plugin Architecture • Built to be extensible from the beginning • Make use of interfaces in Go • Each plugin type has an interface
  21. 21. © 2018 InfluxData. All rights reserved.
  22. 22. © 2018 InfluxData. All rights reserved. package simple // simple.go import ( "github.com/influxdata/telegraf" "github.com/influxdata/telegraf/plugins/inputs" ) type Simple struct { Ok bool } func (s *Simple) Description() string { return "a demo plugin" } func (s *Simple) SampleConfig() string { return ` ## Indicate if everything is fine ok = true ` } func (s *Simple) Gather(acc telegraf.Accumulator) error { if s.Ok { acc.AddFields("state", map[string]interface{}{"value": "pretty good"}, nil) } else { acc.AddFields("state", map[string]interface{}{"value": "not great"}, nil) } return nil } func init() { inputs.Add("simple", func() telegraf.Input { return &Simple{} }) }
  23. 23. Building a Plugin
  24. 24. © 2018 InfluxData. All rights reserved. Getting started… • Recommend using Go 1.11 • Understanding of Git & Pull Requests • What is our plugin going to do? • Generate data using “sine” and “cosine” trigonometric functions • Read the CONTRIBUTING.md file to ensure you can build & test Telegraf before continuing
  25. 25. © 2018 InfluxData. All rights reserved. Getting started… • Set up your workspace, get the code, and create a branch: • Create the required directories and files: • Add boilerplate… $ go get github.com/influxdata/telegraf
 $ cd $GOPATH/github.com/influxdata/telegraf
 $ git checkout -b trig-demo $ cd plugins/inputs $ mkdir trig $ touch trig/trig.go
  26. 26. © 2018 InfluxData. All rights reserved. package trig
 
 import (
     "github.com/influxdata/telegraf"
     "github.com/influxdata/telegraf/plugins/inputs"
 )
 
 type Trig struct {
 }
 
 func (s *Trig) SampleConfig() string {
     return ""
 }
 
 func (s *Trig) Description() string {
     return ""
 }
 
 func (s *Trig) Gather(acc telegraf.Accumulator) error {
     return nil
 }
 
 func init() {
     inputs.Add("trig", func() telegraf.Input { return &Trig{} })
 } trig.go
  27. 27. © 2018 InfluxData. All rights reserved. Import your plugin • Add the following to telegraf/plugins/inputs/all/all.go • This will your plugin into the main Telegraf package and ensure that it can run. _ "github.com/influxdata/telegraf/plugins/inputs/trig"
  28. 28. © 2018 InfluxData. All rights reserved. Add configuration variables • Each plugin has a struct • There must be a property on the struct to hold any configuration variables • Any other variables should be unexported • We’ll add two variables: • “x” will hold the state of the plugin between collection intervals • “Amplitude” will hold the amplitude value for the sin wave
  29. 29. © 2018 InfluxData. All rights reserved. package trig
 
 import (
     "github.com/influxdata/telegraf"
     "github.com/influxdata/telegraf/plugins/inputs"
 )
 
 type Trig struct {     x         float64     Amplitude float64
 }
 
 func (s *Trig) SampleConfig() string {
     return ""
 }
 
 func (s *Trig) Description() string {
     return ""
 }
 
 func (s *Trig) Gather(acc telegraf.Accumulator) error {
 trig.go
  30. 30. © 2018 InfluxData. All rights reserved. Add sample configuration • Telegraf can dynamically construct configuration files • Aggregates sample configs for all plugins • The CLI allows you to provide arguments to filter which plugins are included • In 2.0, InfluxDB will use this info to generate Telegraf configs
  31. 31. © 2018 InfluxData. All rights reserved. package trig
 
 import (
     "github.com/influxdata/telegraf"
     "github.com/influxdata/telegraf/plugins/inputs"
 )
 
 type Trig struct {     x         float64     Amplitude float64
 }
 
 var TrigConfig = `
  ## Set the amplitude
  amplitude = 10.0
 `
 
 func (s *Trig) SampleConfig() string {
     return TrigConfig
 } 
 trig.go
  32. 32. © 2018 InfluxData. All rights reserved. Add a simple description • This description is included above the plugin configuration and should briefly describe what your plugin does
  33. 33. © 2018 InfluxData. All rights reserved. type Trig struct {     x         float64     Amplitude float64
 }
 
 var TrigConfig = `
  ## Set the amplitude
  amplitude = 10.0
 `
 
 func (s *Trig) SampleConfig() string {
     return TrigConfig
 } 
 func (s *Trig) Description() string {
     return "Inserts sine and cosine waves for demonstration purposes"
 }
 
 func (s *Trig) Gather(acc telegraf.Accumulator) error {
 trig.go
  34. 34. © 2018 InfluxData. All rights reserved. Test your config! • First, build Telegraf • Generate a config: $ make telegraf $ telegraf -sample-config -input-filter trig -output-filter influxdb -debug
  35. 35. © 2018 InfluxData. All rights reserved. ...
 ############################################################################### # INPUT PLUGINS # ############################################################################### # Inserts sine and cosine waves for demonstration purposes [[inputs.trig]] ## Set the amplitude amplitude = 10.0
  36. 36. © 2018 InfluxData. All rights reserved. The “Gather” function • The heart of the plugin, it is run every time telegraf collects metrics • This is where you’ll do the majority of the work • We’ll generate sine and cosine values • At the end of the function, we add any data we’ve collected and to the Telegraf “Accumulator” by calling .AddFields(measurement, tags, fields). • This defines a new point in InfluxDB with the timestamp being the collection time.
  37. 37. © 2018 InfluxData. All rights reserved.     return "Inserts sine and cosine waves for demonstration purposes"
 }
 
 func (s *Trig) Gather(acc telegraf.Accumulator) error {
     sinner := math.Sin((s.x*math.Pi)/5.0) * s.Amplitude
     cosinner := math.Cos((s.x*math.Pi)/5.0) * s.Amplitude
 
     fields := make(map[string]interface{})
     fields["sine"] = sinner
     fields["cosine"] = cosinner
 
     tags := make(map[string]string)
 
     s.x += 1.0
     acc.AddFields("trig", fields, tags)
 
     return nil
 } trig.go
  38. 38. © 2018 InfluxData. All rights reserved. Add initial state • In our boilerplate, we added an init() function • Called when the plugin is first initialized • Makes the plugin available to the Telegraf agent • We can define starting state as part of this function • We need to initialize our “x” variable
  39. 39. © 2018 InfluxData. All rights reserved.     cosinner := math.Cos((s.x*math.Pi)/5.0) * s.Amplitude
 
     fields := make(map[string]interface{})
     fields["sine"] = sinner
     fields["cosine"] = cosinner
 
     tags := make(map[string]string)
 
     s.x += 1.0
     acc.AddFields("trig", fields, tags)
 
     return nil
 } 
 
 func init() {     inputs.Add("trig", func() telegraf.Input { return &Trig{x: 0.0} }) } trig.go
  40. 40. © 2018 InfluxData. All rights reserved. Compile and test • Once again, build Telegraf: • Generate a config and write it to a file: $ make telegraf $ ./telegraf -sample-config -input-filter trig -output-filter influxdb -debug >> telegraf.conf.test • Run Telegraf with the new config: $ ./telegraf -config telegraf.conf.test -debug
  41. 41. © 2018 InfluxData. All rights reserved. $ ./telegraf -config telegraf.conf.test -debug 2019-03-14T06:32:12Z I! Starting Telegraf 2019-03-14T06:32:12Z I! Loaded inputs: trig 2019-03-14T06:32:12Z I! Loaded aggregators: 2019-03-14T06:32:12Z I! Loaded processors: 2019-03-14T06:32:12Z I! Loaded outputs: influxdb 2019-03-14T06:32:12Z I! Tags enabled: host=noah-mbp.local 2019-03-14T06:32:12Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"noah-mbp.local", Flush Interval:10s 2019-03-14T06:32:12Z D! [agent] Connecting outputs 2019-03-14T06:32:12Z D! [agent] Attempting connection to output: influxdb 2019-03-14T06:32:12Z D! [agent] Successfully connected to output: influxdb 2019-03-14T06:32:12Z D! [agent] Starting service inputs 2019-03-14T06:32:30Z D! [outputs.influxdb] wrote batch of 1 metrics in 23.857525ms 2019-03-14T06:32:30Z D! [outputs.influxdb] buffer fullness: 1 / 10000 metrics.
  42. 42. © 2018 InfluxData. All rights reserved. Final steps before submitting • Add a few things to your plugin… • a README.md • a LICENSE • A sample of the input/output format • Tests!
  43. 43. © 2018 InfluxData. All rights reserved. package trig
 
 import (
     "math"
     "testing"
     "github.com/influxdata/telegraf/testutil"
 )
 
 func TestTrig(t *testing.T) {
 
     s := &Trig{
 
         Amplitude: 10.0,
     }
 
     for i := 0.0; i < 10.0; i++ {
         var acc testutil.Accumulator
         sine := math.Sin((i*math.Pi)/5.0) * s.Amplitude
         cosine := math.Cos((i*math.Pi)/5.0) * s.Amplitude
         s.Gather(&acc)
         fields := make(map[string]interface{})
         fields["sine"] = sine
         fields["cosine"] = cosine
 
         acc.AssertContainsFields(t, "trig", fields)
 
     }
 
 } trig_test.go
  44. 44. © 2018 InfluxData. All rights reserved. Final steps before submitting • Add a few things to your plugin… • a README.md • a LICENSE • A sample of the input/output format • Tests! • …and submit! • github.com/influxdata/telegraf/pulls
  45. 45. Thank You!

×