Successfully reported this slideshow.
Your SlideShare is downloading. ×

Taming the Tiger: Tips and Tricks for Using Telegraf

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 110 Ad

Taming the Tiger: Tips and Tricks for Using Telegraf

Download to read offline

On 17th May as part of InfluxDays EMEA 2021 Virtual Experience, the Technical Services team will be offering a free live InfluxDB training to the first 300 registered attendees. This will be hosted over Zoom with two main trainers and there will be assistants to help participants with the course work. The training will be recorded and made available on the InfluxDays website and the InfluxData YouTube channel.

On 17th May as part of InfluxDays EMEA 2021 Virtual Experience, the Technical Services team will be offering a free live InfluxDB training to the first 300 registered attendees. This will be hosted over Zoom with two main trainers and there will be assistants to help participants with the course work. The training will be recorded and made available on the InfluxDays website and the InfluxData YouTube channel.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Taming the Tiger: Tips and Tricks for Using Telegraf (20)

Advertisement

More from InfluxData (20)

Recently uploaded (20)

Advertisement

Taming the Tiger: Tips and Tricks for Using Telegraf

  1. 1. © 2021 InfluxData. All rights reserved. 1 © 2020 InfluxData. All rights reserved. 1 Ignacio Van Droogenbroeck | Solutions Architect, InfluxData Suleman Khan | Solutions Architect, InfluxData May 2021
  2. 2. Agenda ■ Overview ■ Setup (if not completed) ■ Global configuration ■ Enabling plugins ■ Filtering data
  3. 3. Please join us in the InfluxDB Community Slack at www.influxdata.com/slack. To participate in the conversations, join the #influxdays-virtual & #telegraf-training channels. JOIN THE CONVERSATION
  4. 4. Workshop Logistics • Cloud account free tier • No CC • Rate limits → Let’s avoid them • Browser • SSH client • Have documentation open for exercise • General config: https:/ /bit.ly/3t1r03K
  5. 5. © 2021 InfluxData. All rights reserved. 5 Telegraf • Lightweight; written in Go • Plug-in driven • Optimized for writing to InfluxDB • Formatting • Retries • Modifiable batch sizes and jitter • Tag sorting • Preprocessing • Converting tags to fields, fields to tags • Regex transformations • Renaming measurements, tags • Aggregations (mean, min, max, count, variance, stddev, etc.)
  6. 6. © 2021 InfluxData. All rights reserved. 6 Popular Inputs (Integrations) Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (producer/consumer) Prometheus (/metrics) SNMP Exec System (Linux,Windows,Unix) StatsD Nginx HAproxy OPC, AMQP, Modbus & MQTT (IoT)
  7. 7. © 2021 InfluxData. All rights reserved. 7 Telegraf CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL Process -transform -decorate -filter Aggregate -mean -min,max -count -variance -stddev File InfluxDB Kafka CloudWatch CloudWatch
  8. 8. © 2021 InfluxData. All rights reserved. 8
  9. 9. © 2021 InfluxData. All rights reserved. 9 Telegraf Common Architecture Messaging Queue Kafka Rabbit Active NSQ AWS Kinesis Google PubSub MQTT
  10. 10. © 2021 InfluxData. All rights reserved. 10 Telegraf Architecture Message Queue Kafka Rabbit Active NSQ AWS Kinesis Google PubSub MQTT
  11. 11. © 2021 InfluxData. All rights reserved. 11 Parsing ● JSON ● CSV ● Graphite ● CollectD ● Dropwizard ● Form URL-encoded ● Grok
  12. 12. © 2021 InfluxData. All rights reserved. 12 Telegraf Latest version available at https:/ /portal.influxdata.com/downloads/#telegraf Create the configuration file $ telegraf config > telegraf.conf # ~7500 lines OR $ telegraf -sample-config -input-filter cpu:mem -output-filter influxdb > telegraf.conf # 830 lines
  13. 13. © 2021 InfluxData. All rights reserved. 13 Starlark Embedded Python dialect – Not full std lib – But most basic Python functionality supported Current processors and aggregators + more [[processors.starlark]] namepass = ["mymeasurement"] source = ''' def apply(metric): I = metric.fields['current'] V = metric.fields['voltage'] metric.fields['power'] = I * V return metric ''' [[processors.starlark]] source = ''' def apply(metric): used = float(metric.fields['used']) total = float(metric.fields['total']) metric.fields['usage'] = (used / total) * 100 return metric '''
  14. 14. Setting up InfluxDB Cloud Sign up: https:/ /cloud2.influxdata.com/signup
  15. 15. © 2021 InfluxData. All rights reserved. 15 InfluxDB Cloud Setup - Choose your Provider & Region - Enter your company name - Read & Tick the Service Agreement
  16. 16. © 2021 InfluxData. All rights reserved. 16 InfluxDB Cloud Setup ● Select “Free” account
  17. 17. © 2021 InfluxData. All rights reserved. 17 InfluxDB Cloud Setup - Entering “Getting Started” page
  18. 18. © 2021 InfluxData. All rights reserved. 18 Demo Data
  19. 19. © 2021 InfluxData. All rights reserved. 19 Finding your Organization ID 1 2
  20. 20. © 2021 InfluxData. All rights reserved. 20 Token 2 3 4 1
  21. 21. © 2021 InfluxData. All rights reserved. 21 Find your host Go to… https://bit.ly/3tGLuPy 1. Find your name 2. Copy from column E 3. Paste into console 4. Enter the password
  22. 22. © 2021 InfluxData. All rights reserved. 22 Login Using an SSH client $ ssh centos@[ip-address] $ ssh centos@3.133.90.20 The authenticity of host '3.133.90.20 (3.133.90.20)' can't be established. ECDSA key fingerprint is SHA256:ki5Hm1AQ4UyyOn29uvoexeNO/Mk/Z0qourjEn+EpHdk. Are you sure you want to continue connecting (yes/no)? yes Password: InfluxDaysIsTheBestTrainingEver!!!!
  23. 23. © 2021 InfluxData. All rights reserved. 23
  24. 24. © 2021 InfluxData. All rights reserved. 24 Getting Setup Install Telegraf [centos@influxdays ~]$ sudo rpm -ivh /home/centos/telegraf-1.18.2-1.x86_64.rpm Preparing... ################################# [100%] Updating / installing… 1:telegraf-1.18.2-1 ################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/telegraf.service to /usr/lib/systemd/system/telegraf.service. [centos@influxdays ~]$
  25. 25. © 2021 InfluxData. All rights reserved. 25 Create a Bucket 1 2
  26. 26. © 2020 InfluxData. All rights reserved. 26 3 4
  27. 27. © 2020 InfluxData. All rights reserved. 27 Create Telegraf Configuration 1 2
  28. 28. © 2021 InfluxData. All rights reserved. 28 Create Telegraf Configuration 1 2 3 4 5
  29. 29. © 2021 InfluxData. All rights reserved. 29 Test your Configuration (optional) 1 2 3 Paste in console
  30. 30. © 2021 InfluxData. All rights reserved. 30 Select All Update configuration file 1 2 3 4
  31. 31. © 2021 InfluxData. All rights reserved. 31 VI Cheat Sheet [centos@influxdays ~]$ sudo vi /etc/telegraf/telegraf.conf gg # Go to first line; this may not do anything dG # Delete all lines i # Insert mode <paste> # Paste the full config from InfluxDB Cloud Escape # End Insert mode :64 # Jump to line 64 to update Token :wq # Save and exit
  32. 32. © 2021 InfluxData. All rights reserved. 32 Hardcode the Token Be sure to retain the double quotes 1 2 3
  33. 33. Start the Service [centos@influxdays ~]$ sudo systemctl start telegraf Check the logs [centos@influxdays ~]$ $ journalctl -u telegraf -- Logs begin at Mon 2020-11-02 19:23:33 UTC, end at Mon 2020-11-02 19:55:58 UTC. -- Nov 02 19:55:58 influxdays systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB. Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Starting Telegraf 1.16.1 Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded inputs: cpu disk diskio mem net processes swap system Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded aggregators: Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded processors: Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded outputs: influxdb_v2 Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Tags enabled: host=influxdays Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"influxdays", Flush Interval:10s
  34. 34. © 2021 InfluxData. All rights reserved. 34
  35. 35. Alternate Deployment Launch Telegraf using the provided commands in the Cloud Web UI Open a new SSH console window to continue with the exercises.
  36. 36. - Agent Configuration - Enable Plugins - Filtering - Dual Writes - Telegraf Gateway - Debugging Exercise
  37. 37. Exercise 1 Agent Configuration
  38. 38. © 2021 InfluxData. All rights reserved. 38 Exercise 1 : Agent Configuration [agent] interval = "10s" round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = "0s" flush_interval = "10s" flush_jitter = "0s" precision = "" debug = false quiet = false logfile = "" hostname = "" omit_hostname = false Objective: To understand Agent Configuration and make changes to collection and flush behaviour.
  39. 39. Exercise 2 Enable Input Plugin
  40. 40. © 2021 InfluxData. All rights reserved. 40 Exercise 2 : Enabling Input Plugin Objective: To enable plugin into your current telegraf configuration file and see the results in your InfluxDB Cloud platform. [global_tags] ... [agent] ... [[outputs.influxdb_v2]] ... [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] [[inputs.mem]] [[inputs.net]] [[inputs.processes]] [[inputs.swap]] [[inputs.system]] [[inputs.http_response]] urls = ["https://www.influxdata.com/"]
  41. 41. Exercise 3 Filtering Data
  42. 42. © 2021 InfluxData. All rights reserved. 42 Exercise 3 : Filtering Data Objective: To filter data streaming from “mem” plugin to only pass “field” data that has exact description called “available_percent” and “used_percent”. [global_tags] ... [agent] ... [[outputs.influxdb_v2]] ... ... [[inputs.mem]] fieldpass = [“ available_percent”, “ used_percent”] ...
  43. 43. Exercise 4 Dual Writes
  44. 44. © 2021 InfluxData. All rights reserved. 44 Exercise 4 : Dual Write Objective: Write data to two different outputs. One output to write to your InfluxDB Cloud and the other one will write to a file in your local system. Copy the output plugin configuration that we post in the Slack channel. [global_tags] ... [agent] ... [[outputs.influxdb_v2]] urls = ["https://eu-central-1-1.aws.cloud2.influxdata.com"] ## Token for authentication. token = "$INFLUX_TOKEN" ... [[outputs.file]] files = ["/tmp/metrics.out"] data_format = "influx" ... [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] ...
  45. 45. Exercise 5 Telegraf Gateway
  46. 46. © 2021 InfluxData. All rights reserved. 46 Exercise 5 : Telegraf Gateway Objective: Instead of send the data directly to InfluxDB Cloud, we going to send to a Telegraf Gateway. Add a new output in your telegraf configuration to point to the Gateway. Copy the output code that we posted in the Slack channel. [global_tags] ... [agent] ... [[outputs.influxdb]] urls = ["http://3.135.207.41:8086"] database_tag = "telegraf" exclude_database_tag = true [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] ...
  47. 47. Exercise 6 Debugging
  48. 48. © 2021 InfluxData. All rights reserved. 48 Exercise 6 : Debugging Objective: Let’s see how can we debug what’s going on with our Telegraf Instance. Run journalctl -u telegraf Enable the Debug parameter, change, save and restart Telegraf. Run journalctl -u telegraf [global_tags] ... [agent] debug = true ... [[outputs.influxdb]] ... [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] ...
  49. 49. © 2020 InfluxData. All rights reserved. 49 Telegraf Configuration /home/centos/lab/exercise [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[outputs.influxdb]]
  50. 50. © 2020 InfluxData. All rights reserved. 50 Exercise 1 - Lay of the land Inspect the output of a given configuration $ telegraf --test --config exercise Wait up to this many seconds for service $ telegraf --test-wait 10 --config exercise Gather metrics once, write them, and exit $ telegraf --once --config exercise
  51. 51. © 2020 InfluxData. All rights reserved. 51 Inspect the output of a given configuration $ telegraf --test --config exercise [centos@influxdays lab]$ date; telegraf --test --config exercise ; date Fri Nov 6 22:29:46 UTC 2020 2020-11-06T22:29:46Z I! Starting Telegraf 1.16.1 > mem,host=influxdays active=151506944i,available=732938240i,available_percent=72.28993540203045,buffered=211 7632i,cached=249655296i,commit_limit=506941440i,committed_as=399839232i,dirty=16384i,fr ee=636620800i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,hug e_pages_total=0i,inactive=140464128i,low_free=0i,low_total=0i,mapped=38670336i,page_tab les=4157440i,shared=13058048i,slab=46571520i,sreclaimable=23306240i,sunreclaim=23265280 i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=125493248i,used_perc ent=12.377439593424661,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmal loc_used=6332416i,write_back=0i,write_back_tmp=0i 1604701786000000000 Fri Nov 6 22:29:46 UTC 2020
  52. 52. © 2020 InfluxData. All rights reserved. 52 Wait up to this many seconds for service $ telegraf --test-wait 10 --config exercise $ date; telegraf --test-wait 10 --config exercise; date Fri Nov 6 22:30: 10 UTC 2020 2020-11-06T22:30:10Z I! Starting Telegraf 1.16.1 > mem,host=influxdays active=155013120i,available=728371200i,available_percent=71.8394867713538 9,buffered=2117632i,cached=249663488i,commit_limit=506941440i,committed_a s=403222528i,dirty=16384i,free=632045568i,high_free=0i,high_total=0i,huge _page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=14044 7744i,low_free=0i,low_total=0i,mapped=39026688i,page_tables=4534272i,shar ed=13058048i,slab=46686208i,sreclaimable=23306240i,sunreclaim=23379968i,s wap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=130060288 i,used_percent=12.827888224101224,vmalloc_chunk=35184362651648i,vmalloc_t otal=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0 i 1604701810000000000 Fri Nov 6 22:30: 20 UTC 2020
  53. 53. © 2020 InfluxData. All rights reserved. 53 Write once on start up $ telegraf --once --config exercise $ date; telegraf --once --config exercise; date Fri Nov 6 22:31:51 UTC 2020 2020-11-06T22:31:51Z I! Starting Telegraf 1.16.1 2020-11-06T22:31:51Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:31:51Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:31:51Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:31:51Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:31:51Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics Fri Nov 6 22:31:51 UTC 2020
  54. 54. © 2020 InfluxData. All rights reserved. 54 Mem
  55. 55. © 2020 InfluxData. All rights reserved. 55 Exercise 2 Setup for Debugging
  56. 56. © 2020 InfluxData. All rights reserved. 56 Telegraf Configuration File [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[outputs.influxdb]] $ telegraf --once --config exercise 2020-11-06T22:34:07Z I! Starting Telegraf 1.16.1 2020-11-06T22:34:07Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:07Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:34:07Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:07Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:34:07Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics
  57. 57. © 2020 InfluxData. All rights reserved. 57 $ telegraf --debug --once --config exercise 2020-11-06T22:34:42Z I! Starting Telegraf 1.16.1 2020-11-06T22:34:42Z D! [agent] Initializing plugins 2020-11-06T22:34:42Z D! [agent] Connecting outputs 2020-11-06T22:34:42Z D! [agent] Attempting connection to [outputs.influxdb] 2020-11-06T22:34:42Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:42Z D! [agent] Successfully connected to outputs.influxdb 2020-11-06T22:34:42Z D! [agent] Starting service inputs 2020-11-06T22:34:42Z D! [agent] Stopping service inputs 2020-11-06T22:34:42Z D! [agent] Input channel closed 2020-11-06T22:34:42Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:34:42Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:42Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics 2020-11-06T22:34:42Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:34:42Z D! [agent] Stopped Successfully 2020-11-06T22:34:42Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics
  58. 58. © 2020 InfluxData. All rights reserved. 58 telegraf --once --config exercise 2020-11-06T22:35:58Z I! Starting Telegraf 1.16.1 2020-11-06T22:35:58Z D! [agent] Initializing plugins 2020-11-06T22:35:58Z D! [agent] Connecting outputs 2020-11-06T22:35:58Z D! [agent] Attempting connection to [outputs.influxdb] 2020-11-06T22:35:58Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:35:58Z D! [agent] Successfully connected to outputs.influxdb 2020-11-06T22:35:58Z D! [agent] Starting service inputs 2020-11-06T22:35:58Z D! [agent] Stopping service inputs 2020-11-06T22:35:58Z D! [agent] Input channel closed 2020-11-06T22:35:58Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:35:58Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:35:58Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics 2020-11-06T22:35:58Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:35:58Z D! [agent] Stopped Successfully 2020-11-06T22:35:58Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics Updated Telegraf Configuration File [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[outputs.influxdb]]
  59. 59. © 2020 InfluxData. All rights reserved. 59 Mem Stdout
  60. 60. © 2020 InfluxData. All rights reserved. 60 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[outputs.influxdb]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[outputs.file]] $ telegraf --once --config exercise Output on next slide...
  61. 61. © 2020 InfluxData. All rights reserved. 61 $ telegraf --once --config exercise 2020-11-06T22:37:05Z I! Starting Telegraf 1.16.1 2020-11-06T22:37:05Z D! [agent] Initializing plugins 2020-11-06T22:37:05Z D! [agent] Connecting outputs 2020-11-06T22:37:05Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:37:05Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:37:05Z D! [agent] Starting service inputs 2020-11-06T22:37:05Z D! [agent] Stopping service inputs 2020-11-06T22:37:05Z D! [agent] Input channel closed 2020-11-06T22:37:05Z I! [agent] Hang on, flushing any cached metrics before shutdown mem,host=influxdays sreclaimable=23232512i,page_tables=4255744i,shared=13058048i,swap_cached=0i,vmalloc_total=351843720878 08i,write_back_tmp=0i,dirty=20480i,low_free=0i,huge_pages_free=0i,committed_as=403091456i,free=6308044 80i,sunreclaim=23289856i,swap_free=0i,cached=250294272i,huge_pages_total=0i,write_back=0i,total=101388 6976i,active=156626944i,slab=46522368i,vmalloc_chunk=35184362651648i,vmalloc_used=6332416i,used=130670 592i,inactive=140836864i,available_percent=71.78292819889226,buffered=2117632i,available=727797760i,us ed_percent=12.88808270479253,high_total=0i,huge_page_size=2097152i,low_total=0i,mapped=40583168i,swap_ total=0i,commit_limit=506941440i,high_free=0i 1604702225000000000 2020-11-06T22:37:05Z D! [outputs.file] Wrote batch of 1 metrics in 92.731µs 2020-11-06T22:37:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2020-11-06T22:37:05Z D! [agent] Stopped Successfully
  62. 62. © 2020 InfluxData. All rights reserved. 62 Exercise 3 Enabling a plugin
  63. 63. © 2020 InfluxData. All rights reserved. 63 Mem Net Stdout
  64. 64. © 2020 InfluxData. All rights reserved. 64 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[inputs.net]] [[outputs.file]]
  65. 65. © 2020 InfluxData. All rights reserved. 65 Validate using --test $ telegraf --test --config exercise $ telegraf --test --config exercise 2020-11-06T22:38:39Z I! Starting Telegraf 1.16.1 2020-11-06T22:38:39Z D! [agent] Initializing plugins 2020-11-06T22:38:39Z D! [agent] Starting service inputs > net,host=influxdays,interface=ens5 bytes_recv=1708843i,bytes_sent=2007423i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=7752i,packets_sent=5597i 1604702319000000000 > net,host=influxdays,interface=all icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_i nerrors=0i,icmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps= 0i,icmp_intimestamps=0i,icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0 i,icmp_outerrors=0i,icmp_outmsgs=2i,icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_ou ttimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagra ms=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_inaddrerrors=0i,ip_indelivers=7981i,ip_indiscards=0i,ip_inhdrerrors=0i, ip_inreceives=7986i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnoroutes=16i,ip_outrequests=5516i,ip_reasmfails=0i,ip_reasmoks =0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tcp_currestab=4i,tcp_estabresets=4i,tcp_incsum errors=5i,tcp_inerrs=5i,tcp_insegs=7733i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5563i,tcp_passiveopens=68i,tcp_retranssegs= 43i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=312i,udp_inerrors=0i,udp_noports= 2i,udp_outdatagrams=315i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_inerrors =0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702319000000000 2020-11-06T22:38:39Z D! [agent] Stopping service inputs 2020-11-06T22:38:39Z D! [agent] Input channel closed 2020-11-06T22:38:39Z D! [agent] Stopped Successfully > mem,host=influxdays active=159830016i,available=723709952i,available_percent=71.37974637520149,buffered=2117632i,cached=250327040i,commit_limit=5069 41440i,committed_as=407773184i,dirty=8192i,free=626688000i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i ,huge_pages_total=0i,inactive=140804096i,low_free=0i,low_total=0i,mapped=40611840i,page_tables=4898816i,shared=13058048i,slab=46 534656i,sreclaimable=23240704i,sunreclaim=23293952i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=134754304i, used_percent=13.290860538680004,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i, write_back_tmp=0i 1604702319000000000
  66. 66. © 2020 InfluxData. All rights reserved. 66 $ telegraf --config exercise 2020-11-06T22:39:26Z I! Starting Telegraf 1.16.1 2020-11-06T22:39:26Z I! Loaded inputs: mem net 2020-11-06T22:39:26Z I! Loaded aggregators: 2020-11-06T22:39:26Z I! Loaded processors: 2020-11-06T22:39:26Z I! Loaded outputs: file 2020-11-06T22:39:26Z I! Tags enabled: host=influxdays 2020-11-06T22:39:26Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s 2020-11-06T22:39:26Z D! [agent] Initializing plugins 2020-11-06T22:39:26Z D! [agent] Connecting outputs 2020-11-06T22:39:26Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:39:26Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:39:26Z D! [agent] Starting service inputs mem,host=influxdays free=629272576i,huge_pages_free=0i,low_total=0i,commit_limit=506941440i,high_free=0i,huge_pages_total=0i,vmalloc_chunk=35184362651648i,used=132153344i,high_tot al=0i,shared=13062144i,sunreclaim=23162880i,vmalloc_total=35184372087808i,write_back_tmp=0i,inactive=140779520i,used_percent=13.034327013586177,swap_cached =0i,vmalloc_used=6332416i,write_back=0i,buffered=2117632i,cached=250343424i,committed_as=403091456i,swap_total=0i,total=1013886976i,available=726310912i,availa ble_percent=71.63627990029532,active=158572544i,dirty=4096i,page_tables=4214784i,slab=46395392i,huge_page_size=2097152i,low_free=0i,mapped=40656896i,sreclai mable=23232512i,swap_free=0i 1604702368000000000 net,host=influxdays,interface=ens5 drop_in=0i,drop_out=0i,bytes_sent=2037575i,bytes_recv=1723821i,packets_sent=5696i,packets_recv=7891i,err_in=0i,err_out=0i 1604702368000000000 net,host=influxdays,interface=all udplite_rcvbuferrors=0i,ip_outdiscards=0i,ip_inaddrerrors=0i,icmp_inechoreps=0i,icmp_outtimestampreps=0i,icmp_outerrors=0i,icmp_outechoreps=0i,icmp_outaddrmaskreps =0i,icmpmsg_outtype3=2i,udplite_noports=0i,udplite_incsumerrors=0i,udp_noports=2i,icmp_incsumerrors=0i,icmp_outechos=0i,icmp_outaddrmasks=0i,icmp_inechos=0i,icmp _outparmprobs=0i,ip_inhdrerrors=0i,ip_outnoroutes=16i,ip_defaultttl=64i,icmp_outdestunreachs=2i,udp_indatagrams=316i,tcp_currestab=1i,ip_fragfails=0i,ip_reasmtimeout=0 i,ip_reasmoks=0i,icmp_outredirects=0i,tcp_retranssegs=45i,tcp_passiveopens=69i,tcp_maxconn=-1i,tcp_inerrs=5i,ip_inreceives=8124i,ip_forwarding=2i,icmp_inmsgs=2i,icmp _indestunreachs=2i,icmpmsg_intype3=2i,ip_indiscards=0i,icmp_intimeexcds=0i,udp_incsumerrors=0i,tcp_outsegs=5655i,ip_forwdatagrams=0i,ip_fragcreates=0i,icmp_inredir ects=0i,icmp_outsrcquenchs=0i,icmp_inaddrmasks=0i,ip_indelivers=8119i,icmp_inparmprobs=0i,tcp_insegs=7867i,tcp_attemptfails=72i,udp_inerrors=0i,tcp_estabresets=4i,tc p_outrsts=68i,udp_rcvbuferrors=0i,ip_reasmfails=0i,ip_fragoks=0i,ip_reasmreqds=0i,icmp_outtimestamps=0i,icmp_intimestampreps=0i,icmp_inerrors=0i,tcp_incsumerrors=5i, udplite_inerrors=0i,ip_outrequests=5610i,icmp_insrcquenchs=0i,icmp_outmsgs=2i,icmp_outtimeexcds=0i,udp_sndbuferrors=0i,tcp_rtoalgorithm=1i,udp_outdatagrams=319i,u dplite_outdatagrams=0i,tcp_rtomin=200i,icmp_inaddrmaskreps=0i,udplite_sndbuferrors=0i,ip_inunknownprotos=0i,icmp_intimestamps=0i,tcp_rtomax=120000i,tcp_activeopen s=142i,udplite_indatagrams=0i 1604702368000000000 2020-11-06T22:39:28Z D! [outputs.file] Wrote batch of 3 metrics in 139.162µs 2020-11-06T22:39:28Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics Validate without --test
  67. 67. © 2020 InfluxData. All rights reserved. 67 Filtering
  68. 68. © 2021 InfluxData. All rights reserved. 68 © 2019 InfluxData. All rights reserved. 68 What Is An InfluxDB Series? First, some definitions: • Measurement: namespace for categorically alike metrics (~Table) • Tag: key-value pair used as metadata for measurable object – host=prod001 or app=cart or customer_id=00012 • Field: key-value pair representing a metric that will append new values over time – cpu_usage=35.5 or errors=42 or temp=65.0 • TagSet: unique combination of Tag key-value pairs • FieldSet: unique combination of Field key-value pairs
  69. 69. © 2021 InfluxData. All rights reserved. 69 © 2019 InfluxData. All rights reserved. 69 What Is An InfluxDB Series? Line/Record: <Measurement>,<TagSet> <FieldSet> <timestamp> → cpu,host=prod001,region=us-west usage_user=35,usage_system=15 t0 Point: <Measurement>,<TagSet> <FieldKey> <timestamp> → cpu,host=prod001,region=us-west usage_user=35 t0 Series: <Measurement>,<TagSet> <FieldKey> → cpu,host=prod001,region=us-west usage_user
  70. 70. © 2021 InfluxData. All rights reserved. 70 © 2019 InfluxData. All rights reserved. 70 Series Visualized (tree) Cardinality is total number of “leaf nodes”
  71. 71. © 2020 InfluxData. All rights reserved. 71 Filtering metrics • Happens at plugin level • Specify which metrics and which parts of metrics the given plugin sees or doesn’t see • Given above, filtering is often used for “routing” • By Measurement: • namepass: passes only records with specified Measurement name • namedrop: drops only records with specified Measurement name • By Field: • fieldpass: pass specified Field • fielddrop: drop specified Field • By Tag (slightly different implementation): • tagpass or tagdrop: pass/drop whole record containing matched Tag • taginclude or tagexclude: keep/drop matching Tag (key) in record
  72. 72. © 2020 InfluxData. All rights reserved. 72 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA free=85.0,cached=12.5,buffered=3.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA free=85.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA cached=85.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA buffered=85.0 <timestamp> namepass = “mem”
  73. 73. © 2020 InfluxData. All rights reserved. 73 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0 ,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp> fieldpass = “*_system”
  74. 74. © 2020 InfluxData. All rights reserved. 74 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0 ,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp> fielddrop = “*_softirq”
  75. 75. © 2020 InfluxData. All rights reserved. 75 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp> tagpass: host = [“hostA”]
  76. 76. © 2020 InfluxData. All rights reserved. 76 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp> tagdrop: host = [“hostB”]
  77. 77. © 2020 InfluxData. All rights reserved. 77 How to think of this Telegraf metric: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp> taginclude: host = [“host”]
  78. 78. © 2020 InfluxData. All rights reserved. 78 How to think of this Telegraf metric: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp> tagexclude: host = [“host”]
  79. 79. © 2020 InfluxData. All rights reserved. 79 Exercise 4 Filtering
  80. 80. © 2020 InfluxData. All rights reserved. 80 Sample Data mem,host=influxdays available=735477760i,inactive=167567360i,page_tables=4345856i,sreclaimable=22962176i,committed_as=402309120i,slab=46772224i,vmalloc_total=35184 372087808i,cached=243228672i,high_free=0i,huge_page_size=2097152i,huge_pages_total=0i,write_back=0i,total=1013886976i,dirty=0i,free=645414912i,lo w_total=0i,sunreclaim=23810048i,swap_cached=0i,huge_pages_free=0i,low_free=0i,shared=13058048i,swap_total=0i,used=123125760i,used_percent=12.1 43933487118785,buffered=2117632i,vmalloc_used=6332416i,available_percent=72.54040908007482,high_total=0i,vmalloc_chunk=35184362651648i,active =113328128i,commit_limit=506941440i,mapped=40271872i,swap_free=0i,write_back_tmp=0i 1602630666000000000 net,host=influxdays,interface=ens5 packets_recv=11760i,err_in=0i,err_out=0i,drop_in=0i,drop_out=0i,bytes_sent=1730127i,bytes_recv=1082190i,packets_sent=10073i 1602630666000000000 net,host=influxdays,interface=all icmp_outechos=0i,icmp_inechoreps=0i,tcp_rtoalgorithm=1i,tcp_insegs=9422i,ip_inunknownprotos=0i,ip_indiscards=0i,ip_reasmreqds=0i,icmp_outdestunreac hs=2i,icmpmsg_outtype3=2i,icmp_outaddrmasks=0i,icmp_outredirects=0i,icmp_indestunreachs=3i,icmp_incsumerrors=0i,udp_incsumerrors=0i,ip_inhdrerrors =0i,ip_reasmoks=0i,tcp_maxconn=-1i,udp_sndbuferrors=0i,tcp_incsumerrors=0i,tcp_estabresets=21i,udplite_rcvbuferrors=0i,ip_reasmtimeout=0i,icmp_outerr ors=0i,icmp_intimeexcds=0i,udplite_sndbuferrors=0i,icmpmsg_intype3=3i,tcp_activeopens=188i,tcp_rtomax=120000i,udp_outdatagrams=991i,udplite_incsum errors=0i,ip_fragcreates=0i,ip_fragoks=0i,icmp_inerrors=0i,tcp_rtomin=200i,udp_rcvbuferrors=0i,udp_noports=2i,ip_inaddrerrors=0i,ip_reasmfails=0i,ip_forwar ding=2i,icmp_outechoreps=0i,icmp_intimestamps=0i,icmp_outmsgs=2i,tcp_outrsts=141i,udp_indatagrams=932i,ip_inreceives=10233i,ip_outdiscards=0i,icmp _intimestampreps=0i,icmp_outtimestamps=0i,icmp_outsrcquenchs=0i,tcp_passiveopens=269i,tcp_currestab=1i,udplite_indatagrams=0i,ip_outnoroutes=16i,ic mp_outtimestampreps=0i,icmp_outparmprobs=0i,icmp_inaddrmasks=0i,icmp_outaddrmaskreps=0i,icmp_inparmprobs=0i,ip_defaultttl=64i,ip_outrequests=836 7i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_outtimeexcds=0i,icmp_inmsgs=3i,icmp_inaddrmaskreps=0i,ip_forwdatagrams=0i,ip_fragfails=0i,icmp_inec hos=0i,tcp_inerrs=0i,tcp_attemptfails=175i,tcp_outsegs=7517i,udplite_inerrors=0i,udplite_noports=0i,ip_indelivers=10227i,tcp_retranssegs=118i,udp_inerrors =0i,udplite_outdatagrams=0i 1602630666000000000
  81. 81. © 2020 InfluxData. All rights reserved. 81 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[inputs.net]] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] Pass fields ending in “percent” exact named “total” [[inputs.net]] Pass fields Beginning with “bytes” Pass metrics where interface begins with “en” [[outputs.file]]
  82. 82. © 2020 InfluxData. All rights reserved. 82 Exercise 4 Filtering Solution
  83. 83. © 2020 InfluxData. All rights reserved. 83 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[inputs.net]] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] ...
  84. 84. © 2020 InfluxData. All rights reserved. 84 Correct Placement [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] ...
  85. 85. © 2020 InfluxData. All rights reserved. 85 Before & After $ telegraf --test --config exercise > net,host=influxdays,interface= ens5 bytes_recv=1739020i,bytes_sent=2088482i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=8042i,packets_sent=5843i 1604702475000000000 > mem,host=influxdays active=154382336i,available=730259456i,available_percent=72.02572607067398,buffered=2117632i,cached=250368000i,commit_limit=506941440i,commi tted_as=403091456i,dirty=20480i,free=633192448i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,in active=140689408i,low_free=0i,low_total=0i,mapped=40558592i,page_tables=4190208i,shared=13058048i,slab=46452736i,sreclaimable=23232512i,sunr eclaim=23220224i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=128208896i,used_percent=12.645284833010814,vmalloc_chunk=3 5184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i 1604702475000000000 2020-11-06T22:41:14Z D! [agent] Stopping service inputs 2020-11-06T22:41:14Z D! [agent] Input channel closed 2020-11-06T22:41:14Z D! [agent] Stopped Successfully > net,host=influxdays,interface= all icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_inerrors=0i,i cmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=0i,icmp_intimestamps=0i, icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0i,icmp_outerrors=0i,icmp_outmsgs=2i, icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_outtimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_ intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagrams=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_ina ddrerrors=0i,ip_indelivers=8268i,ip_indiscards=0i,ip_inhdrerrors=0i,ip_inreceives=8273i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnorout es=16i,ip_outrequests=5747i,ip_reasmfails=0i,ip_reasmoks=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tc p_currestab=1i,tcp_estabresets=4i,tcp_incsumerrors=5i,tcp_inerrs=5i,tcp_insegs=8010i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5792i,tcp_p assiveopens=74i,tcp_retranssegs=47i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=322i,udp_iner rors=0i,udp_noports=2i,udp_outdatagrams=325i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_ inerrors=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702475000000000 …. After filtering... …. > mem,host=influxdays available_percent=71.50255927540388,total=1013886976i,used_percent=13.168451628280902 1604702498000000000 > net,host=influxdays,interface= ens5 bytes_recv=1745105i,bytes_sent=2111895i 1604702498000000000
  86. 86. © 2020 InfluxData. All rights reserved. 86 Placement Matters, Indentation Does Not [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] [inputs.net.tagpass] interface = ["en*"] fieldpass = ["bytes*"] [[outputs.file]] > net,host=influxdays,interface=ens5 bytes_recv=2082848i,bytes_sent=3148789i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=21247 i,packets_sent=19323i 1604702498000000000
  87. 87. © 2020 InfluxData. All rights reserved. 87 Exercise 5 Dual Write
  88. 88. © 2020 InfluxData. All rights reserved. 88 Mem Net net /tmp/net m em /tmp/mem
  89. 89. © 2020 InfluxData. All rights reserved. 89 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] Pass the “mem” data to file /tmp/mem [[outputs.file]] Pass the “net” data to file /tmp/net
  90. 90. © 2020 InfluxData. All rights reserved. 90 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] namepass = ["mem"] files = ["/tmp/mem"] [[outputs.file]] namepass = ["net"] files = ["/tmp/net"]
  91. 91. © 2020 InfluxData. All rights reserved. 91 Exercise 6 Listener
  92. 92. © 2020 InfluxData. All rights reserved. 92 Mem Net ne t /tmp/net.txt m em /tmp/mem.txt :8086 Add
  93. 93. © 2020 InfluxData. All rights reserved. 93 telegraf-listener.conf [global_tags] [agent] interval = "2s" flush_interval = "2s" round_interval = true metric_batch_size = 5000 metric_buffer_limit = 50000 debug = true logfile = "/tmp/telegraf-aggregator.log" [[inputs.influxdb_listener]] Set the listener port to ":8086" read_timeout = "2s" write_timeout = "2s" [[outputs.file]]
  94. 94. © 2020 InfluxData. All rights reserved. 94 telegraf-listener.conf [global_tags] [agent] interval = "2s" flush_interval = "2s" round_interval = true metric_batch_size = 5000 metric_buffer_limit = 50000 debug = true logfile = "/tmp/telegraf-aggregator.log" [[inputs.influxdb_listener]] service_address = ": 8086" read_timeout = "2s" write_timeout = "2s" [[outputs.file]]
  95. 95. © 2020 InfluxData. All rights reserved. 95 Launch Telegraf Listener as background job $ telegraf --config telegraf-listener.conf & [1] 2219 [centos@influxdays ~]$ 2020-11-06T22:44:22Z I! Starting Telegraf 1.16.1 [centos@influxdays ~]$ [centos@influxdays ~]$ [centos@influxdays ~]$
  96. 96. © 2020 InfluxData. All rights reserved. 96 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] namepass = ["mem"] files = ["/tmp/mem"] [[outputs.file]] namepass = ["net"] files = ["/tmp/net"] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] namepass = ["mem"] files = ["/tmp/mem"] [[outputs.file]] namepass = ["net"] files = ["/tmp/net"] [[outputs.influxdb]]
  97. 97. © 2020 InfluxData. All rights reserved. 97 Launch Telegraf $ telegraf -config exercise 2020-11-06T22:45:09Z I! Starting Telegraf 1.16.1 2020-11-06T22:45:09Z I! Loaded inputs: mem net 2020-11-06T22:45:09Z I! Loaded aggregators: 2020-11-06T22:45:09Z I! Loaded processors: 2020-11-06T22:45:09Z I! Loaded outputs: file (2x) influxdb 2020-11-06T22:45:09Z I! Tags enabled: host=influxdays 2020-11-06T22:45:09Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s 2020-11-06T22:45:09Z D! [agent] Initializing plugins 2020-11-06T22:45:09Z D! [agent] Connecting outputs 2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.influxdb] 2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.influxdb 2020-11-06T22:45:09Z D! [agent] Starting service inputs 2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 101.011µs 2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 23.33µs 2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2020-11-06T22:45:11Z D! [outputs.influxdb] Wrote batch of 2 metrics in 1.205233ms 2020-11-06T22:45:11Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766 1604702710000000000 net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000
  98. 98. © 2020 InfluxData. All rights reserved. 98 Inspect the filesystem $ cat /tmp/mem mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766 1604702710000000000 $ cat /tmp/net net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000
  99. 99. © 2020 InfluxData. All rights reserved. 99 telegraf-listener.conf [global_tags] [agent] interval = "2s" flush_interval = "2s" round_interval = true metric_batch_size = 5000 metric_buffer_limit = 50000 debug = true logfile = "/tmp/telegraf-aggregator.log" [[inputs.influxdb_listener]] service_address = ":8086" read_timeout = "2s" write_timeout = "2s" [[outputs.file]]
  100. 100. © 2020 InfluxData. All rights reserved. 100 Exercise 7 Starlark
  101. 101. © 2020 InfluxData. All rights reserved. 101 Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[processors.starlark]] source = ''' def apply(metric): for k, v in metric.fields.items(): if type(v) == "float" or type(v) == "int": metric.fields[k] = v * 1000 return metric ''' [[outputs.file]] Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] .. .. [[outputs.file]]
  102. 102. © 2020 InfluxData. All rights reserved. 102 Launch Telegraf Without Starlark processor > net,host=influxdays,interface=ens5 bytes_recv=9566941i,bytes_sent=23302741i 1604939494000000000 After multiplying by 1000 > net,host=influxdays,interface=ens5 bytes_recv=9555668000i,bytes_sent=23274132000i 1604939466000000000
  103. 103. © 2020 InfluxData. All rights reserved. 103 Exercise 8 JSON
  104. 104. © 2020 InfluxData. All rights reserved. 104 Mem Net net /tmp/net.txt m em /tmp/mem.txt HTTP
  105. 105. © 2020 InfluxData. All rights reserved. 105 Add [[inputs.http]] urls = ["https://api.mocki.io/v1/29dfca5b"] data_format = “json” [[outputs.file]] Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]]
  106. 106. © 2020 InfluxData. All rights reserved. 106 Add [[inputs.http]] urls = ["https://api.mocki.io/v1/29dfca5b"] data_format = “json” name_override = “apps” json_query = “apps” tag_keys = [“name”,”id”] [[outputs.file]] Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]]
  107. 107. © 2021 InfluxData. All rights reserved. 107 Telegraf Resources Blogs https:/ /www.influxdata.com/blog/using-a-telegraf-gateway/ https:/ /www.influxdata.com/blog/telegraf-go-collection-agent/ Docs https:/ /docs.influxdata.com/telegraf/latest Prebuilt dashboards https:/ /docs.influxdata.com/platform/monitoring/influxdata-platform/monitoring-dashb oards/
  108. 108. Questions?
  109. 109. Thank you See you tomorrow at InfluxDays EMEA 2021!
  110. 110. © 2021 InfluxData. All rights reserved. 110 InfluxDays EMEA 2021 See you tomorrow: 18-19 May 2021

×