1. © 2021 InfluxData. All rights reserved. 1
© 2020 InfluxData. All rights reserved.
1
Ignacio Van Droogenbroeck | Solutions Architect, InfluxData
Suleman Khan | Solutions Architect, InfluxData
May 2021

2. Agenda
■ Overview
■ Setup (if not completed)
■ Global configuration
■ Enabling plugins
■ Filtering data

3. Please join us in the InfluxDB Community Slack at
www.influxdata.com/slack.
To participate in the conversations,
join the #influxdays-virtual & #telegraf-training channels.
JOIN THE CONVERSATION

4. Workshop Logistics
• Cloud account free tier
• No CC
• Rate limits → Let’s avoid them
• Browser
• SSH client
• Have documentation open for exercise
• General config: https:/
/bit.ly/3t1r03K

5. © 2021 InfluxData. All rights reserved. 5
Telegraf
• Lightweight; written in Go
• Plug-in driven
• Optimized for writing to InfluxDB
• Formatting
• Retries
• Modifiable batch sizes and jitter
• Tag sorting
• Preprocessing
• Converting tags to fields, fields to tags
• Regex transformations
• Renaming measurements, tags
• Aggregations (mean, min, max, count, variance, stddev, etc.)

6. © 2021 InfluxData. All rights reserved. 6
Popular Inputs (Integrations)
Out-of-the-box Custom
Kubernetes (kubelet) HTTP/socket listener
Kube_inventory (apiserver) HTTP (formatted endpoints)
Kafka (producer/consumer) Prometheus (/metrics)
SNMP Exec
System (Linux,Windows,Unix) StatsD
Nginx
HAproxy
OPC, AMQP, Modbus & MQTT (IoT)

7. © 2021 InfluxData. All rights reserved. 7
Telegraf
CPU
Mem
Disk
Docker
Kubernetes
/metrics
Kafka
MySQL
Process
-transform
-decorate
-filter
Aggregate
-mean
-min,max
-count
-variance
-stddev
File
InfluxDB
Kafka
CloudWatch
CloudWatch

8. © 2021 InfluxData. All rights reserved. 8

9. © 2021 InfluxData. All rights reserved. 9
Telegraf Common Architecture
Messaging
Queue
Kafka
Rabbit
Active
NSQ
AWS Kinesis
Google PubSub
MQTT

10. © 2021 InfluxData. All rights reserved. 10
Telegraf Architecture
Message Queue
Kafka
Rabbit
Active
NSQ
AWS Kinesis
Google PubSub
MQTT

11. © 2021 InfluxData. All rights reserved. 11
Parsing
● JSON
● CSV
● Graphite
● CollectD
● Dropwizard
● Form URL-encoded
● Grok

12. © 2021 InfluxData. All rights reserved. 12
Telegraf
Latest version available at
https:/
/portal.influxdata.com/downloads/#telegraf
Create the configuration file
$ telegraf config > telegraf.conf
# ~7500 lines
OR
$ telegraf -sample-config -input-filter cpu:mem -output-filter influxdb > telegraf.conf
# 830 lines

13. © 2021 InfluxData. All rights reserved. 13
Starlark
Embedded Python dialect
– Not full std lib
– But most basic Python functionality supported
Current processors and aggregators + more
[[processors.starlark]]
namepass = ["mymeasurement"]
source = '''
def apply(metric):
I = metric.fields['current']
V = metric.fields['voltage']
metric.fields['power'] = I * V
return metric
'''
[[processors.starlark]]
source = '''
def apply(metric):
used = float(metric.fields['used'])
total = float(metric.fields['total'])
metric.fields['usage'] = (used / total) * 100
return metric
'''

14. Setting up InfluxDB Cloud
Sign up: https:/
/cloud2.influxdata.com/signup

15. © 2021 InfluxData. All rights reserved. 15
InfluxDB Cloud Setup
- Choose your Provider & Region
- Enter your company name
- Read & Tick the Service Agreement

16. © 2021 InfluxData. All rights reserved. 16
InfluxDB Cloud Setup
● Select “Free” account

17. © 2021 InfluxData. All rights reserved. 17
InfluxDB Cloud Setup
- Entering “Getting Started” page

18. © 2021 InfluxData. All rights reserved. 18
Demo Data

19. © 2021 InfluxData. All rights reserved. 19
Finding your Organization ID
1
2

20. © 2021 InfluxData. All rights reserved. 20
Token
2
3 4
1

21. © 2021 InfluxData. All rights reserved. 21
Find your host
Go to… https://bit.ly/3tGLuPy
1. Find your name
2. Copy from column E
3. Paste into console
4. Enter the password

22. © 2021 InfluxData. All rights reserved. 22
Login
Using an SSH client
$ ssh centos@[ip-address]
$ ssh centos@3.133.90.20
The authenticity of host '3.133.90.20 (3.133.90.20)' can't be established.
ECDSA key fingerprint is SHA256:ki5Hm1AQ4UyyOn29uvoexeNO/Mk/Z0qourjEn+EpHdk.
Are you sure you want to continue connecting (yes/no)? yes
Password:
InfluxDaysIsTheBestTrainingEver!!!!

23. © 2021 InfluxData. All rights reserved. 23

24. © 2021 InfluxData. All rights reserved. 24
Getting Setup
Install Telegraf
[centos@influxdays ~]$ sudo rpm -ivh
/home/centos/telegraf-1.18.2-1.x86_64.rpm
Preparing... #################################
[100%]
Updating / installing…
1:telegraf-1.18.2-1 #################################
[100%]
Created symlink from
/etc/systemd/system/multi-user.target.wants/telegraf.service to
/usr/lib/systemd/system/telegraf.service.
[centos@influxdays ~]$

25. © 2021 InfluxData. All rights reserved. 25
Create a Bucket
1
2

26. © 2020 InfluxData. All rights reserved.
26
3
4

27. © 2020 InfluxData. All rights reserved.
27
Create Telegraf Configuration
1
2

28. © 2021 InfluxData. All rights reserved. 28
Create Telegraf Configuration
1
2
3
4
5

29. © 2021 InfluxData. All rights reserved. 29
Test your Configuration (optional)
1 2
3
Paste in console

30. © 2021 InfluxData. All rights reserved. 30
Select All
Update configuration file
1
2
3
4

31. © 2021 InfluxData. All rights reserved. 31
VI Cheat Sheet
[centos@influxdays ~]$ sudo vi /etc/telegraf/telegraf.conf
gg # Go to first line; this may not do anything
dG # Delete all lines
i # Insert mode
<paste> # Paste the full config from InfluxDB Cloud
Escape # End Insert mode
:64 # Jump to line 64 to update Token
:wq # Save and exit

32. © 2021 InfluxData. All rights reserved. 32
Hardcode the Token
Be sure to retain the double quotes
1
2
3

33. Start the Service
[centos@influxdays ~]$ sudo systemctl
start telegraf
Check the logs
[centos@influxdays ~]$ $ journalctl
-u telegraf
-- Logs begin at Mon 2020-11-02 19:23:33 UTC, end at Mon 2020-11-02 19:55:58 UTC.
--
Nov 02 19:55:58 influxdays systemd[1]: Started The plugin-driven server agent for
reporting metrics into InfluxDB.
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Starting
Telegraf 1.16.1
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded inputs:
cpu disk diskio mem net processes swap system
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded
aggregators:
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded
processors:
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded
outputs: influxdb_v2
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Tags enabled:
host=influxdays
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! [agent]
Config: Interval:10s, Quiet:false, Hostname:"influxdays", Flush Interval:10s

34. © 2021 InfluxData. All rights reserved. 34

35. Alternate Deployment
Launch Telegraf using the provided commands in
the Cloud Web UI
Open a new SSH console window to continue with
the exercises.

36. - Agent Configuration
- Enable Plugins
- Filtering
- Dual Writes
- Telegraf Gateway
- Debugging
Exercise

37. Exercise 1
Agent Configuration

38. © 2021 InfluxData. All rights reserved. 38
Exercise 1 : Agent Configuration
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
debug = false
quiet = false
logfile = ""
hostname = ""
omit_hostname = false
Objective: To understand Agent
Configuration and make changes to
collection and flush behaviour.

39. Exercise 2
Enable Input Plugin

40. © 2021 InfluxData. All rights reserved. 40
Exercise 2 : Enabling Input Plugin
Objective: To enable plugin into your current
telegraf configuration file and see the results in
your InfluxDB Cloud platform.
[global_tags]
...
[agent]
...
[[outputs.influxdb_v2]]
...
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.http_response]]
urls = ["https://www.influxdata.com/"]

41. Exercise 3
Filtering Data

42. © 2021 InfluxData. All rights reserved. 42
Exercise 3 : Filtering Data
Objective: To filter data streaming from “mem”
plugin to only pass “field” data that has exact
description called “available_percent” and
“used_percent”.
[global_tags]
...
[agent]
...
[[outputs.influxdb_v2]]
...
...
[[inputs.mem]]
fieldpass = [“
available_percent”, “
used_percent”]
...

43. Exercise 4
Dual Writes

44. © 2021 InfluxData. All rights reserved. 44
Exercise 4 : Dual Write
Objective: Write data to two different
outputs. One output to write to your
InfluxDB Cloud and the other one will
write to a file in your local system.
Copy the output plugin configuration
that we post in the Slack channel.
[global_tags]
...
[agent]
...
[[outputs.influxdb_v2]]
urls = ["https://eu-central-1-1.aws.cloud2.influxdata.com"]
## Token for authentication.
token = "$INFLUX_TOKEN"
...
[[outputs.file]]
files = ["/tmp/metrics.out"]
data_format = "influx"
...
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
...

45. Exercise 5
Telegraf Gateway

46. © 2021 InfluxData. All rights reserved. 46
Exercise 5 : Telegraf Gateway
Objective: Instead of send the data
directly to InfluxDB Cloud, we going to
send to a Telegraf Gateway. Add a
new output in your telegraf
configuration to point to the Gateway.
Copy the output code that we posted
in the Slack channel.
[global_tags]
...
[agent]
...
[[outputs.influxdb]]
urls = ["http://3.135.207.41:8086"]
database_tag = "telegraf"
exclude_database_tag = true
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
...

47. Exercise 6
Debugging

48. © 2021 InfluxData. All rights reserved. 48
Exercise 6 : Debugging
Objective: Let’s see how can we
debug what’s going on with our
Telegraf Instance.
Run journalctl -u telegraf
Enable the Debug parameter, change,
save and restart Telegraf.
Run journalctl -u telegraf
[global_tags]
...
[agent]
debug = true
...
[[outputs.influxdb]]
...
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
...

49. © 2020 InfluxData. All rights reserved.
49
Telegraf Configuration
/home/centos/lab/exercise
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
[[inputs.mem]]
[[outputs.influxdb]]

50. © 2020 InfluxData. All rights reserved.
50
Exercise 1 - Lay of the land
Inspect the output of a given configuration
$ telegraf --test --config exercise
Wait up to this many seconds for service
$ telegraf --test-wait 10 --config exercise
Gather metrics once, write them, and exit
$ telegraf --once --config exercise

51. © 2020 InfluxData. All rights reserved.
51
Inspect the output of a given configuration
$ telegraf --test --config exercise
[centos@influxdays lab]$ date; telegraf --test --config exercise ; date
Fri Nov 6 22:29:46 UTC 2020
2020-11-06T22:29:46Z I! Starting Telegraf 1.16.1
> mem,host=influxdays
active=151506944i,available=732938240i,available_percent=72.28993540203045,buffered=211
7632i,cached=249655296i,commit_limit=506941440i,committed_as=399839232i,dirty=16384i,fr
ee=636620800i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,hug
e_pages_total=0i,inactive=140464128i,low_free=0i,low_total=0i,mapped=38670336i,page_tab
les=4157440i,shared=13058048i,slab=46571520i,sreclaimable=23306240i,sunreclaim=23265280
i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=125493248i,used_perc
ent=12.377439593424661,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmal
loc_used=6332416i,write_back=0i,write_back_tmp=0i 1604701786000000000
Fri Nov 6 22:29:46 UTC 2020

52. © 2020 InfluxData. All rights reserved.
52
Wait up to this many seconds for service
$ telegraf --test-wait 10 --config exercise
$ date; telegraf --test-wait 10 --config exercise; date
Fri Nov 6 22:30: 10 UTC 2020
2020-11-06T22:30:10Z I! Starting Telegraf 1.16.1
> mem,host=influxdays
active=155013120i,available=728371200i,available_percent=71.8394867713538
9,buffered=2117632i,cached=249663488i,commit_limit=506941440i,committed_a
s=403222528i,dirty=16384i,free=632045568i,high_free=0i,high_total=0i,huge
_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=14044
7744i,low_free=0i,low_total=0i,mapped=39026688i,page_tables=4534272i,shar
ed=13058048i,slab=46686208i,sreclaimable=23306240i,sunreclaim=23379968i,s
wap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=130060288
i,used_percent=12.827888224101224,vmalloc_chunk=35184362651648i,vmalloc_t
otal=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0
i 1604701810000000000
Fri Nov 6 22:30: 20 UTC 2020

53. © 2020 InfluxData. All rights reserved.
53
Write once on start up
$ telegraf --once --config exercise
$ date; telegraf --once --config exercise; date
Fri Nov 6 22:31:51 UTC 2020
2020-11-06T22:31:51Z I! Starting Telegraf 1.16.1
2020-11-06T22:31:51Z W! [outputs.influxdb] When writing to
[http://localhost:8086]: database "telegraf" creation failed: Post
"http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:31:51Z I! [agent] Hang on, flushing any cached metrics before
shutdown
2020-11-06T22:31:51Z E! [outputs.influxdb] When writing to
[http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp
[::1]:8086: connect: connection refused
2020-11-06T22:31:51Z E! [agent] Error writing to outputs.influxdb: could not write
any address
2020-11-06T22:31:51Z E! [telegraf] Error running agent: output plugins unable to
send 1 metrics
Fri Nov 6 22:31:51 UTC 2020

54. © 2020 InfluxData. All rights reserved.
54
Mem

55. © 2020 InfluxData. All rights reserved.
55
Exercise 2
Setup for Debugging

56. © 2020 InfluxData. All rights reserved.
56
Telegraf Configuration File
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
[[inputs.mem]]
[[outputs.influxdb]]
$ telegraf --once --config exercise
2020-11-06T22:34:07Z I! Starting Telegraf 1.16.1
2020-11-06T22:34:07Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database
"telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect:
connection refused
2020-11-06T22:34:07Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-11-06T22:34:07Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post
"http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:34:07Z E! [agent] Error writing to outputs.influxdb: could not write any
address
2020-11-06T22:34:07Z E! [telegraf] Error running agent: output plugins unable to send 1
metrics

57. © 2020 InfluxData. All rights reserved.
57
$ telegraf --debug --once --config exercise
2020-11-06T22:34:42Z I! Starting Telegraf 1.16.1
2020-11-06T22:34:42Z D! [agent] Initializing plugins
2020-11-06T22:34:42Z D! [agent] Connecting outputs
2020-11-06T22:34:42Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:34:42Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database
"telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect:
connection refused
2020-11-06T22:34:42Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:34:42Z D! [agent] Starting service inputs
2020-11-06T22:34:42Z D! [agent] Stopping service inputs
2020-11-06T22:34:42Z D! [agent] Input channel closed
2020-11-06T22:34:42Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-11-06T22:34:42Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post
"http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:34:42Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics
2020-11-06T22:34:42Z E! [agent] Error writing to outputs.influxdb: could not write any
address
2020-11-06T22:34:42Z D! [agent] Stopped Successfully
2020-11-06T22:34:42Z E! [telegraf] Error running agent: output plugins unable to send 1
metrics

58. © 2020 InfluxData. All rights reserved.
58
telegraf --once --config exercise
2020-11-06T22:35:58Z I! Starting Telegraf 1.16.1
2020-11-06T22:35:58Z D! [agent] Initializing plugins
2020-11-06T22:35:58Z D! [agent] Connecting outputs
2020-11-06T22:35:58Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:35:58Z W! [outputs.influxdb] When writing to
[http://localhost:8086]: database "telegraf" creation failed: Post
"http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:35:58Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:35:58Z D! [agent] Starting service inputs
2020-11-06T22:35:58Z D! [agent] Stopping service inputs
2020-11-06T22:35:58Z D! [agent] Input channel closed
2020-11-06T22:35:58Z I! [agent] Hang on, flushing any cached metrics before
shutdown
2020-11-06T22:35:58Z E! [outputs.influxdb] When writing to
[http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial
tcp [::1]:8086: connect: connection refused
2020-11-06T22:35:58Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics
2020-11-06T22:35:58Z E! [agent] Error writing to outputs.influxdb: could not
write any address
2020-11-06T22:35:58Z D! [agent] Stopped Successfully
2020-11-06T22:35:58Z E! [telegraf] Error running agent: output plugins unable to
send 1 metrics
Updated Telegraf Configuration File
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.influxdb]]

59. © 2020 InfluxData. All rights reserved.
59
Mem
Stdout

60. © 2020 InfluxData. All rights reserved.
60
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.influxdb]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.file]]
$ telegraf --once --config exercise
Output on next slide...

61. © 2020 InfluxData. All rights reserved.
61
$ telegraf --once --config exercise
2020-11-06T22:37:05Z I! Starting Telegraf 1.16.1
2020-11-06T22:37:05Z D! [agent] Initializing plugins
2020-11-06T22:37:05Z D! [agent] Connecting outputs
2020-11-06T22:37:05Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:37:05Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:37:05Z D! [agent] Starting service inputs
2020-11-06T22:37:05Z D! [agent] Stopping service inputs
2020-11-06T22:37:05Z D! [agent] Input channel closed
2020-11-06T22:37:05Z I! [agent] Hang on, flushing any cached metrics before shutdown
mem,host=influxdays
sreclaimable=23232512i,page_tables=4255744i,shared=13058048i,swap_cached=0i,vmalloc_total=351843720878
08i,write_back_tmp=0i,dirty=20480i,low_free=0i,huge_pages_free=0i,committed_as=403091456i,free=6308044
80i,sunreclaim=23289856i,swap_free=0i,cached=250294272i,huge_pages_total=0i,write_back=0i,total=101388
6976i,active=156626944i,slab=46522368i,vmalloc_chunk=35184362651648i,vmalloc_used=6332416i,used=130670
592i,inactive=140836864i,available_percent=71.78292819889226,buffered=2117632i,available=727797760i,us
ed_percent=12.88808270479253,high_total=0i,huge_page_size=2097152i,low_total=0i,mapped=40583168i,swap_
total=0i,commit_limit=506941440i,high_free=0i 1604702225000000000
2020-11-06T22:37:05Z D! [outputs.file] Wrote batch of 1 metrics in 92.731µs
2020-11-06T22:37:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-11-06T22:37:05Z D! [agent] Stopped Successfully

62. © 2020 InfluxData. All rights reserved.
62
Exercise 3
Enabling a plugin

63. © 2020 InfluxData. All rights reserved.
63
Mem
Net
Stdout

64. © 2020 InfluxData. All rights reserved.
64
Start
[global_tags]
[agent]
interval = "2s"
flush_interval =
"2s"
[[inputs.mem]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval =
"2s"
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]

65. © 2020 InfluxData. All rights reserved.
65
Validate using --test
$ telegraf --test --config exercise
$ telegraf --test --config exercise
2020-11-06T22:38:39Z I! Starting Telegraf 1.16.1
2020-11-06T22:38:39Z D! [agent] Initializing plugins
2020-11-06T22:38:39Z D! [agent] Starting service inputs
> net,host=influxdays,interface=ens5
bytes_recv=1708843i,bytes_sent=2007423i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=7752i,packets_sent=5597i
1604702319000000000
> net,host=influxdays,interface=all
icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_i
nerrors=0i,icmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=
0i,icmp_intimestamps=0i,icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0
i,icmp_outerrors=0i,icmp_outmsgs=2i,icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_ou
ttimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagra
ms=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_inaddrerrors=0i,ip_indelivers=7981i,ip_indiscards=0i,ip_inhdrerrors=0i,
ip_inreceives=7986i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnoroutes=16i,ip_outrequests=5516i,ip_reasmfails=0i,ip_reasmoks
=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tcp_currestab=4i,tcp_estabresets=4i,tcp_incsum
errors=5i,tcp_inerrs=5i,tcp_insegs=7733i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5563i,tcp_passiveopens=68i,tcp_retranssegs=
43i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=312i,udp_inerrors=0i,udp_noports=
2i,udp_outdatagrams=315i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_inerrors
=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702319000000000
2020-11-06T22:38:39Z D! [agent] Stopping service inputs
2020-11-06T22:38:39Z D! [agent] Input channel closed
2020-11-06T22:38:39Z D! [agent] Stopped Successfully
> mem,host=influxdays
active=159830016i,available=723709952i,available_percent=71.37974637520149,buffered=2117632i,cached=250327040i,commit_limit=5069
41440i,committed_as=407773184i,dirty=8192i,free=626688000i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i
,huge_pages_total=0i,inactive=140804096i,low_free=0i,low_total=0i,mapped=40611840i,page_tables=4898816i,shared=13058048i,slab=46
534656i,sreclaimable=23240704i,sunreclaim=23293952i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=134754304i,
used_percent=13.290860538680004,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,
write_back_tmp=0i 1604702319000000000

66. © 2020 InfluxData. All rights reserved.
66
$ telegraf --config exercise
2020-11-06T22:39:26Z I! Starting Telegraf 1.16.1
2020-11-06T22:39:26Z I! Loaded inputs: mem net
2020-11-06T22:39:26Z I! Loaded aggregators:
2020-11-06T22:39:26Z I! Loaded processors:
2020-11-06T22:39:26Z I! Loaded outputs: file
2020-11-06T22:39:26Z I! Tags enabled: host=influxdays
2020-11-06T22:39:26Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s
2020-11-06T22:39:26Z D! [agent] Initializing plugins
2020-11-06T22:39:26Z D! [agent] Connecting outputs
2020-11-06T22:39:26Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:39:26Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:39:26Z D! [agent] Starting service inputs
mem,host=influxdays
free=629272576i,huge_pages_free=0i,low_total=0i,commit_limit=506941440i,high_free=0i,huge_pages_total=0i,vmalloc_chunk=35184362651648i,used=132153344i,high_tot
al=0i,shared=13062144i,sunreclaim=23162880i,vmalloc_total=35184372087808i,write_back_tmp=0i,inactive=140779520i,used_percent=13.034327013586177,swap_cached
=0i,vmalloc_used=6332416i,write_back=0i,buffered=2117632i,cached=250343424i,committed_as=403091456i,swap_total=0i,total=1013886976i,available=726310912i,availa
ble_percent=71.63627990029532,active=158572544i,dirty=4096i,page_tables=4214784i,slab=46395392i,huge_page_size=2097152i,low_free=0i,mapped=40656896i,sreclai
mable=23232512i,swap_free=0i 1604702368000000000
net,host=influxdays,interface=ens5 drop_in=0i,drop_out=0i,bytes_sent=2037575i,bytes_recv=1723821i,packets_sent=5696i,packets_recv=7891i,err_in=0i,err_out=0i
1604702368000000000
net,host=influxdays,interface=all
udplite_rcvbuferrors=0i,ip_outdiscards=0i,ip_inaddrerrors=0i,icmp_inechoreps=0i,icmp_outtimestampreps=0i,icmp_outerrors=0i,icmp_outechoreps=0i,icmp_outaddrmaskreps
=0i,icmpmsg_outtype3=2i,udplite_noports=0i,udplite_incsumerrors=0i,udp_noports=2i,icmp_incsumerrors=0i,icmp_outechos=0i,icmp_outaddrmasks=0i,icmp_inechos=0i,icmp
_outparmprobs=0i,ip_inhdrerrors=0i,ip_outnoroutes=16i,ip_defaultttl=64i,icmp_outdestunreachs=2i,udp_indatagrams=316i,tcp_currestab=1i,ip_fragfails=0i,ip_reasmtimeout=0
i,ip_reasmoks=0i,icmp_outredirects=0i,tcp_retranssegs=45i,tcp_passiveopens=69i,tcp_maxconn=-1i,tcp_inerrs=5i,ip_inreceives=8124i,ip_forwarding=2i,icmp_inmsgs=2i,icmp
_indestunreachs=2i,icmpmsg_intype3=2i,ip_indiscards=0i,icmp_intimeexcds=0i,udp_incsumerrors=0i,tcp_outsegs=5655i,ip_forwdatagrams=0i,ip_fragcreates=0i,icmp_inredir
ects=0i,icmp_outsrcquenchs=0i,icmp_inaddrmasks=0i,ip_indelivers=8119i,icmp_inparmprobs=0i,tcp_insegs=7867i,tcp_attemptfails=72i,udp_inerrors=0i,tcp_estabresets=4i,tc
p_outrsts=68i,udp_rcvbuferrors=0i,ip_reasmfails=0i,ip_fragoks=0i,ip_reasmreqds=0i,icmp_outtimestamps=0i,icmp_intimestampreps=0i,icmp_inerrors=0i,tcp_incsumerrors=5i,
udplite_inerrors=0i,ip_outrequests=5610i,icmp_insrcquenchs=0i,icmp_outmsgs=2i,icmp_outtimeexcds=0i,udp_sndbuferrors=0i,tcp_rtoalgorithm=1i,udp_outdatagrams=319i,u
dplite_outdatagrams=0i,tcp_rtomin=200i,icmp_inaddrmaskreps=0i,udplite_sndbuferrors=0i,ip_inunknownprotos=0i,icmp_intimestamps=0i,tcp_rtomax=120000i,tcp_activeopen
s=142i,udplite_indatagrams=0i 1604702368000000000
2020-11-06T22:39:28Z D! [outputs.file] Wrote batch of 3 metrics in 139.162µs
2020-11-06T22:39:28Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Validate without --test

67. © 2020 InfluxData. All rights reserved.
67
Filtering

68. © 2021 InfluxData. All rights reserved. 68
© 2019 InfluxData. All rights reserved.
68
What Is An InfluxDB Series?
First, some definitions:
• Measurement: namespace for categorically alike metrics (~Table)
• Tag: key-value pair used as metadata for measurable object
– host=prod001 or app=cart or customer_id=00012
• Field: key-value pair representing a metric that will append new values
over time
– cpu_usage=35.5 or errors=42 or temp=65.0
• TagSet: unique combination of Tag key-value pairs
• FieldSet: unique combination of Field key-value pairs

69. © 2021 InfluxData. All rights reserved. 69
© 2019 InfluxData. All rights reserved.
69
What Is An InfluxDB Series?
Line/Record: <Measurement>,<TagSet> <FieldSet> <timestamp>
→ cpu,host=prod001,region=us-west usage_user=35,usage_system=15 t0
Point: <Measurement>,<TagSet> <FieldKey> <timestamp>
→ cpu,host=prod001,region=us-west usage_user=35 t0
Series: <Measurement>,<TagSet> <FieldKey>
→ cpu,host=prod001,region=us-west usage_user

70. © 2021 InfluxData. All rights reserved. 70
© 2019 InfluxData. All rights reserved.
70
Series Visualized (tree)
Cardinality is total number of “leaf nodes”

71. © 2020 InfluxData. All rights reserved.
71
Filtering metrics
• Happens at plugin level
• Specify which metrics and which parts of metrics the given plugin sees or doesn’t see
• Given above, filtering is often used for “routing”
• By Measurement:
• namepass: passes only records with specified Measurement name
• namedrop: drops only records with specified Measurement name
• By Field:
• fieldpass: pass specified Field
• fielddrop: drop specified Field
• By Tag (slightly different implementation):
• tagpass or tagdrop: pass/drop whole record containing matched Tag
• taginclude or tagexclude: keep/drop matching Tag (key) in record

72. © 2020 InfluxData. All rights reserved.
72
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA free=85.0,cached=12.5,buffered=3.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA free=85.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA cached=85.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA buffered=85.0 <timestamp>
namepass = “mem”

73. © 2020 InfluxData. All rights reserved.
73
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0
,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp>
fieldpass = “*_system”

74. © 2020 InfluxData. All rights reserved.
74
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0
,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp>
fielddrop = “*_softirq”

75. © 2020 InfluxData. All rights reserved.
75
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp>
tagpass: host = [“hostA”]

76. © 2020 InfluxData. All rights reserved.
76
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp>
tagdrop: host = [“hostB”]

77. © 2020 InfluxData. All rights reserved.
77
How to think of this
Telegraf metric:
cpu,region=us-west-1,container=containerA,host=hostA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp>
taginclude:
host = [“host”]

78. © 2020 InfluxData. All rights reserved.
78
How to think of this
Telegraf metric:
cpu,region=us-west-1,container=containerA,host=hostA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp>
tagexclude:
host = [“host”]

79. © 2020 InfluxData. All rights reserved.
79
Exercise 4
Filtering

80. © 2020 InfluxData. All rights reserved.
80
Sample Data
mem,host=influxdays
available=735477760i,inactive=167567360i,page_tables=4345856i,sreclaimable=22962176i,committed_as=402309120i,slab=46772224i,vmalloc_total=35184
372087808i,cached=243228672i,high_free=0i,huge_page_size=2097152i,huge_pages_total=0i,write_back=0i,total=1013886976i,dirty=0i,free=645414912i,lo
w_total=0i,sunreclaim=23810048i,swap_cached=0i,huge_pages_free=0i,low_free=0i,shared=13058048i,swap_total=0i,used=123125760i,used_percent=12.1
43933487118785,buffered=2117632i,vmalloc_used=6332416i,available_percent=72.54040908007482,high_total=0i,vmalloc_chunk=35184362651648i,active
=113328128i,commit_limit=506941440i,mapped=40271872i,swap_free=0i,write_back_tmp=0i 1602630666000000000
net,host=influxdays,interface=ens5
packets_recv=11760i,err_in=0i,err_out=0i,drop_in=0i,drop_out=0i,bytes_sent=1730127i,bytes_recv=1082190i,packets_sent=10073i 1602630666000000000
net,host=influxdays,interface=all
icmp_outechos=0i,icmp_inechoreps=0i,tcp_rtoalgorithm=1i,tcp_insegs=9422i,ip_inunknownprotos=0i,ip_indiscards=0i,ip_reasmreqds=0i,icmp_outdestunreac
hs=2i,icmpmsg_outtype3=2i,icmp_outaddrmasks=0i,icmp_outredirects=0i,icmp_indestunreachs=3i,icmp_incsumerrors=0i,udp_incsumerrors=0i,ip_inhdrerrors
=0i,ip_reasmoks=0i,tcp_maxconn=-1i,udp_sndbuferrors=0i,tcp_incsumerrors=0i,tcp_estabresets=21i,udplite_rcvbuferrors=0i,ip_reasmtimeout=0i,icmp_outerr
ors=0i,icmp_intimeexcds=0i,udplite_sndbuferrors=0i,icmpmsg_intype3=3i,tcp_activeopens=188i,tcp_rtomax=120000i,udp_outdatagrams=991i,udplite_incsum
errors=0i,ip_fragcreates=0i,ip_fragoks=0i,icmp_inerrors=0i,tcp_rtomin=200i,udp_rcvbuferrors=0i,udp_noports=2i,ip_inaddrerrors=0i,ip_reasmfails=0i,ip_forwar
ding=2i,icmp_outechoreps=0i,icmp_intimestamps=0i,icmp_outmsgs=2i,tcp_outrsts=141i,udp_indatagrams=932i,ip_inreceives=10233i,ip_outdiscards=0i,icmp
_intimestampreps=0i,icmp_outtimestamps=0i,icmp_outsrcquenchs=0i,tcp_passiveopens=269i,tcp_currestab=1i,udplite_indatagrams=0i,ip_outnoroutes=16i,ic
mp_outtimestampreps=0i,icmp_outparmprobs=0i,icmp_inaddrmasks=0i,icmp_outaddrmaskreps=0i,icmp_inparmprobs=0i,ip_defaultttl=64i,ip_outrequests=836
7i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_outtimeexcds=0i,icmp_inmsgs=3i,icmp_inaddrmaskreps=0i,ip_forwdatagrams=0i,ip_fragfails=0i,icmp_inec
hos=0i,tcp_inerrs=0i,tcp_attemptfails=175i,tcp_outsegs=7517i,udplite_inerrors=0i,udplite_noports=0i,ip_indelivers=10227i,tcp_retranssegs=118i,udp_inerrors
=0i,udplite_outdatagrams=0i 1602630666000000000

81. © 2020 InfluxData. All rights reserved.
81
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
Pass fields
ending in “percent”
exact named “total”
[[inputs.net]]
Pass fields Beginning with “bytes”
Pass metrics where interface begins
with “en”
[[outputs.file]]

82. © 2020 InfluxData. All rights reserved.
82
Exercise 4
Filtering
Solution

83. © 2020 InfluxData. All rights reserved.
83
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
...

84. © 2020 InfluxData. All rights reserved.
84
Correct Placement
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
...

85. © 2020 InfluxData. All rights reserved.
85
Before & After
$ telegraf --test --config exercise
> net,host=influxdays,interface= ens5
bytes_recv=1739020i,bytes_sent=2088482i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=8042i,packets_sent=5843i
1604702475000000000
> mem,host=influxdays
active=154382336i,available=730259456i,available_percent=72.02572607067398,buffered=2117632i,cached=250368000i,commit_limit=506941440i,commi
tted_as=403091456i,dirty=20480i,free=633192448i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,in
active=140689408i,low_free=0i,low_total=0i,mapped=40558592i,page_tables=4190208i,shared=13058048i,slab=46452736i,sreclaimable=23232512i,sunr
eclaim=23220224i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=128208896i,used_percent=12.645284833010814,vmalloc_chunk=3
5184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i 1604702475000000000
2020-11-06T22:41:14Z D! [agent] Stopping service inputs
2020-11-06T22:41:14Z D! [agent] Input channel closed
2020-11-06T22:41:14Z D! [agent] Stopped Successfully
> net,host=influxdays,interface= all
icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_inerrors=0i,i
cmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=0i,icmp_intimestamps=0i,
icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0i,icmp_outerrors=0i,icmp_outmsgs=2i,
icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_outtimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_
intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagrams=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_ina
ddrerrors=0i,ip_indelivers=8268i,ip_indiscards=0i,ip_inhdrerrors=0i,ip_inreceives=8273i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnorout
es=16i,ip_outrequests=5747i,ip_reasmfails=0i,ip_reasmoks=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tc
p_currestab=1i,tcp_estabresets=4i,tcp_incsumerrors=5i,tcp_inerrs=5i,tcp_insegs=8010i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5792i,tcp_p
assiveopens=74i,tcp_retranssegs=47i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=322i,udp_iner
rors=0i,udp_noports=2i,udp_outdatagrams=325i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_
inerrors=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702475000000000
….
After filtering...
….
> mem,host=influxdays available_percent=71.50255927540388,total=1013886976i,used_percent=13.168451628280902 1604702498000000000
> net,host=influxdays,interface= ens5 bytes_recv=1745105i,bytes_sent=2111895i 1604702498000000000

86. © 2020 InfluxData. All rights reserved.
86
Placement Matters,
Indentation Does Not
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
[inputs.net.tagpass]
interface = ["en*"]
fieldpass = ["bytes*"]
[[outputs.file]]
> net,host=influxdays,interface=ens5
bytes_recv=2082848i,bytes_sent=3148789i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=21247
i,packets_sent=19323i 1604702498000000000

87. © 2020 InfluxData. All rights reserved.
87
Exercise 5
Dual Write

88. © 2020 InfluxData. All rights reserved.
88
Mem
Net net
/tmp/net
m
em
/tmp/mem

89. © 2020 InfluxData. All rights reserved.
89
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
Pass the “mem” data to file /tmp/mem
[[outputs.file]]
Pass the “net” data to file /tmp/net

90. © 2020 InfluxData. All rights reserved.
90
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]

91. © 2020 InfluxData. All rights reserved.
91
Exercise 6
Listener

92. © 2020 InfluxData. All rights reserved.
92
Mem
Net
ne
t
/tmp/net.txt
m
em
/tmp/mem.txt
:8086
Add

93. © 2020 InfluxData. All rights reserved.
93
telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
Set the listener port to ":8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]

94. © 2020 InfluxData. All rights reserved.
94
telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
service_address = ": 8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]

95. © 2020 InfluxData. All rights reserved.
95
Launch Telegraf Listener as background job
$ telegraf --config telegraf-listener.conf &
[1] 2219
[centos@influxdays ~]$ 2020-11-06T22:44:22Z I! Starting Telegraf
1.16.1
[centos@influxdays ~]$
[centos@influxdays ~]$
[centos@influxdays ~]$

96. © 2020 InfluxData. All rights reserved.
96
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]
[[outputs.influxdb]]

97. © 2020 InfluxData. All rights reserved.
97
Launch Telegraf
$ telegraf -config exercise
2020-11-06T22:45:09Z I! Starting Telegraf 1.16.1
2020-11-06T22:45:09Z I! Loaded inputs: mem net
2020-11-06T22:45:09Z I! Loaded aggregators:
2020-11-06T22:45:09Z I! Loaded processors:
2020-11-06T22:45:09Z I! Loaded outputs: file (2x) influxdb
2020-11-06T22:45:09Z I! Tags enabled: host=influxdays
2020-11-06T22:45:09Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s
2020-11-06T22:45:09Z D! [agent] Initializing plugins
2020-11-06T22:45:09Z D! [agent] Connecting outputs
2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:45:09Z D! [agent] Starting service inputs
2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 101.011µs
2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 23.33µs
2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-11-06T22:45:11Z D! [outputs.influxdb] Wrote batch of 2 metrics in 1.205233ms
2020-11-06T22:45:11Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766
1604702710000000000
net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000

98. © 2020 InfluxData. All rights reserved.
98
Inspect the filesystem
$ cat /tmp/mem
mem,host=influxdays
used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766
1604702710000000000
$ cat /tmp/net
net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000

99. © 2020 InfluxData. All rights reserved.
99
telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
service_address = ":8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]

100. © 2020 InfluxData. All rights reserved.
100
Exercise 7
Starlark

101. © 2020 InfluxData. All rights reserved.
101
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[processors.starlark]]
source = '''
def apply(metric):
for k, v in metric.fields.items():
if type(v) == "float" or type(v) == "int":
metric.fields[k] = v * 1000
return metric
'''
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
..
..
[[outputs.file]]

102. © 2020 InfluxData. All rights reserved.
102
Launch Telegraf
Without Starlark processor
> net,host=influxdays,interface=ens5 bytes_recv=9566941i,bytes_sent=23302741i 1604939494000000000
After multiplying by 1000
> net,host=influxdays,interface=ens5 bytes_recv=9555668000i,bytes_sent=23274132000i 1604939466000000000

103. © 2020 InfluxData. All rights reserved.
103
Exercise 8
JSON

104. © 2020 InfluxData. All rights reserved.
104
Mem
Net net
/tmp/net.txt
m
em
/tmp/mem.txt
HTTP

105. © 2020 InfluxData. All rights reserved.
105
Add
[[inputs.http]]
urls = ["https://api.mocki.io/v1/29dfca5b"]
data_format = “json”
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]

106. © 2020 InfluxData. All rights reserved.
106
Add
[[inputs.http]]
urls = ["https://api.mocki.io/v1/29dfca5b"]
data_format = “json”
name_override = “apps”
json_query = “apps”
tag_keys = [“name”,”id”]
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]

107. © 2021 InfluxData. All rights reserved. 107
Telegraf Resources
Blogs
https:/
/www.influxdata.com/blog/using-a-telegraf-gateway/
https:/
/www.influxdata.com/blog/telegraf-go-collection-agent/
Docs
https:/
/docs.influxdata.com/telegraf/latest
Prebuilt dashboards
https:/
/docs.influxdata.com/platform/monitoring/influxdata-platform/monitoring-dashb
oards/

108. Questions?

109. Thank you
See you tomorrow at InfluxDays EMEA 2021!

110. © 2021 InfluxData. All rights reserved. 110
InfluxDays EMEA 2021
See you tomorrow: 18-19 May 2021