Taming the Tiger: Tips and Tricks for Using Telegraf

InfluxData
InfluxDataInfluxData
© 2021 InfluxData. All rights reserved. 1
© 2020 InfluxData. All rights reserved.
1
Ignacio Van Droogenbroeck | Solutions Architect, InfluxData
Suleman Khan | Solutions Architect, InfluxData
May 2021
Agenda
■ Overview
■ Setup (if not completed)
■ Global configuration
■ Enabling plugins
■ Filtering data
Please join us in the InfluxDB Community Slack at
www.influxdata.com/slack.
To participate in the conversations,
join the #influxdays-virtual & #telegraf-training channels.
JOIN THE CONVERSATION
Workshop Logistics
• Cloud account free tier
• No CC
• Rate limits → Let’s avoid them
• Browser
• SSH client
• Have documentation open for exercise
• General config: https:/
/bit.ly/3t1r03K
© 2021 InfluxData. All rights reserved. 5
Telegraf
• Lightweight; written in Go
• Plug-in driven
• Optimized for writing to InfluxDB
• Formatting
• Retries
• Modifiable batch sizes and jitter
• Tag sorting
• Preprocessing
• Converting tags to fields, fields to tags
• Regex transformations
• Renaming measurements, tags
• Aggregations (mean, min, max, count, variance, stddev, etc.)
© 2021 InfluxData. All rights reserved. 6
Popular Inputs (Integrations)
Out-of-the-box Custom
Kubernetes (kubelet) HTTP/socket listener
Kube_inventory (apiserver) HTTP (formatted endpoints)
Kafka (producer/consumer) Prometheus (/metrics)
SNMP Exec
System (Linux,Windows,Unix) StatsD
Nginx
HAproxy
OPC, AMQP, Modbus & MQTT (IoT)
© 2021 InfluxData. All rights reserved. 7
Telegraf
CPU
Mem
Disk
Docker
Kubernetes
/metrics
Kafka
MySQL
Process
-transform
-decorate
-filter
Aggregate
-mean
-min,max
-count
-variance
-stddev
File
InfluxDB
Kafka
CloudWatch
CloudWatch
© 2021 InfluxData. All rights reserved. 8
© 2021 InfluxData. All rights reserved. 9
Telegraf Common Architecture
Messaging
Queue
Kafka
Rabbit
Active
NSQ
AWS Kinesis
Google PubSub
MQTT
© 2021 InfluxData. All rights reserved. 10
Telegraf Architecture
Message Queue
Kafka
Rabbit
Active
NSQ
AWS Kinesis
Google PubSub
MQTT
© 2021 InfluxData. All rights reserved. 11
Parsing
● JSON
● CSV
● Graphite
● CollectD
● Dropwizard
● Form URL-encoded
● Grok
© 2021 InfluxData. All rights reserved. 12
Telegraf
Latest version available at
https:/
/portal.influxdata.com/downloads/#telegraf
Create the configuration file
$ telegraf config > telegraf.conf
# ~7500 lines
OR
$ telegraf -sample-config -input-filter cpu:mem -output-filter influxdb > telegraf.conf
# 830 lines
© 2021 InfluxData. All rights reserved. 13
Starlark
Embedded Python dialect
– Not full std lib
– But most basic Python functionality supported
Current processors and aggregators + more
[[processors.starlark]]
namepass = ["mymeasurement"]
source = '''
def apply(metric):
I = metric.fields['current']
V = metric.fields['voltage']
metric.fields['power'] = I * V
return metric
'''
[[processors.starlark]]
source = '''
def apply(metric):
used = float(metric.fields['used'])
total = float(metric.fields['total'])
metric.fields['usage'] = (used / total) * 100
return metric
'''
Setting up InfluxDB Cloud
Sign up: https:/
/cloud2.influxdata.com/signup
© 2021 InfluxData. All rights reserved. 15
InfluxDB Cloud Setup
- Choose your Provider & Region
- Enter your company name
- Read & Tick the Service Agreement
© 2021 InfluxData. All rights reserved. 16
InfluxDB Cloud Setup
● Select “Free” account
© 2021 InfluxData. All rights reserved. 17
InfluxDB Cloud Setup
- Entering “Getting Started” page
© 2021 InfluxData. All rights reserved. 18
Demo Data
© 2021 InfluxData. All rights reserved. 19
Finding your Organization ID
1
2
© 2021 InfluxData. All rights reserved. 20
Token
2
3 4
1
© 2021 InfluxData. All rights reserved. 21
Find your host
Go to… https://bit.ly/3tGLuPy
1. Find your name
2. Copy from column E
3. Paste into console
4. Enter the password
© 2021 InfluxData. All rights reserved. 22
Login
Using an SSH client
$ ssh centos@[ip-address]
$ ssh centos@3.133.90.20
The authenticity of host '3.133.90.20 (3.133.90.20)' can't be established.
ECDSA key fingerprint is SHA256:ki5Hm1AQ4UyyOn29uvoexeNO/Mk/Z0qourjEn+EpHdk.
Are you sure you want to continue connecting (yes/no)? yes
Password:
InfluxDaysIsTheBestTrainingEver!!!!
© 2021 InfluxData. All rights reserved. 23
© 2021 InfluxData. All rights reserved. 24
Getting Setup
Install Telegraf
[centos@influxdays ~]$ sudo rpm -ivh
/home/centos/telegraf-1.18.2-1.x86_64.rpm
Preparing... #################################
[100%]
Updating / installing…
1:telegraf-1.18.2-1 #################################
[100%]
Created symlink from
/etc/systemd/system/multi-user.target.wants/telegraf.service to
/usr/lib/systemd/system/telegraf.service.
[centos@influxdays ~]$
© 2021 InfluxData. All rights reserved. 25
Create a Bucket
1
2
© 2020 InfluxData. All rights reserved.
26
3
4
© 2020 InfluxData. All rights reserved.
27
Create Telegraf Configuration
1
2
© 2021 InfluxData. All rights reserved. 28
Create Telegraf Configuration
1
2
3
4
5
© 2021 InfluxData. All rights reserved. 29
Test your Configuration (optional)
1 2
3
Paste in console
© 2021 InfluxData. All rights reserved. 30
Select All
Update configuration file
1
2
3
4
© 2021 InfluxData. All rights reserved. 31
VI Cheat Sheet
[centos@influxdays ~]$ sudo vi /etc/telegraf/telegraf.conf
gg # Go to first line; this may not do anything
dG # Delete all lines
i # Insert mode
<paste> # Paste the full config from InfluxDB Cloud
Escape # End Insert mode
:64 # Jump to line 64 to update Token
:wq # Save and exit
© 2021 InfluxData. All rights reserved. 32
Hardcode the Token
Be sure to retain the double quotes
1
2
3
Start the Service
[centos@influxdays ~]$ sudo systemctl
start telegraf
Check the logs
[centos@influxdays ~]$ $ journalctl
-u telegraf
-- Logs begin at Mon 2020-11-02 19:23:33 UTC, end at Mon 2020-11-02 19:55:58 UTC.
--
Nov 02 19:55:58 influxdays systemd[1]: Started The plugin-driven server agent for
reporting metrics into InfluxDB.
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Starting
Telegraf 1.16.1
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded inputs:
cpu disk diskio mem net processes swap system
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded
aggregators:
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded
processors:
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded
outputs: influxdb_v2
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Tags enabled:
host=influxdays
Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! [agent]
Config: Interval:10s, Quiet:false, Hostname:"influxdays", Flush Interval:10s
© 2021 InfluxData. All rights reserved. 34
Alternate Deployment
Launch Telegraf using the provided commands in
the Cloud Web UI
Open a new SSH console window to continue with
the exercises.
- Agent Configuration
- Enable Plugins
- Filtering
- Dual Writes
- Telegraf Gateway
- Debugging
Exercise
Exercise 1
Agent Configuration
© 2021 InfluxData. All rights reserved. 38
Exercise 1 : Agent Configuration
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
debug = false
quiet = false
logfile = ""
hostname = ""
omit_hostname = false
Objective: To understand Agent
Configuration and make changes to
collection and flush behaviour.
Exercise 2
Enable Input Plugin
© 2021 InfluxData. All rights reserved. 40
Exercise 2 : Enabling Input Plugin
Objective: To enable plugin into your current
telegraf configuration file and see the results in
your InfluxDB Cloud platform.
[global_tags]
...
[agent]
...
[[outputs.influxdb_v2]]
...
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
[[inputs.mem]]
[[inputs.net]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.http_response]]
urls = ["https://www.influxdata.com/"]
Exercise 3
Filtering Data
© 2021 InfluxData. All rights reserved. 42
Exercise 3 : Filtering Data
Objective: To filter data streaming from “mem”
plugin to only pass “field” data that has exact
description called “available_percent” and
“used_percent”.
[global_tags]
...
[agent]
...
[[outputs.influxdb_v2]]
...
...
[[inputs.mem]]
fieldpass = [“
available_percent”, “
used_percent”]
...
Exercise 4
Dual Writes
© 2021 InfluxData. All rights reserved. 44
Exercise 4 : Dual Write
Objective: Write data to two different
outputs. One output to write to your
InfluxDB Cloud and the other one will
write to a file in your local system.
Copy the output plugin configuration
that we post in the Slack channel.
[global_tags]
...
[agent]
...
[[outputs.influxdb_v2]]
urls = ["https://eu-central-1-1.aws.cloud2.influxdata.com"]
## Token for authentication.
token = "$INFLUX_TOKEN"
...
[[outputs.file]]
files = ["/tmp/metrics.out"]
data_format = "influx"
...
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
...
Exercise 5
Telegraf Gateway
© 2021 InfluxData. All rights reserved. 46
Exercise 5 : Telegraf Gateway
Objective: Instead of send the data
directly to InfluxDB Cloud, we going to
send to a Telegraf Gateway. Add a
new output in your telegraf
configuration to point to the Gateway.
Copy the output code that we posted
in the Slack channel.
[global_tags]
...
[agent]
...
[[outputs.influxdb]]
urls = ["http://3.135.207.41:8086"]
database_tag = "telegraf"
exclude_database_tag = true
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
...
Exercise 6
Debugging
© 2021 InfluxData. All rights reserved. 48
Exercise 6 : Debugging
Objective: Let’s see how can we
debug what’s going on with our
Telegraf Instance.
Run journalctl -u telegraf
Enable the Debug parameter, change,
save and restart Telegraf.
Run journalctl -u telegraf
[global_tags]
...
[agent]
debug = true
...
[[outputs.influxdb]]
...
[[inputs.cpu]]
[[inputs.disk]]
[[input.diskio]]
...
© 2020 InfluxData. All rights reserved.
49
Telegraf Configuration
/home/centos/lab/exercise
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
[[inputs.mem]]
[[outputs.influxdb]]
© 2020 InfluxData. All rights reserved.
50
Exercise 1 - Lay of the land
Inspect the output of a given configuration
$ telegraf --test --config exercise
Wait up to this many seconds for service
$ telegraf --test-wait 10 --config exercise
Gather metrics once, write them, and exit
$ telegraf --once --config exercise
© 2020 InfluxData. All rights reserved.
51
Inspect the output of a given configuration
$ telegraf --test --config exercise
[centos@influxdays lab]$ date; telegraf --test --config exercise ; date
Fri Nov 6 22:29:46 UTC 2020
2020-11-06T22:29:46Z I! Starting Telegraf 1.16.1
> mem,host=influxdays
active=151506944i,available=732938240i,available_percent=72.28993540203045,buffered=211
7632i,cached=249655296i,commit_limit=506941440i,committed_as=399839232i,dirty=16384i,fr
ee=636620800i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,hug
e_pages_total=0i,inactive=140464128i,low_free=0i,low_total=0i,mapped=38670336i,page_tab
les=4157440i,shared=13058048i,slab=46571520i,sreclaimable=23306240i,sunreclaim=23265280
i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=125493248i,used_perc
ent=12.377439593424661,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmal
loc_used=6332416i,write_back=0i,write_back_tmp=0i 1604701786000000000
Fri Nov 6 22:29:46 UTC 2020
© 2020 InfluxData. All rights reserved.
52
Wait up to this many seconds for service
$ telegraf --test-wait 10 --config exercise
$ date; telegraf --test-wait 10 --config exercise; date
Fri Nov 6 22:30: 10 UTC 2020
2020-11-06T22:30:10Z I! Starting Telegraf 1.16.1
> mem,host=influxdays
active=155013120i,available=728371200i,available_percent=71.8394867713538
9,buffered=2117632i,cached=249663488i,commit_limit=506941440i,committed_a
s=403222528i,dirty=16384i,free=632045568i,high_free=0i,high_total=0i,huge
_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=14044
7744i,low_free=0i,low_total=0i,mapped=39026688i,page_tables=4534272i,shar
ed=13058048i,slab=46686208i,sreclaimable=23306240i,sunreclaim=23379968i,s
wap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=130060288
i,used_percent=12.827888224101224,vmalloc_chunk=35184362651648i,vmalloc_t
otal=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0
i 1604701810000000000
Fri Nov 6 22:30: 20 UTC 2020
© 2020 InfluxData. All rights reserved.
53
Write once on start up
$ telegraf --once --config exercise
$ date; telegraf --once --config exercise; date
Fri Nov 6 22:31:51 UTC 2020
2020-11-06T22:31:51Z I! Starting Telegraf 1.16.1
2020-11-06T22:31:51Z W! [outputs.influxdb] When writing to
[http://localhost:8086]: database "telegraf" creation failed: Post
"http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:31:51Z I! [agent] Hang on, flushing any cached metrics before
shutdown
2020-11-06T22:31:51Z E! [outputs.influxdb] When writing to
[http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp
[::1]:8086: connect: connection refused
2020-11-06T22:31:51Z E! [agent] Error writing to outputs.influxdb: could not write
any address
2020-11-06T22:31:51Z E! [telegraf] Error running agent: output plugins unable to
send 1 metrics
Fri Nov 6 22:31:51 UTC 2020
© 2020 InfluxData. All rights reserved.
54
Mem
© 2020 InfluxData. All rights reserved.
55
Exercise 2
Setup for Debugging
© 2020 InfluxData. All rights reserved.
56
Telegraf Configuration File
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
[[inputs.mem]]
[[outputs.influxdb]]
$ telegraf --once --config exercise
2020-11-06T22:34:07Z I! Starting Telegraf 1.16.1
2020-11-06T22:34:07Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database
"telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect:
connection refused
2020-11-06T22:34:07Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-11-06T22:34:07Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post
"http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:34:07Z E! [agent] Error writing to outputs.influxdb: could not write any
address
2020-11-06T22:34:07Z E! [telegraf] Error running agent: output plugins unable to send 1
metrics
© 2020 InfluxData. All rights reserved.
57
$ telegraf --debug --once --config exercise
2020-11-06T22:34:42Z I! Starting Telegraf 1.16.1
2020-11-06T22:34:42Z D! [agent] Initializing plugins
2020-11-06T22:34:42Z D! [agent] Connecting outputs
2020-11-06T22:34:42Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:34:42Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database
"telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect:
connection refused
2020-11-06T22:34:42Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:34:42Z D! [agent] Starting service inputs
2020-11-06T22:34:42Z D! [agent] Stopping service inputs
2020-11-06T22:34:42Z D! [agent] Input channel closed
2020-11-06T22:34:42Z I! [agent] Hang on, flushing any cached metrics before shutdown
2020-11-06T22:34:42Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post
"http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:34:42Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics
2020-11-06T22:34:42Z E! [agent] Error writing to outputs.influxdb: could not write any
address
2020-11-06T22:34:42Z D! [agent] Stopped Successfully
2020-11-06T22:34:42Z E! [telegraf] Error running agent: output plugins unable to send 1
metrics
© 2020 InfluxData. All rights reserved.
58
telegraf --once --config exercise
2020-11-06T22:35:58Z I! Starting Telegraf 1.16.1
2020-11-06T22:35:58Z D! [agent] Initializing plugins
2020-11-06T22:35:58Z D! [agent] Connecting outputs
2020-11-06T22:35:58Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:35:58Z W! [outputs.influxdb] When writing to
[http://localhost:8086]: database "telegraf" creation failed: Post
"http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused
2020-11-06T22:35:58Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:35:58Z D! [agent] Starting service inputs
2020-11-06T22:35:58Z D! [agent] Stopping service inputs
2020-11-06T22:35:58Z D! [agent] Input channel closed
2020-11-06T22:35:58Z I! [agent] Hang on, flushing any cached metrics before
shutdown
2020-11-06T22:35:58Z E! [outputs.influxdb] When writing to
[http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial
tcp [::1]:8086: connect: connection refused
2020-11-06T22:35:58Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics
2020-11-06T22:35:58Z E! [agent] Error writing to outputs.influxdb: could not
write any address
2020-11-06T22:35:58Z D! [agent] Stopped Successfully
2020-11-06T22:35:58Z E! [telegraf] Error running agent: output plugins unable to
send 1 metrics
Updated Telegraf Configuration File
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.influxdb]]
© 2020 InfluxData. All rights reserved.
59
Mem
Stdout
© 2020 InfluxData. All rights reserved.
60
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.influxdb]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[outputs.file]]
$ telegraf --once --config exercise
Output on next slide...
© 2020 InfluxData. All rights reserved.
61
$ telegraf --once --config exercise
2020-11-06T22:37:05Z I! Starting Telegraf 1.16.1
2020-11-06T22:37:05Z D! [agent] Initializing plugins
2020-11-06T22:37:05Z D! [agent] Connecting outputs
2020-11-06T22:37:05Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:37:05Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:37:05Z D! [agent] Starting service inputs
2020-11-06T22:37:05Z D! [agent] Stopping service inputs
2020-11-06T22:37:05Z D! [agent] Input channel closed
2020-11-06T22:37:05Z I! [agent] Hang on, flushing any cached metrics before shutdown
mem,host=influxdays
sreclaimable=23232512i,page_tables=4255744i,shared=13058048i,swap_cached=0i,vmalloc_total=351843720878
08i,write_back_tmp=0i,dirty=20480i,low_free=0i,huge_pages_free=0i,committed_as=403091456i,free=6308044
80i,sunreclaim=23289856i,swap_free=0i,cached=250294272i,huge_pages_total=0i,write_back=0i,total=101388
6976i,active=156626944i,slab=46522368i,vmalloc_chunk=35184362651648i,vmalloc_used=6332416i,used=130670
592i,inactive=140836864i,available_percent=71.78292819889226,buffered=2117632i,available=727797760i,us
ed_percent=12.88808270479253,high_total=0i,huge_page_size=2097152i,low_total=0i,mapped=40583168i,swap_
total=0i,commit_limit=506941440i,high_free=0i 1604702225000000000
2020-11-06T22:37:05Z D! [outputs.file] Wrote batch of 1 metrics in 92.731µs
2020-11-06T22:37:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-11-06T22:37:05Z D! [agent] Stopped Successfully
© 2020 InfluxData. All rights reserved.
62
Exercise 3
Enabling a plugin
© 2020 InfluxData. All rights reserved.
63
Mem
Net
Stdout
© 2020 InfluxData. All rights reserved.
64
Start
[global_tags]
[agent]
interval = "2s"
flush_interval =
"2s"
[[inputs.mem]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval =
"2s"
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
65
Validate using --test
$ telegraf --test --config exercise
$ telegraf --test --config exercise
2020-11-06T22:38:39Z I! Starting Telegraf 1.16.1
2020-11-06T22:38:39Z D! [agent] Initializing plugins
2020-11-06T22:38:39Z D! [agent] Starting service inputs
> net,host=influxdays,interface=ens5
bytes_recv=1708843i,bytes_sent=2007423i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=7752i,packets_sent=5597i
1604702319000000000
> net,host=influxdays,interface=all
icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_i
nerrors=0i,icmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=
0i,icmp_intimestamps=0i,icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0
i,icmp_outerrors=0i,icmp_outmsgs=2i,icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_ou
ttimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagra
ms=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_inaddrerrors=0i,ip_indelivers=7981i,ip_indiscards=0i,ip_inhdrerrors=0i,
ip_inreceives=7986i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnoroutes=16i,ip_outrequests=5516i,ip_reasmfails=0i,ip_reasmoks
=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tcp_currestab=4i,tcp_estabresets=4i,tcp_incsum
errors=5i,tcp_inerrs=5i,tcp_insegs=7733i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5563i,tcp_passiveopens=68i,tcp_retranssegs=
43i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=312i,udp_inerrors=0i,udp_noports=
2i,udp_outdatagrams=315i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_inerrors
=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702319000000000
2020-11-06T22:38:39Z D! [agent] Stopping service inputs
2020-11-06T22:38:39Z D! [agent] Input channel closed
2020-11-06T22:38:39Z D! [agent] Stopped Successfully
> mem,host=influxdays
active=159830016i,available=723709952i,available_percent=71.37974637520149,buffered=2117632i,cached=250327040i,commit_limit=5069
41440i,committed_as=407773184i,dirty=8192i,free=626688000i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i
,huge_pages_total=0i,inactive=140804096i,low_free=0i,low_total=0i,mapped=40611840i,page_tables=4898816i,shared=13058048i,slab=46
534656i,sreclaimable=23240704i,sunreclaim=23293952i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=134754304i,
used_percent=13.290860538680004,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,
write_back_tmp=0i 1604702319000000000
© 2020 InfluxData. All rights reserved.
66
$ telegraf --config exercise
2020-11-06T22:39:26Z I! Starting Telegraf 1.16.1
2020-11-06T22:39:26Z I! Loaded inputs: mem net
2020-11-06T22:39:26Z I! Loaded aggregators:
2020-11-06T22:39:26Z I! Loaded processors:
2020-11-06T22:39:26Z I! Loaded outputs: file
2020-11-06T22:39:26Z I! Tags enabled: host=influxdays
2020-11-06T22:39:26Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s
2020-11-06T22:39:26Z D! [agent] Initializing plugins
2020-11-06T22:39:26Z D! [agent] Connecting outputs
2020-11-06T22:39:26Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:39:26Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:39:26Z D! [agent] Starting service inputs
mem,host=influxdays
free=629272576i,huge_pages_free=0i,low_total=0i,commit_limit=506941440i,high_free=0i,huge_pages_total=0i,vmalloc_chunk=35184362651648i,used=132153344i,high_tot
al=0i,shared=13062144i,sunreclaim=23162880i,vmalloc_total=35184372087808i,write_back_tmp=0i,inactive=140779520i,used_percent=13.034327013586177,swap_cached
=0i,vmalloc_used=6332416i,write_back=0i,buffered=2117632i,cached=250343424i,committed_as=403091456i,swap_total=0i,total=1013886976i,available=726310912i,availa
ble_percent=71.63627990029532,active=158572544i,dirty=4096i,page_tables=4214784i,slab=46395392i,huge_page_size=2097152i,low_free=0i,mapped=40656896i,sreclai
mable=23232512i,swap_free=0i 1604702368000000000
net,host=influxdays,interface=ens5 drop_in=0i,drop_out=0i,bytes_sent=2037575i,bytes_recv=1723821i,packets_sent=5696i,packets_recv=7891i,err_in=0i,err_out=0i
1604702368000000000
net,host=influxdays,interface=all
udplite_rcvbuferrors=0i,ip_outdiscards=0i,ip_inaddrerrors=0i,icmp_inechoreps=0i,icmp_outtimestampreps=0i,icmp_outerrors=0i,icmp_outechoreps=0i,icmp_outaddrmaskreps
=0i,icmpmsg_outtype3=2i,udplite_noports=0i,udplite_incsumerrors=0i,udp_noports=2i,icmp_incsumerrors=0i,icmp_outechos=0i,icmp_outaddrmasks=0i,icmp_inechos=0i,icmp
_outparmprobs=0i,ip_inhdrerrors=0i,ip_outnoroutes=16i,ip_defaultttl=64i,icmp_outdestunreachs=2i,udp_indatagrams=316i,tcp_currestab=1i,ip_fragfails=0i,ip_reasmtimeout=0
i,ip_reasmoks=0i,icmp_outredirects=0i,tcp_retranssegs=45i,tcp_passiveopens=69i,tcp_maxconn=-1i,tcp_inerrs=5i,ip_inreceives=8124i,ip_forwarding=2i,icmp_inmsgs=2i,icmp
_indestunreachs=2i,icmpmsg_intype3=2i,ip_indiscards=0i,icmp_intimeexcds=0i,udp_incsumerrors=0i,tcp_outsegs=5655i,ip_forwdatagrams=0i,ip_fragcreates=0i,icmp_inredir
ects=0i,icmp_outsrcquenchs=0i,icmp_inaddrmasks=0i,ip_indelivers=8119i,icmp_inparmprobs=0i,tcp_insegs=7867i,tcp_attemptfails=72i,udp_inerrors=0i,tcp_estabresets=4i,tc
p_outrsts=68i,udp_rcvbuferrors=0i,ip_reasmfails=0i,ip_fragoks=0i,ip_reasmreqds=0i,icmp_outtimestamps=0i,icmp_intimestampreps=0i,icmp_inerrors=0i,tcp_incsumerrors=5i,
udplite_inerrors=0i,ip_outrequests=5610i,icmp_insrcquenchs=0i,icmp_outmsgs=2i,icmp_outtimeexcds=0i,udp_sndbuferrors=0i,tcp_rtoalgorithm=1i,udp_outdatagrams=319i,u
dplite_outdatagrams=0i,tcp_rtomin=200i,icmp_inaddrmaskreps=0i,udplite_sndbuferrors=0i,ip_inunknownprotos=0i,icmp_intimestamps=0i,tcp_rtomax=120000i,tcp_activeopen
s=142i,udplite_indatagrams=0i 1604702368000000000
2020-11-06T22:39:28Z D! [outputs.file] Wrote batch of 3 metrics in 139.162µs
2020-11-06T22:39:28Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Validate without --test
© 2020 InfluxData. All rights reserved.
67
Filtering
© 2021 InfluxData. All rights reserved. 68
© 2019 InfluxData. All rights reserved.
68
What Is An InfluxDB Series?
First, some definitions:
• Measurement: namespace for categorically alike metrics (~Table)
• Tag: key-value pair used as metadata for measurable object
– host=prod001 or app=cart or customer_id=00012
• Field: key-value pair representing a metric that will append new values
over time
– cpu_usage=35.5 or errors=42 or temp=65.0
• TagSet: unique combination of Tag key-value pairs
• FieldSet: unique combination of Field key-value pairs
© 2021 InfluxData. All rights reserved. 69
© 2019 InfluxData. All rights reserved.
69
What Is An InfluxDB Series?
Line/Record: <Measurement>,<TagSet> <FieldSet> <timestamp>
→ cpu,host=prod001,region=us-west usage_user=35,usage_system=15 t0
Point: <Measurement>,<TagSet> <FieldKey> <timestamp>
→ cpu,host=prod001,region=us-west usage_user=35 t0
Series: <Measurement>,<TagSet> <FieldKey>
→ cpu,host=prod001,region=us-west usage_user
© 2021 InfluxData. All rights reserved. 70
© 2019 InfluxData. All rights reserved.
70
Series Visualized (tree)
Cardinality is total number of “leaf nodes”
© 2020 InfluxData. All rights reserved.
71
Filtering metrics
• Happens at plugin level
• Specify which metrics and which parts of metrics the given plugin sees or doesn’t see
• Given above, filtering is often used for “routing”
• By Measurement:
• namepass: passes only records with specified Measurement name
• namedrop: drops only records with specified Measurement name
• By Field:
• fieldpass: pass specified Field
• fielddrop: drop specified Field
• By Tag (slightly different implementation):
• tagpass or tagdrop: pass/drop whole record containing matched Tag
• taginclude or tagexclude: keep/drop matching Tag (key) in record
© 2020 InfluxData. All rights reserved.
72
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA free=85.0,cached=12.5,buffered=3.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA free=85.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA cached=85.0 <timestamp>
mem,region=us-west-1,host=hostA,container=containerA buffered=85.0 <timestamp>
namepass = “mem”
© 2020 InfluxData. All rights reserved.
73
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0
,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp>
fieldpass = “*_system”
© 2020 InfluxData. All rights reserved.
74
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0
,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp>
fielddrop = “*_softirq”
© 2020 InfluxData. All rights reserved.
75
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp>
tagpass: host = [“hostA”]
© 2020 InfluxData. All rights reserved.
76
How to think of this
Telegraf metric:
cpu,region=us-west-1,host=hostA,container=containerA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp>
tagdrop: host = [“hostB”]
© 2020 InfluxData. All rights reserved.
77
How to think of this
Telegraf metric:
cpu,region=us-west-1,container=containerA,host=hostA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp>
taginclude:
host = [“host”]
© 2020 InfluxData. All rights reserved.
78
How to think of this
Telegraf metric:
cpu,region=us-west-1,container=containerA,host=hostA
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB
usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp>
How InfluxDB sees it:
cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp>
cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp>
tagexclude:
host = [“host”]
© 2020 InfluxData. All rights reserved.
79
Exercise 4
Filtering
© 2020 InfluxData. All rights reserved.
80
Sample Data
mem,host=influxdays
available=735477760i,inactive=167567360i,page_tables=4345856i,sreclaimable=22962176i,committed_as=402309120i,slab=46772224i,vmalloc_total=35184
372087808i,cached=243228672i,high_free=0i,huge_page_size=2097152i,huge_pages_total=0i,write_back=0i,total=1013886976i,dirty=0i,free=645414912i,lo
w_total=0i,sunreclaim=23810048i,swap_cached=0i,huge_pages_free=0i,low_free=0i,shared=13058048i,swap_total=0i,used=123125760i,used_percent=12.1
43933487118785,buffered=2117632i,vmalloc_used=6332416i,available_percent=72.54040908007482,high_total=0i,vmalloc_chunk=35184362651648i,active
=113328128i,commit_limit=506941440i,mapped=40271872i,swap_free=0i,write_back_tmp=0i 1602630666000000000
net,host=influxdays,interface=ens5
packets_recv=11760i,err_in=0i,err_out=0i,drop_in=0i,drop_out=0i,bytes_sent=1730127i,bytes_recv=1082190i,packets_sent=10073i 1602630666000000000
net,host=influxdays,interface=all
icmp_outechos=0i,icmp_inechoreps=0i,tcp_rtoalgorithm=1i,tcp_insegs=9422i,ip_inunknownprotos=0i,ip_indiscards=0i,ip_reasmreqds=0i,icmp_outdestunreac
hs=2i,icmpmsg_outtype3=2i,icmp_outaddrmasks=0i,icmp_outredirects=0i,icmp_indestunreachs=3i,icmp_incsumerrors=0i,udp_incsumerrors=0i,ip_inhdrerrors
=0i,ip_reasmoks=0i,tcp_maxconn=-1i,udp_sndbuferrors=0i,tcp_incsumerrors=0i,tcp_estabresets=21i,udplite_rcvbuferrors=0i,ip_reasmtimeout=0i,icmp_outerr
ors=0i,icmp_intimeexcds=0i,udplite_sndbuferrors=0i,icmpmsg_intype3=3i,tcp_activeopens=188i,tcp_rtomax=120000i,udp_outdatagrams=991i,udplite_incsum
errors=0i,ip_fragcreates=0i,ip_fragoks=0i,icmp_inerrors=0i,tcp_rtomin=200i,udp_rcvbuferrors=0i,udp_noports=2i,ip_inaddrerrors=0i,ip_reasmfails=0i,ip_forwar
ding=2i,icmp_outechoreps=0i,icmp_intimestamps=0i,icmp_outmsgs=2i,tcp_outrsts=141i,udp_indatagrams=932i,ip_inreceives=10233i,ip_outdiscards=0i,icmp
_intimestampreps=0i,icmp_outtimestamps=0i,icmp_outsrcquenchs=0i,tcp_passiveopens=269i,tcp_currestab=1i,udplite_indatagrams=0i,ip_outnoroutes=16i,ic
mp_outtimestampreps=0i,icmp_outparmprobs=0i,icmp_inaddrmasks=0i,icmp_outaddrmaskreps=0i,icmp_inparmprobs=0i,ip_defaultttl=64i,ip_outrequests=836
7i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_outtimeexcds=0i,icmp_inmsgs=3i,icmp_inaddrmaskreps=0i,ip_forwdatagrams=0i,ip_fragfails=0i,icmp_inec
hos=0i,tcp_inerrs=0i,tcp_attemptfails=175i,tcp_outsegs=7517i,udplite_inerrors=0i,udplite_noports=0i,ip_indelivers=10227i,tcp_retranssegs=118i,udp_inerrors
=0i,udplite_outdatagrams=0i 1602630666000000000
© 2020 InfluxData. All rights reserved.
81
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
Pass fields
ending in “percent”
exact named “total”
[[inputs.net]]
Pass fields Beginning with “bytes”
Pass metrics where interface begins
with “en”
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
82
Exercise 4
Filtering
Solution
© 2020 InfluxData. All rights reserved.
83
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
[[inputs.net]]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
...
© 2020 InfluxData. All rights reserved.
84
Correct Placement
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
...
© 2020 InfluxData. All rights reserved.
85
Before & After
$ telegraf --test --config exercise
> net,host=influxdays,interface= ens5
bytes_recv=1739020i,bytes_sent=2088482i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=8042i,packets_sent=5843i
1604702475000000000
> mem,host=influxdays
active=154382336i,available=730259456i,available_percent=72.02572607067398,buffered=2117632i,cached=250368000i,commit_limit=506941440i,commi
tted_as=403091456i,dirty=20480i,free=633192448i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,in
active=140689408i,low_free=0i,low_total=0i,mapped=40558592i,page_tables=4190208i,shared=13058048i,slab=46452736i,sreclaimable=23232512i,sunr
eclaim=23220224i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=128208896i,used_percent=12.645284833010814,vmalloc_chunk=3
5184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i 1604702475000000000
2020-11-06T22:41:14Z D! [agent] Stopping service inputs
2020-11-06T22:41:14Z D! [agent] Input channel closed
2020-11-06T22:41:14Z D! [agent] Stopped Successfully
> net,host=influxdays,interface= all
icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_inerrors=0i,i
cmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=0i,icmp_intimestamps=0i,
icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0i,icmp_outerrors=0i,icmp_outmsgs=2i,
icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_outtimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_
intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagrams=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_ina
ddrerrors=0i,ip_indelivers=8268i,ip_indiscards=0i,ip_inhdrerrors=0i,ip_inreceives=8273i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnorout
es=16i,ip_outrequests=5747i,ip_reasmfails=0i,ip_reasmoks=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tc
p_currestab=1i,tcp_estabresets=4i,tcp_incsumerrors=5i,tcp_inerrs=5i,tcp_insegs=8010i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5792i,tcp_p
assiveopens=74i,tcp_retranssegs=47i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=322i,udp_iner
rors=0i,udp_noports=2i,udp_outdatagrams=325i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_
inerrors=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702475000000000
….
After filtering...
….
> mem,host=influxdays available_percent=71.50255927540388,total=1013886976i,used_percent=13.168451628280902 1604702498000000000
> net,host=influxdays,interface= ens5 bytes_recv=1745105i,bytes_sent=2111895i 1604702498000000000
© 2020 InfluxData. All rights reserved.
86
Placement Matters,
Indentation Does Not
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
[inputs.net.tagpass]
interface = ["en*"]
fieldpass = ["bytes*"]
[[outputs.file]]
> net,host=influxdays,interface=ens5
bytes_recv=2082848i,bytes_sent=3148789i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=21247
i,packets_sent=19323i 1604702498000000000
© 2020 InfluxData. All rights reserved.
87
Exercise 5
Dual Write
© 2020 InfluxData. All rights reserved.
88
Mem
Net net
/tmp/net
m
em
/tmp/mem
© 2020 InfluxData. All rights reserved.
89
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
Pass the “mem” data to file /tmp/mem
[[outputs.file]]
Pass the “net” data to file /tmp/net
© 2020 InfluxData. All rights reserved.
90
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]
© 2020 InfluxData. All rights reserved.
91
Exercise 6
Listener
© 2020 InfluxData. All rights reserved.
92
Mem
Net
ne
t
/tmp/net.txt
m
em
/tmp/mem.txt
:8086
Add
© 2020 InfluxData. All rights reserved.
93
telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
Set the listener port to ":8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
94
telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
service_address = ": 8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
95
Launch Telegraf Listener as background job
$ telegraf --config telegraf-listener.conf &
[1] 2219
[centos@influxdays ~]$ 2020-11-06T22:44:22Z I! Starting Telegraf
1.16.1
[centos@influxdays ~]$
[centos@influxdays ~]$
[centos@influxdays ~]$
© 2020 InfluxData. All rights reserved.
96
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
[[inputs.mem]]
fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
namepass = ["mem"]
files = ["/tmp/mem"]
[[outputs.file]]
namepass = ["net"]
files = ["/tmp/net"]
[[outputs.influxdb]]
© 2020 InfluxData. All rights reserved.
97
Launch Telegraf
$ telegraf -config exercise
2020-11-06T22:45:09Z I! Starting Telegraf 1.16.1
2020-11-06T22:45:09Z I! Loaded inputs: mem net
2020-11-06T22:45:09Z I! Loaded aggregators:
2020-11-06T22:45:09Z I! Loaded processors:
2020-11-06T22:45:09Z I! Loaded outputs: file (2x) influxdb
2020-11-06T22:45:09Z I! Tags enabled: host=influxdays
2020-11-06T22:45:09Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s
2020-11-06T22:45:09Z D! [agent] Initializing plugins
2020-11-06T22:45:09Z D! [agent] Connecting outputs
2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file]
2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file
2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.influxdb]
2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.influxdb
2020-11-06T22:45:09Z D! [agent] Starting service inputs
2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 101.011µs
2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 23.33µs
2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-11-06T22:45:11Z D! [outputs.influxdb] Wrote batch of 2 metrics in 1.205233ms
2020-11-06T22:45:11Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766
1604702710000000000
net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000
© 2020 InfluxData. All rights reserved.
98
Inspect the filesystem
$ cat /tmp/mem
mem,host=influxdays
used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766
1604702710000000000
$ cat /tmp/net
net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000
© 2020 InfluxData. All rights reserved.
99
telegraf-listener.conf
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
round_interval = true
metric_batch_size = 5000
metric_buffer_limit = 50000
debug = true
logfile = "/tmp/telegraf-aggregator.log"
[[inputs.influxdb_listener]]
service_address = ":8086"
read_timeout = "2s"
write_timeout = "2s"
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
100
Exercise 7
Starlark
© 2020 InfluxData. All rights reserved.
101
Finish
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[processors.starlark]]
source = '''
def apply(metric):
for k, v in metric.fields.items():
if type(v) == "float" or type(v) == "int":
metric.fields[k] = v * 1000
return metric
'''
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
..
..
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
102
Launch Telegraf
Without Starlark processor
> net,host=influxdays,interface=ens5 bytes_recv=9566941i,bytes_sent=23302741i 1604939494000000000
After multiplying by 1000
> net,host=influxdays,interface=ens5 bytes_recv=9555668000i,bytes_sent=23274132000i 1604939466000000000
© 2020 InfluxData. All rights reserved.
103
Exercise 8
JSON
© 2020 InfluxData. All rights reserved.
104
Mem
Net net
/tmp/net.txt
m
em
/tmp/mem.txt
HTTP
© 2020 InfluxData. All rights reserved.
105
Add
[[inputs.http]]
urls = ["https://api.mocki.io/v1/29dfca5b"]
data_format = “json”
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
© 2020 InfluxData. All rights reserved.
106
Add
[[inputs.http]]
urls = ["https://api.mocki.io/v1/29dfca5b"]
data_format = “json”
name_override = “apps”
json_query = “apps”
tag_keys = [“name”,”id”]
[[outputs.file]]
Start
[global_tags]
[agent]
interval = "2s"
flush_interval = "2s"
debug = true
#[[inputs.mem]]
# fieldpass = ["*percent","total"]
[[inputs.net]]
fieldpass = ["bytes*"]
[inputs.net.tagpass]
interface = ["en*"]
[[outputs.file]]
© 2021 InfluxData. All rights reserved. 107
Telegraf Resources
Blogs
https:/
/www.influxdata.com/blog/using-a-telegraf-gateway/
https:/
/www.influxdata.com/blog/telegraf-go-collection-agent/
Docs
https:/
/docs.influxdata.com/telegraf/latest
Prebuilt dashboards
https:/
/docs.influxdata.com/platform/monitoring/influxdata-platform/monitoring-dashb
oards/
Questions?
Thank you
See you tomorrow at InfluxDays EMEA 2021!
© 2021 InfluxData. All rights reserved. 110
InfluxDays EMEA 2021
See you tomorrow: 18-19 May 2021
1 of 110

Recommended

Grafana optimization for Prometheus by
Grafana optimization for PrometheusGrafana optimization for Prometheus
Grafana optimization for PrometheusMitsuhiro Tanda
3.9K views28 slides
Monitoring and alerting as code with Terraform and New Relic by
Monitoring and alerting as code with Terraform and New RelicMonitoring and alerting as code with Terraform and New Relic
Monitoring and alerting as code with Terraform and New RelicRichard Bullington-McGuire
1.2K views16 slides
eBPF - Observability In Deep by
eBPF - Observability In DeepeBPF - Observability In Deep
eBPF - Observability In DeepMydbops
638 views25 slides
How to Monitoring the SRE Golden Signals (E-Book) by
How to Monitoring the SRE Golden Signals (E-Book)How to Monitoring the SRE Golden Signals (E-Book)
How to Monitoring the SRE Golden Signals (E-Book)Siglos
14.1K views36 slides
InfluxDB + Kepware: Start Monitoring Industrial Data Quickly by
InfluxDB + Kepware: Start Monitoring Industrial Data QuicklyInfluxDB + Kepware: Start Monitoring Industrial Data Quickly
InfluxDB + Kepware: Start Monitoring Industrial Data QuicklyInfluxData
1.2K views20 slides
Monitoring, the Prometheus Way - Julius Voltz, Prometheus by
Monitoring, the Prometheus Way - Julius Voltz, Prometheus Monitoring, the Prometheus Way - Julius Voltz, Prometheus
Monitoring, the Prometheus Way - Julius Voltz, Prometheus Docker, Inc.
4.6K views32 slides

More Related Content

What's hot

Getting Started Monitoring with Prometheus and Grafana by
Getting Started Monitoring with Prometheus and GrafanaGetting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and GrafanaSyah Dwi Prihatmoko
3.5K views29 slides
Terraform modules and some of best-practices - March 2019 by
Terraform modules and some of best-practices - March 2019Terraform modules and some of best-practices - March 2019
Terraform modules and some of best-practices - March 2019Anton Babenko
3.9K views102 slides
Intro to Telegraf by
Intro to TelegrafIntro to Telegraf
Intro to TelegrafInfluxData
704 views45 slides
Py.test by
Py.testPy.test
Py.testsoasme
6.7K views51 slides
OpenTelemetry For Operators by
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For OperatorsKevin Brockhoff
930 views32 slides
Complete-NGINX-Cookbook-2019.pdf by
Complete-NGINX-Cookbook-2019.pdfComplete-NGINX-Cookbook-2019.pdf
Complete-NGINX-Cookbook-2019.pdfTomaszWojciechowski22
159 views175 slides

What's hot(20)

Getting Started Monitoring with Prometheus and Grafana by Syah Dwi Prihatmoko
Getting Started Monitoring with Prometheus and GrafanaGetting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and Grafana
Syah Dwi Prihatmoko3.5K views
Terraform modules and some of best-practices - March 2019 by Anton Babenko
Terraform modules and some of best-practices - March 2019Terraform modules and some of best-practices - March 2019
Terraform modules and some of best-practices - March 2019
Anton Babenko3.9K views
Intro to Telegraf by InfluxData
Intro to TelegrafIntro to Telegraf
Intro to Telegraf
InfluxData704 views
Py.test by soasme
Py.testPy.test
Py.test
soasme6.7K views
Intro to GitOps & Flux.pdf by Weaveworks
Intro to GitOps & Flux.pdfIntro to GitOps & Flux.pdf
Intro to GitOps & Flux.pdf
Weaveworks331 views
Docker Networking Overview by Sreenivas Makam
Docker Networking OverviewDocker Networking Overview
Docker Networking Overview
Sreenivas Makam21.1K views
Monitoring with prometheus by Kasper Nissen
Monitoring with prometheusMonitoring with prometheus
Monitoring with prometheus
Kasper Nissen6.8K views
Full Stack Visualization: Build A React App With A Sankey Diagram by Neo4j
Full Stack Visualization: Build A React App With A Sankey DiagramFull Stack Visualization: Build A React App With A Sankey Diagram
Full Stack Visualization: Build A React App With A Sankey Diagram
Neo4j374 views
Terraform introduction by Jason Vance
Terraform introductionTerraform introduction
Terraform introduction
Jason Vance763 views
Monitoring the Hashistack with Prometheus by Grafana Labs
Monitoring the Hashistack with PrometheusMonitoring the Hashistack with Prometheus
Monitoring the Hashistack with Prometheus
Grafana Labs1.4K views
Jenkins for java world by Ashok Kumar
Jenkins for java worldJenkins for java world
Jenkins for java world
Ashok Kumar2.9K views
Prometheus-Grafana-RahulSoni1584KnolX.pptx.pdf by Knoldus Inc.
Prometheus-Grafana-RahulSoni1584KnolX.pptx.pdfPrometheus-Grafana-RahulSoni1584KnolX.pptx.pdf
Prometheus-Grafana-RahulSoni1584KnolX.pptx.pdf
Knoldus Inc.160 views
Server monitoring using grafana and prometheus by Celine George
Server monitoring using grafana and prometheusServer monitoring using grafana and prometheus
Server monitoring using grafana and prometheus
Celine George2K views
Understand your system like never before with OpenTelemetry, Grafana, and Pro... by LibbySchulze
Understand your system like never before with OpenTelemetry, Grafana, and Pro...Understand your system like never before with OpenTelemetry, Grafana, and Pro...
Understand your system like never before with OpenTelemetry, Grafana, and Pro...
LibbySchulze356 views
Static code analysis with sonar qube by Hayi Nukman
Static code analysis with sonar qubeStatic code analysis with sonar qube
Static code analysis with sonar qube
Hayi Nukman1.2K views
A GitOps Kubernetes Native CICD Solution with Argo Events, Workflows, and CD by Julian Mazzitelli
A GitOps Kubernetes Native CICD Solution with Argo Events, Workflows, and CDA GitOps Kubernetes Native CICD Solution with Argo Events, Workflows, and CD
A GitOps Kubernetes Native CICD Solution with Argo Events, Workflows, and CD
Julian Mazzitelli428 views
CouchDB Vs MongoDB by Gabriele Lana
CouchDB Vs MongoDBCouchDB Vs MongoDB
CouchDB Vs MongoDB
Gabriele Lana69.5K views

Similar to Taming the Tiger: Tips and Tricks for Using Telegraf

Taming the Tiger: Tips and Tricks for Using Telegraf by
Taming the Tiger: Tips and Tricks for Using TelegrafTaming the Tiger: Tips and Tricks for Using Telegraf
Taming the Tiger: Tips and Tricks for Using TelegrafInfluxData
146 views91 slides
Getting Started: Intro to Telegraf - July 2021 by
Getting Started: Intro to Telegraf - July 2021Getting Started: Intro to Telegraf - July 2021
Getting Started: Intro to Telegraf - July 2021InfluxData
537 views48 slides
How to Use Telegraf and Its Plugin Ecosystem by
How to Use Telegraf and Its Plugin EcosystemHow to Use Telegraf and Its Plugin Ecosystem
How to Use Telegraf and Its Plugin EcosystemInfluxData
1.2K views42 slides
InfluxDB Live Product Training by
InfluxDB Live Product TrainingInfluxDB Live Product Training
InfluxDB Live Product TrainingInfluxData
160 views34 slides
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ... by
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...InfluxData
2.4K views33 slides
Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ... by
Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ...Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ...
Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ...InfluxData
71 views25 slides

Similar to Taming the Tiger: Tips and Tricks for Using Telegraf(20)

Taming the Tiger: Tips and Tricks for Using Telegraf by InfluxData
Taming the Tiger: Tips and Tricks for Using TelegrafTaming the Tiger: Tips and Tricks for Using Telegraf
Taming the Tiger: Tips and Tricks for Using Telegraf
InfluxData146 views
Getting Started: Intro to Telegraf - July 2021 by InfluxData
Getting Started: Intro to Telegraf - July 2021Getting Started: Intro to Telegraf - July 2021
Getting Started: Intro to Telegraf - July 2021
InfluxData537 views
How to Use Telegraf and Its Plugin Ecosystem by InfluxData
How to Use Telegraf and Its Plugin EcosystemHow to Use Telegraf and Its Plugin Ecosystem
How to Use Telegraf and Its Plugin Ecosystem
InfluxData1.2K views
InfluxDB Live Product Training by InfluxData
InfluxDB Live Product TrainingInfluxDB Live Product Training
InfluxDB Live Product Training
InfluxData160 views
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ... by InfluxData
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...
Scaling Prometheus Metrics in Kubernetes with Telegraf | Chris Goller | Influ...
InfluxData2.4K views
Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ... by InfluxData
Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ...Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ...
Jess Ingrassellino [InfluxData] | How to Get Data Into InfluxDB | InfluxDays ...
InfluxData71 views
The Flink - Apache Bigtop integration by Márton Balassi
The Flink - Apache Bigtop integrationThe Flink - Apache Bigtop integration
The Flink - Apache Bigtop integration
Márton Balassi635 views
Write your own telegraf plugin by InfluxData
Write your own telegraf pluginWrite your own telegraf plugin
Write your own telegraf plugin
InfluxData13.8K views
InfluxDB + Telegraf Operator: Easy Kubernetes Monitoring by InfluxData
InfluxDB + Telegraf Operator: Easy Kubernetes MonitoringInfluxDB + Telegraf Operator: Easy Kubernetes Monitoring
InfluxDB + Telegraf Operator: Easy Kubernetes Monitoring
InfluxData769 views
Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud by InfluxData
Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud
Announcing: Native MQTT Integration with HiveMQ and InfluxDB Cloud
InfluxData272 views
Shashi Raina [AWS] & Al Sargent [InfluxData] | Build Modern Monitoring with I... by InfluxData
Shashi Raina [AWS] & Al Sargent [InfluxData] | Build Modern Monitoring with I...Shashi Raina [AWS] & Al Sargent [InfluxData] | Build Modern Monitoring with I...
Shashi Raina [AWS] & Al Sargent [InfluxData] | Build Modern Monitoring with I...
InfluxData267 views
Advanced coding & deployment for Cisco Video Devices - CL20B - DEVNET-3244 by Cisco DevNet
Advanced coding & deployment for Cisco Video Devices - CL20B - DEVNET-3244Advanced coding & deployment for Cisco Video Devices - CL20B - DEVNET-3244
Advanced coding & deployment for Cisco Video Devices - CL20B - DEVNET-3244
Cisco DevNet224 views
Pivotal Platform - December Release A First Look by VMware Tanzu
Pivotal Platform - December Release A First LookPivotal Platform - December Release A First Look
Pivotal Platform - December Release A First Look
VMware Tanzu470 views
Altinity Quickstart for ClickHouse-2202-09-15.pdf by Altinity Ltd
Altinity Quickstart for ClickHouse-2202-09-15.pdfAltinity Quickstart for ClickHouse-2202-09-15.pdf
Altinity Quickstart for ClickHouse-2202-09-15.pdf
Altinity Ltd395 views
Integrated, Automated Video Room Systems - Webex Devices - Cisco Live Orlando... by Cisco DevNet
Integrated, Automated Video Room Systems - Webex Devices - Cisco Live Orlando...Integrated, Automated Video Room Systems - Webex Devices - Cisco Live Orlando...
Integrated, Automated Video Room Systems - Webex Devices - Cisco Live Orlando...
Cisco DevNet676 views
Ana-Maria Calin [InfluxData] | Migrating from OSS to InfluxDB Cloud | InfluxD... by InfluxData
Ana-Maria Calin [InfluxData] | Migrating from OSS to InfluxDB Cloud | InfluxD...Ana-Maria Calin [InfluxData] | Migrating from OSS to InfluxDB Cloud | InfluxD...
Ana-Maria Calin [InfluxData] | Migrating from OSS to InfluxDB Cloud | InfluxD...
InfluxData78 views
Cloud Composer workshop at Airflow Summit 2023.pdf by Leah Cole
Cloud Composer workshop at Airflow Summit 2023.pdfCloud Composer workshop at Airflow Summit 2023.pdf
Cloud Composer workshop at Airflow Summit 2023.pdf
Leah Cole164 views
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi... by Lightbend
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...
Lightbend1K views
Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M... by InfluxData
Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...
Vasilis Papavasiliou [Mist.io] | Integrating Telegraf, InfluxDB and Mist to M...
InfluxData138 views

More from InfluxData

Announcing InfluxDB Clustered by
Announcing InfluxDB ClusteredAnnouncing InfluxDB Clustered
Announcing InfluxDB ClusteredInfluxData
100 views30 slides
Best Practices for Leveraging the Apache Arrow Ecosystem by
Best Practices for Leveraging the Apache Arrow EcosystemBest Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow EcosystemInfluxData
50 views25 slides
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu... by
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...InfluxData
137 views24 slides
Power Your Predictive Analytics with InfluxDB by
Power Your Predictive Analytics with InfluxDBPower Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDBInfluxData
127 views41 slides
Build an Edge-to-Cloud Solution with the MING Stack by
Build an Edge-to-Cloud Solution with the MING StackBuild an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING StackInfluxData
381 views52 slides
Meet the Founders: An Open Discussion About Rewriting Using Rust by
Meet the Founders: An Open Discussion About Rewriting Using RustMeet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using RustInfluxData
235 views12 slides

More from InfluxData(20)

Announcing InfluxDB Clustered by InfluxData
Announcing InfluxDB ClusteredAnnouncing InfluxDB Clustered
Announcing InfluxDB Clustered
InfluxData100 views
Best Practices for Leveraging the Apache Arrow Ecosystem by InfluxData
Best Practices for Leveraging the Apache Arrow EcosystemBest Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow Ecosystem
InfluxData50 views
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu... by InfluxData
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
InfluxData137 views
Power Your Predictive Analytics with InfluxDB by InfluxData
Power Your Predictive Analytics with InfluxDBPower Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDB
InfluxData127 views
Build an Edge-to-Cloud Solution with the MING Stack by InfluxData
Build an Edge-to-Cloud Solution with the MING StackBuild an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING Stack
InfluxData381 views
Meet the Founders: An Open Discussion About Rewriting Using Rust by InfluxData
Meet the Founders: An Open Discussion About Rewriting Using RustMeet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using Rust
InfluxData235 views
Introducing InfluxDB Cloud Dedicated by InfluxData
Introducing InfluxDB Cloud DedicatedIntroducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud Dedicated
InfluxData129 views
Gain Better Observability with OpenTelemetry and InfluxDB by InfluxData
Gain Better Observability with OpenTelemetry and InfluxDB Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB
InfluxData403 views
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali... by InfluxData
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
InfluxData183 views
How Delft University's Engineering Students Make Their EV Formula-Style Race ... by InfluxData
How Delft University's Engineering Students Make Their EV Formula-Style Race ...How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
InfluxData105 views
Start Automating InfluxDB Deployments at the Edge with balena by InfluxData
Start Automating InfluxDB Deployments at the Edge with balena Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena
InfluxData185 views
Understanding InfluxDB’s New Storage Engine by InfluxData
Understanding InfluxDB’s New Storage EngineUnderstanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage Engine
InfluxData137 views
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB by InfluxData
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDBStreamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
InfluxData63 views
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa... by InfluxData
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
InfluxData74 views
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022 by InfluxData
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
InfluxData26 views
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022 by InfluxData
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
InfluxData9 views
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ... by InfluxData
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
InfluxData10 views
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022 by InfluxData
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
InfluxData5 views
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022 by InfluxData
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022
InfluxData112 views
Jay Clifford [InfluxData] | Tips & Tricks for Analyzing IIoT in Real-Time | I... by InfluxData
Jay Clifford [InfluxData] | Tips & Tricks for Analyzing IIoT in Real-Time | I...Jay Clifford [InfluxData] | Tips & Tricks for Analyzing IIoT in Real-Time | I...
Jay Clifford [InfluxData] | Tips & Tricks for Analyzing IIoT in Real-Time | I...
InfluxData19 views

Recently uploaded

iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...Bernd Ruecker
50 views69 slides
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
344 views86 slides
Five Things You SHOULD Know About Postman by
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About PostmanPostman
40 views43 slides
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITShapeBlue
91 views8 slides
Why and How CloudStack at weSystems - Stephan Bienek - weSystems by
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsShapeBlue
111 views13 slides
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineShapeBlue
102 views19 slides

Recently uploaded(20)

iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker50 views
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software344 views
Five Things You SHOULD Know About Postman by Postman
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About Postman
Postman40 views
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by ShapeBlue
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
ShapeBlue91 views
Why and How CloudStack at weSystems - Stephan Bienek - weSystems by ShapeBlue
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystems
ShapeBlue111 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue102 views
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha... by ShapeBlue
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
ShapeBlue74 views
CloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlue by ShapeBlue
CloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlueCloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlue
CloudStack Managed User Data and Demo - Harikrishna Patnala - ShapeBlue
ShapeBlue46 views
DRBD Deep Dive - Philipp Reisner - LINBIT by ShapeBlue
DRBD Deep Dive - Philipp Reisner - LINBITDRBD Deep Dive - Philipp Reisner - LINBIT
DRBD Deep Dive - Philipp Reisner - LINBIT
ShapeBlue62 views
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue by ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
ShapeBlue96 views
State of the Union - Rohit Yadav - Apache CloudStack by ShapeBlue
State of the Union - Rohit Yadav - Apache CloudStackState of the Union - Rohit Yadav - Apache CloudStack
State of the Union - Rohit Yadav - Apache CloudStack
ShapeBlue145 views
PharoJS - Zürich Smalltalk Group Meetup November 2023 by Noury Bouraqadi
PharoJS - Zürich Smalltalk Group Meetup November 2023PharoJS - Zürich Smalltalk Group Meetup November 2023
PharoJS - Zürich Smalltalk Group Meetup November 2023
Noury Bouraqadi141 views
Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10369 views
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by ShapeBlue
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates
ShapeBlue119 views
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De... by Moses Kemibaro
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Moses Kemibaro29 views
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava... by ShapeBlue
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
ShapeBlue48 views
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ... by ShapeBlue
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...
Backroll, News and Demo - Pierre Charton, Matthias Dhellin, Ousmane Diarra - ...
ShapeBlue83 views

Taming the Tiger: Tips and Tricks for Using Telegraf

  • 1. © 2021 InfluxData. All rights reserved. 1 © 2020 InfluxData. All rights reserved. 1 Ignacio Van Droogenbroeck | Solutions Architect, InfluxData Suleman Khan | Solutions Architect, InfluxData May 2021
  • 2. Agenda ■ Overview ■ Setup (if not completed) ■ Global configuration ■ Enabling plugins ■ Filtering data
  • 3. Please join us in the InfluxDB Community Slack at www.influxdata.com/slack. To participate in the conversations, join the #influxdays-virtual & #telegraf-training channels. JOIN THE CONVERSATION
  • 4. Workshop Logistics • Cloud account free tier • No CC • Rate limits → Let’s avoid them • Browser • SSH client • Have documentation open for exercise • General config: https:/ /bit.ly/3t1r03K
  • 5. © 2021 InfluxData. All rights reserved. 5 Telegraf • Lightweight; written in Go • Plug-in driven • Optimized for writing to InfluxDB • Formatting • Retries • Modifiable batch sizes and jitter • Tag sorting • Preprocessing • Converting tags to fields, fields to tags • Regex transformations • Renaming measurements, tags • Aggregations (mean, min, max, count, variance, stddev, etc.)
  • 6. © 2021 InfluxData. All rights reserved. 6 Popular Inputs (Integrations) Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (producer/consumer) Prometheus (/metrics) SNMP Exec System (Linux,Windows,Unix) StatsD Nginx HAproxy OPC, AMQP, Modbus & MQTT (IoT)
  • 7. © 2021 InfluxData. All rights reserved. 7 Telegraf CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL Process -transform -decorate -filter Aggregate -mean -min,max -count -variance -stddev File InfluxDB Kafka CloudWatch CloudWatch
  • 8. © 2021 InfluxData. All rights reserved. 8
  • 9. © 2021 InfluxData. All rights reserved. 9 Telegraf Common Architecture Messaging Queue Kafka Rabbit Active NSQ AWS Kinesis Google PubSub MQTT
  • 10. © 2021 InfluxData. All rights reserved. 10 Telegraf Architecture Message Queue Kafka Rabbit Active NSQ AWS Kinesis Google PubSub MQTT
  • 11. © 2021 InfluxData. All rights reserved. 11 Parsing ● JSON ● CSV ● Graphite ● CollectD ● Dropwizard ● Form URL-encoded ● Grok
  • 12. © 2021 InfluxData. All rights reserved. 12 Telegraf Latest version available at https:/ /portal.influxdata.com/downloads/#telegraf Create the configuration file $ telegraf config > telegraf.conf # ~7500 lines OR $ telegraf -sample-config -input-filter cpu:mem -output-filter influxdb > telegraf.conf # 830 lines
  • 13. © 2021 InfluxData. All rights reserved. 13 Starlark Embedded Python dialect – Not full std lib – But most basic Python functionality supported Current processors and aggregators + more [[processors.starlark]] namepass = ["mymeasurement"] source = ''' def apply(metric): I = metric.fields['current'] V = metric.fields['voltage'] metric.fields['power'] = I * V return metric ''' [[processors.starlark]] source = ''' def apply(metric): used = float(metric.fields['used']) total = float(metric.fields['total']) metric.fields['usage'] = (used / total) * 100 return metric '''
  • 14. Setting up InfluxDB Cloud Sign up: https:/ /cloud2.influxdata.com/signup
  • 15. © 2021 InfluxData. All rights reserved. 15 InfluxDB Cloud Setup - Choose your Provider & Region - Enter your company name - Read & Tick the Service Agreement
  • 16. © 2021 InfluxData. All rights reserved. 16 InfluxDB Cloud Setup ● Select “Free” account
  • 17. © 2021 InfluxData. All rights reserved. 17 InfluxDB Cloud Setup - Entering “Getting Started” page
  • 18. © 2021 InfluxData. All rights reserved. 18 Demo Data
  • 19. © 2021 InfluxData. All rights reserved. 19 Finding your Organization ID 1 2
  • 20. © 2021 InfluxData. All rights reserved. 20 Token 2 3 4 1
  • 21. © 2021 InfluxData. All rights reserved. 21 Find your host Go to… https://bit.ly/3tGLuPy 1. Find your name 2. Copy from column E 3. Paste into console 4. Enter the password
  • 22. © 2021 InfluxData. All rights reserved. 22 Login Using an SSH client $ ssh centos@[ip-address] $ ssh centos@3.133.90.20 The authenticity of host '3.133.90.20 (3.133.90.20)' can't be established. ECDSA key fingerprint is SHA256:ki5Hm1AQ4UyyOn29uvoexeNO/Mk/Z0qourjEn+EpHdk. Are you sure you want to continue connecting (yes/no)? yes Password: InfluxDaysIsTheBestTrainingEver!!!!
  • 23. © 2021 InfluxData. All rights reserved. 23
  • 24. © 2021 InfluxData. All rights reserved. 24 Getting Setup Install Telegraf [centos@influxdays ~]$ sudo rpm -ivh /home/centos/telegraf-1.18.2-1.x86_64.rpm Preparing... ################################# [100%] Updating / installing… 1:telegraf-1.18.2-1 ################################# [100%] Created symlink from /etc/systemd/system/multi-user.target.wants/telegraf.service to /usr/lib/systemd/system/telegraf.service. [centos@influxdays ~]$
  • 25. © 2021 InfluxData. All rights reserved. 25 Create a Bucket 1 2
  • 26. © 2020 InfluxData. All rights reserved. 26 3 4
  • 27. © 2020 InfluxData. All rights reserved. 27 Create Telegraf Configuration 1 2
  • 28. © 2021 InfluxData. All rights reserved. 28 Create Telegraf Configuration 1 2 3 4 5
  • 29. © 2021 InfluxData. All rights reserved. 29 Test your Configuration (optional) 1 2 3 Paste in console
  • 30. © 2021 InfluxData. All rights reserved. 30 Select All Update configuration file 1 2 3 4
  • 31. © 2021 InfluxData. All rights reserved. 31 VI Cheat Sheet [centos@influxdays ~]$ sudo vi /etc/telegraf/telegraf.conf gg # Go to first line; this may not do anything dG # Delete all lines i # Insert mode <paste> # Paste the full config from InfluxDB Cloud Escape # End Insert mode :64 # Jump to line 64 to update Token :wq # Save and exit
  • 32. © 2021 InfluxData. All rights reserved. 32 Hardcode the Token Be sure to retain the double quotes 1 2 3
  • 33. Start the Service [centos@influxdays ~]$ sudo systemctl start telegraf Check the logs [centos@influxdays ~]$ $ journalctl -u telegraf -- Logs begin at Mon 2020-11-02 19:23:33 UTC, end at Mon 2020-11-02 19:55:58 UTC. -- Nov 02 19:55:58 influxdays systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB. Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Starting Telegraf 1.16.1 Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded inputs: cpu disk diskio mem net processes swap system Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded aggregators: Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded processors: Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Loaded outputs: influxdb_v2 Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! Tags enabled: host=influxdays Nov 02 19:55:58 influxdays telegraf[1340]: 2020-11-02T19:55:58Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"influxdays", Flush Interval:10s
  • 34. © 2021 InfluxData. All rights reserved. 34
  • 35. Alternate Deployment Launch Telegraf using the provided commands in the Cloud Web UI Open a new SSH console window to continue with the exercises.
  • 36. - Agent Configuration - Enable Plugins - Filtering - Dual Writes - Telegraf Gateway - Debugging Exercise
  • 38. © 2021 InfluxData. All rights reserved. 38 Exercise 1 : Agent Configuration [agent] interval = "10s" round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = "0s" flush_interval = "10s" flush_jitter = "0s" precision = "" debug = false quiet = false logfile = "" hostname = "" omit_hostname = false Objective: To understand Agent Configuration and make changes to collection and flush behaviour.
  • 40. © 2021 InfluxData. All rights reserved. 40 Exercise 2 : Enabling Input Plugin Objective: To enable plugin into your current telegraf configuration file and see the results in your InfluxDB Cloud platform. [global_tags] ... [agent] ... [[outputs.influxdb_v2]] ... [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] [[inputs.mem]] [[inputs.net]] [[inputs.processes]] [[inputs.swap]] [[inputs.system]] [[inputs.http_response]] urls = ["https://www.influxdata.com/"]
  • 42. © 2021 InfluxData. All rights reserved. 42 Exercise 3 : Filtering Data Objective: To filter data streaming from “mem” plugin to only pass “field” data that has exact description called “available_percent” and “used_percent”. [global_tags] ... [agent] ... [[outputs.influxdb_v2]] ... ... [[inputs.mem]] fieldpass = [“ available_percent”, “ used_percent”] ...
  • 44. © 2021 InfluxData. All rights reserved. 44 Exercise 4 : Dual Write Objective: Write data to two different outputs. One output to write to your InfluxDB Cloud and the other one will write to a file in your local system. Copy the output plugin configuration that we post in the Slack channel. [global_tags] ... [agent] ... [[outputs.influxdb_v2]] urls = ["https://eu-central-1-1.aws.cloud2.influxdata.com"] ## Token for authentication. token = "$INFLUX_TOKEN" ... [[outputs.file]] files = ["/tmp/metrics.out"] data_format = "influx" ... [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] ...
  • 46. © 2021 InfluxData. All rights reserved. 46 Exercise 5 : Telegraf Gateway Objective: Instead of send the data directly to InfluxDB Cloud, we going to send to a Telegraf Gateway. Add a new output in your telegraf configuration to point to the Gateway. Copy the output code that we posted in the Slack channel. [global_tags] ... [agent] ... [[outputs.influxdb]] urls = ["http://3.135.207.41:8086"] database_tag = "telegraf" exclude_database_tag = true [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] ...
  • 48. © 2021 InfluxData. All rights reserved. 48 Exercise 6 : Debugging Objective: Let’s see how can we debug what’s going on with our Telegraf Instance. Run journalctl -u telegraf Enable the Debug parameter, change, save and restart Telegraf. Run journalctl -u telegraf [global_tags] ... [agent] debug = true ... [[outputs.influxdb]] ... [[inputs.cpu]] [[inputs.disk]] [[input.diskio]] ...
  • 49. © 2020 InfluxData. All rights reserved. 49 Telegraf Configuration /home/centos/lab/exercise [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[outputs.influxdb]]
  • 50. © 2020 InfluxData. All rights reserved. 50 Exercise 1 - Lay of the land Inspect the output of a given configuration $ telegraf --test --config exercise Wait up to this many seconds for service $ telegraf --test-wait 10 --config exercise Gather metrics once, write them, and exit $ telegraf --once --config exercise
  • 51. © 2020 InfluxData. All rights reserved. 51 Inspect the output of a given configuration $ telegraf --test --config exercise [centos@influxdays lab]$ date; telegraf --test --config exercise ; date Fri Nov 6 22:29:46 UTC 2020 2020-11-06T22:29:46Z I! Starting Telegraf 1.16.1 > mem,host=influxdays active=151506944i,available=732938240i,available_percent=72.28993540203045,buffered=211 7632i,cached=249655296i,commit_limit=506941440i,committed_as=399839232i,dirty=16384i,fr ee=636620800i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,hug e_pages_total=0i,inactive=140464128i,low_free=0i,low_total=0i,mapped=38670336i,page_tab les=4157440i,shared=13058048i,slab=46571520i,sreclaimable=23306240i,sunreclaim=23265280 i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=125493248i,used_perc ent=12.377439593424661,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmal loc_used=6332416i,write_back=0i,write_back_tmp=0i 1604701786000000000 Fri Nov 6 22:29:46 UTC 2020
  • 52. © 2020 InfluxData. All rights reserved. 52 Wait up to this many seconds for service $ telegraf --test-wait 10 --config exercise $ date; telegraf --test-wait 10 --config exercise; date Fri Nov 6 22:30: 10 UTC 2020 2020-11-06T22:30:10Z I! Starting Telegraf 1.16.1 > mem,host=influxdays active=155013120i,available=728371200i,available_percent=71.8394867713538 9,buffered=2117632i,cached=249663488i,commit_limit=506941440i,committed_a s=403222528i,dirty=16384i,free=632045568i,high_free=0i,high_total=0i,huge _page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,inactive=14044 7744i,low_free=0i,low_total=0i,mapped=39026688i,page_tables=4534272i,shar ed=13058048i,slab=46686208i,sreclaimable=23306240i,sunreclaim=23379968i,s wap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=130060288 i,used_percent=12.827888224101224,vmalloc_chunk=35184362651648i,vmalloc_t otal=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0 i 1604701810000000000 Fri Nov 6 22:30: 20 UTC 2020
  • 53. © 2020 InfluxData. All rights reserved. 53 Write once on start up $ telegraf --once --config exercise $ date; telegraf --once --config exercise; date Fri Nov 6 22:31:51 UTC 2020 2020-11-06T22:31:51Z I! Starting Telegraf 1.16.1 2020-11-06T22:31:51Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:31:51Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:31:51Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:31:51Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:31:51Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics Fri Nov 6 22:31:51 UTC 2020
  • 54. © 2020 InfluxData. All rights reserved. 54 Mem
  • 55. © 2020 InfluxData. All rights reserved. 55 Exercise 2 Setup for Debugging
  • 56. © 2020 InfluxData. All rights reserved. 56 Telegraf Configuration File [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[outputs.influxdb]] $ telegraf --once --config exercise 2020-11-06T22:34:07Z I! Starting Telegraf 1.16.1 2020-11-06T22:34:07Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:07Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:34:07Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:07Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:34:07Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics
  • 57. © 2020 InfluxData. All rights reserved. 57 $ telegraf --debug --once --config exercise 2020-11-06T22:34:42Z I! Starting Telegraf 1.16.1 2020-11-06T22:34:42Z D! [agent] Initializing plugins 2020-11-06T22:34:42Z D! [agent] Connecting outputs 2020-11-06T22:34:42Z D! [agent] Attempting connection to [outputs.influxdb] 2020-11-06T22:34:42Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:42Z D! [agent] Successfully connected to outputs.influxdb 2020-11-06T22:34:42Z D! [agent] Starting service inputs 2020-11-06T22:34:42Z D! [agent] Stopping service inputs 2020-11-06T22:34:42Z D! [agent] Input channel closed 2020-11-06T22:34:42Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:34:42Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:34:42Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics 2020-11-06T22:34:42Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:34:42Z D! [agent] Stopped Successfully 2020-11-06T22:34:42Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics
  • 58. © 2020 InfluxData. All rights reserved. 58 telegraf --once --config exercise 2020-11-06T22:35:58Z I! Starting Telegraf 1.16.1 2020-11-06T22:35:58Z D! [agent] Initializing plugins 2020-11-06T22:35:58Z D! [agent] Connecting outputs 2020-11-06T22:35:58Z D! [agent] Attempting connection to [outputs.influxdb] 2020-11-06T22:35:58Z W! [outputs.influxdb] When writing to [http://localhost:8086]: database "telegraf" creation failed: Post "http://localhost:8086/query": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:35:58Z D! [agent] Successfully connected to outputs.influxdb 2020-11-06T22:35:58Z D! [agent] Starting service inputs 2020-11-06T22:35:58Z D! [agent] Stopping service inputs 2020-11-06T22:35:58Z D! [agent] Input channel closed 2020-11-06T22:35:58Z I! [agent] Hang on, flushing any cached metrics before shutdown 2020-11-06T22:35:58Z E! [outputs.influxdb] When writing to [http://localhost:8086]: Post "http://localhost:8086/write?db=telegraf": dial tcp [::1]:8086: connect: connection refused 2020-11-06T22:35:58Z D! [outputs.influxdb] Buffer fullness: 1 / 10000 metrics 2020-11-06T22:35:58Z E! [agent] Error writing to outputs.influxdb: could not write any address 2020-11-06T22:35:58Z D! [agent] Stopped Successfully 2020-11-06T22:35:58Z E! [telegraf] Error running agent: output plugins unable to send 1 metrics Updated Telegraf Configuration File [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[outputs.influxdb]]
  • 59. © 2020 InfluxData. All rights reserved. 59 Mem Stdout
  • 60. © 2020 InfluxData. All rights reserved. 60 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[outputs.influxdb]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[outputs.file]] $ telegraf --once --config exercise Output on next slide...
  • 61. © 2020 InfluxData. All rights reserved. 61 $ telegraf --once --config exercise 2020-11-06T22:37:05Z I! Starting Telegraf 1.16.1 2020-11-06T22:37:05Z D! [agent] Initializing plugins 2020-11-06T22:37:05Z D! [agent] Connecting outputs 2020-11-06T22:37:05Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:37:05Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:37:05Z D! [agent] Starting service inputs 2020-11-06T22:37:05Z D! [agent] Stopping service inputs 2020-11-06T22:37:05Z D! [agent] Input channel closed 2020-11-06T22:37:05Z I! [agent] Hang on, flushing any cached metrics before shutdown mem,host=influxdays sreclaimable=23232512i,page_tables=4255744i,shared=13058048i,swap_cached=0i,vmalloc_total=351843720878 08i,write_back_tmp=0i,dirty=20480i,low_free=0i,huge_pages_free=0i,committed_as=403091456i,free=6308044 80i,sunreclaim=23289856i,swap_free=0i,cached=250294272i,huge_pages_total=0i,write_back=0i,total=101388 6976i,active=156626944i,slab=46522368i,vmalloc_chunk=35184362651648i,vmalloc_used=6332416i,used=130670 592i,inactive=140836864i,available_percent=71.78292819889226,buffered=2117632i,available=727797760i,us ed_percent=12.88808270479253,high_total=0i,huge_page_size=2097152i,low_total=0i,mapped=40583168i,swap_ total=0i,commit_limit=506941440i,high_free=0i 1604702225000000000 2020-11-06T22:37:05Z D! [outputs.file] Wrote batch of 1 metrics in 92.731µs 2020-11-06T22:37:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2020-11-06T22:37:05Z D! [agent] Stopped Successfully
  • 62. © 2020 InfluxData. All rights reserved. 62 Exercise 3 Enabling a plugin
  • 63. © 2020 InfluxData. All rights reserved. 63 Mem Net Stdout
  • 64. © 2020 InfluxData. All rights reserved. 64 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" [[inputs.mem]] [[inputs.net]] [[outputs.file]]
  • 65. © 2020 InfluxData. All rights reserved. 65 Validate using --test $ telegraf --test --config exercise $ telegraf --test --config exercise 2020-11-06T22:38:39Z I! Starting Telegraf 1.16.1 2020-11-06T22:38:39Z D! [agent] Initializing plugins 2020-11-06T22:38:39Z D! [agent] Starting service inputs > net,host=influxdays,interface=ens5 bytes_recv=1708843i,bytes_sent=2007423i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=7752i,packets_sent=5597i 1604702319000000000 > net,host=influxdays,interface=all icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_i nerrors=0i,icmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps= 0i,icmp_intimestamps=0i,icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0 i,icmp_outerrors=0i,icmp_outmsgs=2i,icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_ou ttimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagra ms=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_inaddrerrors=0i,ip_indelivers=7981i,ip_indiscards=0i,ip_inhdrerrors=0i, ip_inreceives=7986i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnoroutes=16i,ip_outrequests=5516i,ip_reasmfails=0i,ip_reasmoks =0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tcp_currestab=4i,tcp_estabresets=4i,tcp_incsum errors=5i,tcp_inerrs=5i,tcp_insegs=7733i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5563i,tcp_passiveopens=68i,tcp_retranssegs= 43i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=312i,udp_inerrors=0i,udp_noports= 2i,udp_outdatagrams=315i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_inerrors =0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702319000000000 2020-11-06T22:38:39Z D! [agent] Stopping service inputs 2020-11-06T22:38:39Z D! [agent] Input channel closed 2020-11-06T22:38:39Z D! [agent] Stopped Successfully > mem,host=influxdays active=159830016i,available=723709952i,available_percent=71.37974637520149,buffered=2117632i,cached=250327040i,commit_limit=5069 41440i,committed_as=407773184i,dirty=8192i,free=626688000i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i ,huge_pages_total=0i,inactive=140804096i,low_free=0i,low_total=0i,mapped=40611840i,page_tables=4898816i,shared=13058048i,slab=46 534656i,sreclaimable=23240704i,sunreclaim=23293952i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=134754304i, used_percent=13.290860538680004,vmalloc_chunk=35184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i, write_back_tmp=0i 1604702319000000000
  • 66. © 2020 InfluxData. All rights reserved. 66 $ telegraf --config exercise 2020-11-06T22:39:26Z I! Starting Telegraf 1.16.1 2020-11-06T22:39:26Z I! Loaded inputs: mem net 2020-11-06T22:39:26Z I! Loaded aggregators: 2020-11-06T22:39:26Z I! Loaded processors: 2020-11-06T22:39:26Z I! Loaded outputs: file 2020-11-06T22:39:26Z I! Tags enabled: host=influxdays 2020-11-06T22:39:26Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s 2020-11-06T22:39:26Z D! [agent] Initializing plugins 2020-11-06T22:39:26Z D! [agent] Connecting outputs 2020-11-06T22:39:26Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:39:26Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:39:26Z D! [agent] Starting service inputs mem,host=influxdays free=629272576i,huge_pages_free=0i,low_total=0i,commit_limit=506941440i,high_free=0i,huge_pages_total=0i,vmalloc_chunk=35184362651648i,used=132153344i,high_tot al=0i,shared=13062144i,sunreclaim=23162880i,vmalloc_total=35184372087808i,write_back_tmp=0i,inactive=140779520i,used_percent=13.034327013586177,swap_cached =0i,vmalloc_used=6332416i,write_back=0i,buffered=2117632i,cached=250343424i,committed_as=403091456i,swap_total=0i,total=1013886976i,available=726310912i,availa ble_percent=71.63627990029532,active=158572544i,dirty=4096i,page_tables=4214784i,slab=46395392i,huge_page_size=2097152i,low_free=0i,mapped=40656896i,sreclai mable=23232512i,swap_free=0i 1604702368000000000 net,host=influxdays,interface=ens5 drop_in=0i,drop_out=0i,bytes_sent=2037575i,bytes_recv=1723821i,packets_sent=5696i,packets_recv=7891i,err_in=0i,err_out=0i 1604702368000000000 net,host=influxdays,interface=all udplite_rcvbuferrors=0i,ip_outdiscards=0i,ip_inaddrerrors=0i,icmp_inechoreps=0i,icmp_outtimestampreps=0i,icmp_outerrors=0i,icmp_outechoreps=0i,icmp_outaddrmaskreps =0i,icmpmsg_outtype3=2i,udplite_noports=0i,udplite_incsumerrors=0i,udp_noports=2i,icmp_incsumerrors=0i,icmp_outechos=0i,icmp_outaddrmasks=0i,icmp_inechos=0i,icmp _outparmprobs=0i,ip_inhdrerrors=0i,ip_outnoroutes=16i,ip_defaultttl=64i,icmp_outdestunreachs=2i,udp_indatagrams=316i,tcp_currestab=1i,ip_fragfails=0i,ip_reasmtimeout=0 i,ip_reasmoks=0i,icmp_outredirects=0i,tcp_retranssegs=45i,tcp_passiveopens=69i,tcp_maxconn=-1i,tcp_inerrs=5i,ip_inreceives=8124i,ip_forwarding=2i,icmp_inmsgs=2i,icmp _indestunreachs=2i,icmpmsg_intype3=2i,ip_indiscards=0i,icmp_intimeexcds=0i,udp_incsumerrors=0i,tcp_outsegs=5655i,ip_forwdatagrams=0i,ip_fragcreates=0i,icmp_inredir ects=0i,icmp_outsrcquenchs=0i,icmp_inaddrmasks=0i,ip_indelivers=8119i,icmp_inparmprobs=0i,tcp_insegs=7867i,tcp_attemptfails=72i,udp_inerrors=0i,tcp_estabresets=4i,tc p_outrsts=68i,udp_rcvbuferrors=0i,ip_reasmfails=0i,ip_fragoks=0i,ip_reasmreqds=0i,icmp_outtimestamps=0i,icmp_intimestampreps=0i,icmp_inerrors=0i,tcp_incsumerrors=5i, udplite_inerrors=0i,ip_outrequests=5610i,icmp_insrcquenchs=0i,icmp_outmsgs=2i,icmp_outtimeexcds=0i,udp_sndbuferrors=0i,tcp_rtoalgorithm=1i,udp_outdatagrams=319i,u dplite_outdatagrams=0i,tcp_rtomin=200i,icmp_inaddrmaskreps=0i,udplite_sndbuferrors=0i,ip_inunknownprotos=0i,icmp_intimestamps=0i,tcp_rtomax=120000i,tcp_activeopen s=142i,udplite_indatagrams=0i 1604702368000000000 2020-11-06T22:39:28Z D! [outputs.file] Wrote batch of 3 metrics in 139.162µs 2020-11-06T22:39:28Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics Validate without --test
  • 67. © 2020 InfluxData. All rights reserved. 67 Filtering
  • 68. © 2021 InfluxData. All rights reserved. 68 © 2019 InfluxData. All rights reserved. 68 What Is An InfluxDB Series? First, some definitions: • Measurement: namespace for categorically alike metrics (~Table) • Tag: key-value pair used as metadata for measurable object – host=prod001 or app=cart or customer_id=00012 • Field: key-value pair representing a metric that will append new values over time – cpu_usage=35.5 or errors=42 or temp=65.0 • TagSet: unique combination of Tag key-value pairs • FieldSet: unique combination of Field key-value pairs
  • 69. © 2021 InfluxData. All rights reserved. 69 © 2019 InfluxData. All rights reserved. 69 What Is An InfluxDB Series? Line/Record: <Measurement>,<TagSet> <FieldSet> <timestamp> → cpu,host=prod001,region=us-west usage_user=35,usage_system=15 t0 Point: <Measurement>,<TagSet> <FieldKey> <timestamp> → cpu,host=prod001,region=us-west usage_user=35 t0 Series: <Measurement>,<TagSet> <FieldKey> → cpu,host=prod001,region=us-west usage_user
  • 70. © 2021 InfluxData. All rights reserved. 70 © 2019 InfluxData. All rights reserved. 70 Series Visualized (tree) Cardinality is total number of “leaf nodes”
  • 71. © 2020 InfluxData. All rights reserved. 71 Filtering metrics • Happens at plugin level • Specify which metrics and which parts of metrics the given plugin sees or doesn’t see • Given above, filtering is often used for “routing” • By Measurement: • namepass: passes only records with specified Measurement name • namedrop: drops only records with specified Measurement name • By Field: • fieldpass: pass specified Field • fielddrop: drop specified Field • By Tag (slightly different implementation): • tagpass or tagdrop: pass/drop whole record containing matched Tag • taginclude or tagexclude: keep/drop matching Tag (key) in record
  • 72. © 2020 InfluxData. All rights reserved. 72 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA free=85.0,cached=12.5,buffered=3.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA free=85.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA cached=85.0 <timestamp> mem,region=us-west-1,host=hostA,container=containerA buffered=85.0 <timestamp> namepass = “mem”
  • 73. © 2020 InfluxData. All rights reserved. 73 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0 ,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp> fieldpass = “*_system”
  • 74. © 2020 InfluxData. All rights reserved. 74 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0,usage_iowait=0.2,usage_irq=0.0 ,usage_irq=0.0,usage_nice=1.0,usage_steal=2.0,usage_softirq=2.5 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_iowait=0.2 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_irq=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_nice=1.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_steal=2.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_softirq=2.5 <timestamp> fielddrop = “*_softirq”
  • 75. © 2020 InfluxData. All rights reserved. 75 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp> tagpass: host = [“hostA”]
  • 76. © 2020 InfluxData. All rights reserved. 76 How to think of this Telegraf metric: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,host=hostA,container=containerA usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostA,container=containerA usage_idle=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_user=35.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_system=15.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,host=hostB,container=containerB usage_idle=35.0 <timestamp> tagdrop: host = [“hostB”]
  • 77. © 2020 InfluxData. All rights reserved. 77 How to think of this Telegraf metric: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp> taginclude: host = [“host”]
  • 78. © 2020 InfluxData. All rights reserved. 78 How to think of this Telegraf metric: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0,usage_system=15.0,usage_guest=0.0,usage_guest_nice=0.0,usage_idle=35.0 <timestamp> How InfluxDB sees it: cpu,region=us-west-1,container=containerA,host=hostA usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerA,host=hostA usage_idle=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_user=35.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_system=15.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_guest_nice=0.0 <timestamp> cpu,region=us-west-1,container=containerB,host=hostB usage_idle=35.0 <timestamp> tagexclude: host = [“host”]
  • 79. © 2020 InfluxData. All rights reserved. 79 Exercise 4 Filtering
  • 80. © 2020 InfluxData. All rights reserved. 80 Sample Data mem,host=influxdays available=735477760i,inactive=167567360i,page_tables=4345856i,sreclaimable=22962176i,committed_as=402309120i,slab=46772224i,vmalloc_total=35184 372087808i,cached=243228672i,high_free=0i,huge_page_size=2097152i,huge_pages_total=0i,write_back=0i,total=1013886976i,dirty=0i,free=645414912i,lo w_total=0i,sunreclaim=23810048i,swap_cached=0i,huge_pages_free=0i,low_free=0i,shared=13058048i,swap_total=0i,used=123125760i,used_percent=12.1 43933487118785,buffered=2117632i,vmalloc_used=6332416i,available_percent=72.54040908007482,high_total=0i,vmalloc_chunk=35184362651648i,active =113328128i,commit_limit=506941440i,mapped=40271872i,swap_free=0i,write_back_tmp=0i 1602630666000000000 net,host=influxdays,interface=ens5 packets_recv=11760i,err_in=0i,err_out=0i,drop_in=0i,drop_out=0i,bytes_sent=1730127i,bytes_recv=1082190i,packets_sent=10073i 1602630666000000000 net,host=influxdays,interface=all icmp_outechos=0i,icmp_inechoreps=0i,tcp_rtoalgorithm=1i,tcp_insegs=9422i,ip_inunknownprotos=0i,ip_indiscards=0i,ip_reasmreqds=0i,icmp_outdestunreac hs=2i,icmpmsg_outtype3=2i,icmp_outaddrmasks=0i,icmp_outredirects=0i,icmp_indestunreachs=3i,icmp_incsumerrors=0i,udp_incsumerrors=0i,ip_inhdrerrors =0i,ip_reasmoks=0i,tcp_maxconn=-1i,udp_sndbuferrors=0i,tcp_incsumerrors=0i,tcp_estabresets=21i,udplite_rcvbuferrors=0i,ip_reasmtimeout=0i,icmp_outerr ors=0i,icmp_intimeexcds=0i,udplite_sndbuferrors=0i,icmpmsg_intype3=3i,tcp_activeopens=188i,tcp_rtomax=120000i,udp_outdatagrams=991i,udplite_incsum errors=0i,ip_fragcreates=0i,ip_fragoks=0i,icmp_inerrors=0i,tcp_rtomin=200i,udp_rcvbuferrors=0i,udp_noports=2i,ip_inaddrerrors=0i,ip_reasmfails=0i,ip_forwar ding=2i,icmp_outechoreps=0i,icmp_intimestamps=0i,icmp_outmsgs=2i,tcp_outrsts=141i,udp_indatagrams=932i,ip_inreceives=10233i,ip_outdiscards=0i,icmp _intimestampreps=0i,icmp_outtimestamps=0i,icmp_outsrcquenchs=0i,tcp_passiveopens=269i,tcp_currestab=1i,udplite_indatagrams=0i,ip_outnoroutes=16i,ic mp_outtimestampreps=0i,icmp_outparmprobs=0i,icmp_inaddrmasks=0i,icmp_outaddrmaskreps=0i,icmp_inparmprobs=0i,ip_defaultttl=64i,ip_outrequests=836 7i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_outtimeexcds=0i,icmp_inmsgs=3i,icmp_inaddrmaskreps=0i,ip_forwdatagrams=0i,ip_fragfails=0i,icmp_inec hos=0i,tcp_inerrs=0i,tcp_attemptfails=175i,tcp_outsegs=7517i,udplite_inerrors=0i,udplite_noports=0i,ip_indelivers=10227i,tcp_retranssegs=118i,udp_inerrors =0i,udplite_outdatagrams=0i 1602630666000000000
  • 81. © 2020 InfluxData. All rights reserved. 81 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[inputs.net]] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] Pass fields ending in “percent” exact named “total” [[inputs.net]] Pass fields Beginning with “bytes” Pass metrics where interface begins with “en” [[outputs.file]]
  • 82. © 2020 InfluxData. All rights reserved. 82 Exercise 4 Filtering Solution
  • 83. © 2020 InfluxData. All rights reserved. 83 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] [[inputs.net]] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] ...
  • 84. © 2020 InfluxData. All rights reserved. 84 Correct Placement [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] ...
  • 85. © 2020 InfluxData. All rights reserved. 85 Before & After $ telegraf --test --config exercise > net,host=influxdays,interface= ens5 bytes_recv=1739020i,bytes_sent=2088482i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=8042i,packets_sent=5843i 1604702475000000000 > mem,host=influxdays active=154382336i,available=730259456i,available_percent=72.02572607067398,buffered=2117632i,cached=250368000i,commit_limit=506941440i,commi tted_as=403091456i,dirty=20480i,free=633192448i,high_free=0i,high_total=0i,huge_page_size=2097152i,huge_pages_free=0i,huge_pages_total=0i,in active=140689408i,low_free=0i,low_total=0i,mapped=40558592i,page_tables=4190208i,shared=13058048i,slab=46452736i,sreclaimable=23232512i,sunr eclaim=23220224i,swap_cached=0i,swap_free=0i,swap_total=0i,total=1013886976i,used=128208896i,used_percent=12.645284833010814,vmalloc_chunk=3 5184362651648i,vmalloc_total=35184372087808i,vmalloc_used=6332416i,write_back=0i,write_back_tmp=0i 1604702475000000000 2020-11-06T22:41:14Z D! [agent] Stopping service inputs 2020-11-06T22:41:14Z D! [agent] Input channel closed 2020-11-06T22:41:14Z D! [agent] Stopped Successfully > net,host=influxdays,interface= all icmp_inaddrmaskreps=0i,icmp_inaddrmasks=0i,icmp_incsumerrors=0i,icmp_indestunreachs=2i,icmp_inechoreps=0i,icmp_inechos=0i,icmp_inerrors=0i,i cmp_inmsgs=2i,icmp_inparmprobs=0i,icmp_inredirects=0i,icmp_insrcquenchs=0i,icmp_intimeexcds=0i,icmp_intimestampreps=0i,icmp_intimestamps=0i, icmp_outaddrmaskreps=0i,icmp_outaddrmasks=0i,icmp_outdestunreachs=2i,icmp_outechoreps=0i,icmp_outechos=0i,icmp_outerrors=0i,icmp_outmsgs=2i, icmp_outparmprobs=0i,icmp_outredirects=0i,icmp_outsrcquenchs=0i,icmp_outtimeexcds=0i,icmp_outtimestampreps=0i,icmp_outtimestamps=0i,icmpmsg_ intype3=2i,icmpmsg_outtype3=2i,ip_defaultttl=64i,ip_forwarding=2i,ip_forwdatagrams=0i,ip_fragcreates=0i,ip_fragfails=0i,ip_fragoks=0i,ip_ina ddrerrors=0i,ip_indelivers=8268i,ip_indiscards=0i,ip_inhdrerrors=0i,ip_inreceives=8273i,ip_inunknownprotos=0i,ip_outdiscards=0i,ip_outnorout es=16i,ip_outrequests=5747i,ip_reasmfails=0i,ip_reasmoks=0i,ip_reasmreqds=0i,ip_reasmtimeout=0i,tcp_activeopens=142i,tcp_attemptfails=72i,tc p_currestab=1i,tcp_estabresets=4i,tcp_incsumerrors=5i,tcp_inerrs=5i,tcp_insegs=8010i,tcp_maxconn=-1i,tcp_outrsts=68i,tcp_outsegs=5792i,tcp_p assiveopens=74i,tcp_retranssegs=47i,tcp_rtoalgorithm=1i,tcp_rtomax=120000i,tcp_rtomin=200i,udp_incsumerrors=0i,udp_indatagrams=322i,udp_iner rors=0i,udp_noports=2i,udp_outdatagrams=325i,udp_rcvbuferrors=0i,udp_sndbuferrors=0i,udplite_incsumerrors=0i,udplite_indatagrams=0i,udplite_ inerrors=0i,udplite_noports=0i,udplite_outdatagrams=0i,udplite_rcvbuferrors=0i,udplite_sndbuferrors=0i 1604702475000000000 …. After filtering... …. > mem,host=influxdays available_percent=71.50255927540388,total=1013886976i,used_percent=13.168451628280902 1604702498000000000 > net,host=influxdays,interface= ens5 bytes_recv=1745105i,bytes_sent=2111895i 1604702498000000000
  • 86. © 2020 InfluxData. All rights reserved. 86 Placement Matters, Indentation Does Not [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] [inputs.net.tagpass] interface = ["en*"] fieldpass = ["bytes*"] [[outputs.file]] > net,host=influxdays,interface=ens5 bytes_recv=2082848i,bytes_sent=3148789i,drop_in=0i,drop_out=0i,err_in=0i,err_out=0i,packets_recv=21247 i,packets_sent=19323i 1604702498000000000
  • 87. © 2020 InfluxData. All rights reserved. 87 Exercise 5 Dual Write
  • 88. © 2020 InfluxData. All rights reserved. 88 Mem Net net /tmp/net m em /tmp/mem
  • 89. © 2020 InfluxData. All rights reserved. 89 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] Pass the “mem” data to file /tmp/mem [[outputs.file]] Pass the “net” data to file /tmp/net
  • 90. © 2020 InfluxData. All rights reserved. 90 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] namepass = ["mem"] files = ["/tmp/mem"] [[outputs.file]] namepass = ["net"] files = ["/tmp/net"]
  • 91. © 2020 InfluxData. All rights reserved. 91 Exercise 6 Listener
  • 92. © 2020 InfluxData. All rights reserved. 92 Mem Net ne t /tmp/net.txt m em /tmp/mem.txt :8086 Add
  • 93. © 2020 InfluxData. All rights reserved. 93 telegraf-listener.conf [global_tags] [agent] interval = "2s" flush_interval = "2s" round_interval = true metric_batch_size = 5000 metric_buffer_limit = 50000 debug = true logfile = "/tmp/telegraf-aggregator.log" [[inputs.influxdb_listener]] Set the listener port to ":8086" read_timeout = "2s" write_timeout = "2s" [[outputs.file]]
  • 94. © 2020 InfluxData. All rights reserved. 94 telegraf-listener.conf [global_tags] [agent] interval = "2s" flush_interval = "2s" round_interval = true metric_batch_size = 5000 metric_buffer_limit = 50000 debug = true logfile = "/tmp/telegraf-aggregator.log" [[inputs.influxdb_listener]] service_address = ": 8086" read_timeout = "2s" write_timeout = "2s" [[outputs.file]]
  • 95. © 2020 InfluxData. All rights reserved. 95 Launch Telegraf Listener as background job $ telegraf --config telegraf-listener.conf & [1] 2219 [centos@influxdays ~]$ 2020-11-06T22:44:22Z I! Starting Telegraf 1.16.1 [centos@influxdays ~]$ [centos@influxdays ~]$ [centos@influxdays ~]$
  • 96. © 2020 InfluxData. All rights reserved. 96 Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] namepass = ["mem"] files = ["/tmp/mem"] [[outputs.file]] namepass = ["net"] files = ["/tmp/net"] Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true [[inputs.mem]] fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]] namepass = ["mem"] files = ["/tmp/mem"] [[outputs.file]] namepass = ["net"] files = ["/tmp/net"] [[outputs.influxdb]]
  • 97. © 2020 InfluxData. All rights reserved. 97 Launch Telegraf $ telegraf -config exercise 2020-11-06T22:45:09Z I! Starting Telegraf 1.16.1 2020-11-06T22:45:09Z I! Loaded inputs: mem net 2020-11-06T22:45:09Z I! Loaded aggregators: 2020-11-06T22:45:09Z I! Loaded processors: 2020-11-06T22:45:09Z I! Loaded outputs: file (2x) influxdb 2020-11-06T22:45:09Z I! Tags enabled: host=influxdays 2020-11-06T22:45:09Z I! [agent] Config: Interval:2s, Quiet:false, Hostname:"influxdays", Flush Interval:2s 2020-11-06T22:45:09Z D! [agent] Initializing plugins 2020-11-06T22:45:09Z D! [agent] Connecting outputs 2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.file] 2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.file 2020-11-06T22:45:09Z D! [agent] Attempting connection to [outputs.influxdb] 2020-11-06T22:45:09Z D! [agent] Successfully connected to outputs.influxdb 2020-11-06T22:45:09Z D! [agent] Starting service inputs 2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 101.011µs 2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2020-11-06T22:45:11Z D! [outputs.file] Wrote batch of 1 metrics in 23.33µs 2020-11-06T22:45:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2020-11-06T22:45:11Z D! [outputs.influxdb] Wrote batch of 2 metrics in 1.205233ms 2020-11-06T22:45:11Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766 1604702710000000000 net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000
  • 98. © 2020 InfluxData. All rights reserved. 98 Inspect the filesystem $ cat /tmp/mem mem,host=influxdays used_percent=15.157293429913828,total=1013886976i,available_percent=69.51331348396766 1604702710000000000 $ cat /tmp/net net,host=influxdays,interface=ens5 bytes_sent=2176309i,bytes_recv=1776595i 1604702710000000000
  • 99. © 2020 InfluxData. All rights reserved. 99 telegraf-listener.conf [global_tags] [agent] interval = "2s" flush_interval = "2s" round_interval = true metric_batch_size = 5000 metric_buffer_limit = 50000 debug = true logfile = "/tmp/telegraf-aggregator.log" [[inputs.influxdb_listener]] service_address = ":8086" read_timeout = "2s" write_timeout = "2s" [[outputs.file]]
  • 100. © 2020 InfluxData. All rights reserved. 100 Exercise 7 Starlark
  • 101. © 2020 InfluxData. All rights reserved. 101 Finish [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[processors.starlark]] source = ''' def apply(metric): for k, v in metric.fields.items(): if type(v) == "float" or type(v) == "int": metric.fields[k] = v * 1000 return metric ''' [[outputs.file]] Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] .. .. [[outputs.file]]
  • 102. © 2020 InfluxData. All rights reserved. 102 Launch Telegraf Without Starlark processor > net,host=influxdays,interface=ens5 bytes_recv=9566941i,bytes_sent=23302741i 1604939494000000000 After multiplying by 1000 > net,host=influxdays,interface=ens5 bytes_recv=9555668000i,bytes_sent=23274132000i 1604939466000000000
  • 103. © 2020 InfluxData. All rights reserved. 103 Exercise 8 JSON
  • 104. © 2020 InfluxData. All rights reserved. 104 Mem Net net /tmp/net.txt m em /tmp/mem.txt HTTP
  • 105. © 2020 InfluxData. All rights reserved. 105 Add [[inputs.http]] urls = ["https://api.mocki.io/v1/29dfca5b"] data_format = “json” [[outputs.file]] Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]]
  • 106. © 2020 InfluxData. All rights reserved. 106 Add [[inputs.http]] urls = ["https://api.mocki.io/v1/29dfca5b"] data_format = “json” name_override = “apps” json_query = “apps” tag_keys = [“name”,”id”] [[outputs.file]] Start [global_tags] [agent] interval = "2s" flush_interval = "2s" debug = true #[[inputs.mem]] # fieldpass = ["*percent","total"] [[inputs.net]] fieldpass = ["bytes*"] [inputs.net.tagpass] interface = ["en*"] [[outputs.file]]
  • 107. © 2021 InfluxData. All rights reserved. 107 Telegraf Resources Blogs https:/ /www.influxdata.com/blog/using-a-telegraf-gateway/ https:/ /www.influxdata.com/blog/telegraf-go-collection-agent/ Docs https:/ /docs.influxdata.com/telegraf/latest Prebuilt dashboards https:/ /docs.influxdata.com/platform/monitoring/influxdata-platform/monitoring-dashb oards/
  • 109. Thank you See you tomorrow at InfluxDays EMEA 2021!
  • 110. © 2021 InfluxData. All rights reserved. 110 InfluxDays EMEA 2021 See you tomorrow: 18-19 May 2021