InfluxQL is a powerful query language for InfluxDB, and TICKScript is a domain specific language used by Kapacitor to define tasks involving the extraction, transformation and loading of data and also involving the tracking of arbitrary changes and detection of events within data. The combination of these two can make your monitoring apps powerful. During this session, InfluxData Engineer Michael DeSa will share best practices for using these powerful tools. Prerequisite: Intro To Kapacitor.
2. Michael Desa
Software Engineer,
InfluxData
InfluxQL & TICKscript
Michael Desa is a Software Engineer at InfluxData who
works on the 2.0 API for the InfluxData Platform. He has
led the InfluxDB training course across the US, providing
students with an in depth understanding of how InfluxDB
works as well as sharing best practices. He has a degree
in Math from the University of California, at Berkeley.
8. Select Statement with
WHERE clause
SELECT <field> FROM <measurement> WHERE <conditions>
SELECT * FROM cpu WHERE busy > 50
SELECT free FROM mem WHERE host = 'server1'
SELECT x + y FROM vars WHERE some_tag = 'some_key'
SELECT x,y FROM nums WHERE domain =~ /.*/
9. Select Statement with
Relative Time
SELECT <field> FROM <measurement> WHERE <time>
SELECT * FROM cpu WHERE time > now() - 1h
SELECT * FROM cpu WHERE time > now() - 10s
SELECT free FROM mem WHERE time > now() - 4d
SELECT x + y FROM vars WHERE time > now() - 10w
SELECT x,y FROM nums WHERE time > now() + 15m
10. Select Statement with
GROUP BY clause
[SELECT STATEMENT] GROUP BY <tag>
SELECT * FROM cpu GROUP BY host
SELECT * FROM cpu GROUP BY *
SELECT free FROM mem GROUP BY location, host
11. Select Statement with
GROUP BY clause
> SELECT * FROM h2o_quality GROUP BY location
name: h2o_quality
tags: location = coyote_creek
time index id
---- ----- ---
2015-08-18T00:00:00Z 41 1
2015-08-18T00:00:00Z 41 1
name: h2o_quality
tags: location = santa_monica
time index id
---- ----- ---
2015-08-18T00:00:00Z 99 2
2015-08-18T00:06:00Z 56 2
12. Select Statement with a
function
SELECT <function>(<field>) FROM <measurement>
SELECT count(value) FROM cpu
SELECT mean(free) FROM mem WHERE time > now() - 1h
SELECT sum(x) FROM vars WHERE x > 100
SELECT median(y) FROM nums WHERE domain = 'Z'
13. Select Statement with a
function
> SELECT count(index) FROM h2o_quality
WHERE location = 'coyote_creek'
name: h2o_quality
-----------------
time count
1970-01-01T00:00:00Z 12777
14. Select Statement with a
function
> SELECT max(usage_user) FROM cpu
WHERE time > now() - 10d
name: cpu
time max
---- ---
2018-11-05T22:12:05Z 54
15. Types of
Functions
● Aggregators
○ count
○ distinct
○ integral
○ mean
○ median
○ spread
○ sum
○ stddev
● Selectors
○ bottom
○ first
○ last
○ max
○ min
○ percentile
○ top
● Transformers
○ derivative
○ difference
○ moving_average
○ elapsed
16. Select Statement with
GROUP BY time clause
[SELECT STATEMENT] WHERE <time condition>
GROUP BY time(<period>)
SELECT max(busy) FROM cpu WHERE time > now() - 1h
GROUP BY time(10m)
SELECT mean(free) FROM free WHERE time > now() - 1d
GROUP BY time(1h)
17. Select Statement with
GROUP BY time clause
Invalid queries
SELECT busy FROM cpu WHERE time > now() - 1h
GROUP BY time(10m)
SELECT mean(busy) FROM cpu GROUP BY time(10m)
18. Select Statement with
GROUP BY time clause
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h)
name: average_temperature
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
19. Select Statement with
GROUP BY time and
tag
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h), location
name: average_temperature
tags: location = coyote_creek
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
name: average_temperature
tags: location = santa_monica
-------------------------
time mean
2015-09-18T00:00:00Z
2015-09-18T12:00:00Z 79.95033445378151
20. Select with fill
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h), location fill(<fill>)
name: average_temperature
tags: location = coyote_creek
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
name: average_temperature
tags: location = santa_monica
-------------------------
time mean
2015-09-18T00:00:00Z <fill>
2015-09-18T12:00:00Z 79.95033445378151
21. Select with fill 10
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h), location fill(10)
name: average_temperature
tags: location = coyote_creek
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
name: average_temperature
tags: location = santa_monica
-------------------------
time mean
2015-09-18T00:00:00Z 10
2015-09-18T12:00:00Z 79.95033445378151
22. Select with fill next
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h), location fill(next)
name: average_temperature
tags: location = coyote_creek
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
name: average_temperature
tags: location = santa_monica
-------------------------
time mean
2015-09-18T00:00:00Z 79.95033445378151
2015-09-18T12:00:00Z 79.95033445378151
23. Select with fill none
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h), location fill(none)
name: average_temperature
tags: location = coyote_creek
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
name: average_temperature
tags: location = santa_monica
-------------------------
time mean
2015-09-18T12:00:00Z 79.95033445378151
24. Select with fill null
> SELECT mean(degrees) FROM average_temperature
WHERE time < '2015-09-19'
AND time > '2015-09-18'
GROUP BY time(12h), location fill(null)
name: average_temperature
tags: location = coyote_creek
-------------------------
time mean
2015-09-18T00:00:00Z 79.83613445378151
2015-09-18T12:00:00Z 79.65034965034965
name: average_temperature
tags: location = santa_monica
-------------------------
time mean
2015-09-18T00:00:00Z
2015-09-18T12:00:00Z 79.95033445378151
29. Answer
> SELECT max(usage_user) FROM cpu
WHERE time > now() - 1d GROUP BY time(1d)
name: cpu
time max
---- ---
2018-11-07T00:00:00Z 37.37373737373738
2018-11-08T00:00:00Z 72
30. How to get
one value back
> SELECT max(usage_user) FROM cpu
WHERE time > now() - 1d
GROUP BY time(1d,now())
name: cpu
time max
---- ---
2018-11-07T15:50:38.560319Z 72
2018-11-08T15:50:38.560319Z
31. I only want one!!
> SELECT max(usage_user) FROM cpu
WHERE time > now() - 1d
GROUP BY time(1d,now()) fill(none)
name: cpu
time max
---- ---
2018-11-07T15:51:37.466919Z 72
32. Question
What happens?
// No data in the time range
> SELECT count(usage_user) FROM cpu
WHERE time > now() - 10s
37. Question
Write a query that
computes the mean of
free field for memory
and the mean of the
usage_user field for
cpu in 20s windows
cpu,host=A usage_user=10 10s
cpu,host=A usage_user=20 20s
cpu,host=A usage_user=30 30s
cpu,host=A usage_user=50 40s
mem,host=A free=10 10s
mem,host=A free=20 20s
mem,host=A free=123 30s
mem,host=A free=43 40s
39. Question
Write a query that
computes total rate of
change across all of
the counters grouped
by path
http_req,host=A,path=/ counter=10 10s
http_req,host=A,path=/ counter=11 20s
http_req,host=A,path=/ counter=0 30s
http_req,host=A,path=/ counter=9 40s
http_req,host=A,path=/home counter=20 10s
http_req,host=A,path=/home counter=21 20s
http_req,host=A,path=/home counter=24 30s
http_req,host=A,path=/home counter=25 40s
http_req,host=B,path=/admin counter=24 10s
http_req,host=B,path=/admin counter=30 20s
http_req,host=B,path=/admin counter=34 30s
http_req,host=B,path=/admin counter=34 40s
http_req,host=B,path=/home counter=20 10s
http_req,host=B,path=/home counter=25 20s
http_req,host=B,path=/home counter=28 30s
http_req,host=B,path=/home counter=30 40s
40. Answer
SELECT sum(rate) FROM (
SELECT non_negative_derivative(counter)
FROM http_req
WHERE <time range>
GROUP BY *
) GROUP BY time(10s), path
41. Answer
SELECT sum(rate) FROM (
SELECT non_negative_derivative(max(counter))
FROM http_req
WHERE <time range>
GROUP BY time(10s), *
) GROUP BY time(10s), path
42. Question
Write a query that
computes the ratio of
system load5 to cpu
usage_user
cpu,host=A usage_user=10 10s
cpu,host=A usage_user=20 20s
cpu,host=A usage_user=30 30s
cpu,host=A usage_user=50 40s
sys,host=A load5=10 10s
sys,host=A load5=20 20s
sys,host=A load5=34 30s
sys,host=A load5=43 40s
53. Question
Write a query that
computes the average
of the usage_user and
usage_system fields
grouped by host in 20
second intervals
cpu,host=A usage_user=10,usage_system=70 10s
cpu,host=A usage_user=20,usage_system=67 20s
cpu,host=A usage_user=30,usage_system=72 30s
cpu,host=A usage_user=50,usage_system=74 40s
cpu,host=B usage_user=14,usage_system=40 10s
cpu,host=B usage_user=43,usage_system=56 20s
cpu,host=B usage_user=64,usage_system=77 30s
cpu,host=B usage_user=69,usage_system=74 40s
cpu,host=C usage_user=10,usage_system=12 10s
cpu,host=C usage_user=12,usage_system=23 20s
cpu,host=C usage_user=13,usage_system=78 30s
cpu,host=C usage_user=15,usage_system=88 40s
54. Answer
var data = stream
|from()
.measurement('cpu')
.groupBy('host')
|window()
.period(20s)
.every(20s)
data
|mean('usage_user')
data
|mean('usage_system')
55. Not Possible
var data = stream
|from()
.measurement('cpu')
|window()
.period(20s)
.every(20s)
data
|mean(*) // Cant do this
56. Question
Write a query that
computes the mean of
free field for memory
and the mean of the
usage_user field for
cpu in 20s windows
cpu,host=A usage_user=10 10s
cpu,host=A usage_user=20 20s
cpu,host=A usage_user=30 30s
cpu,host=A usage_user=50 40s
mem,host=A free=10 10s
mem,host=A free=20 20s
mem,host=A free=123 30s
mem,host=A free=43 40s
60. Question
Write a query that
computes the ratio of
system load5 to cpu
usage_user
cpu,host=A usage_user=10 10s
cpu,host=A usage_user=20 20s
cpu,host=A usage_user=30 30s
cpu,host=A usage_user=50 40s
sys,host=A load5=10 10s
sys,host=A load5=20 20s
sys,host=A load5=34 30s
sys,host=A load5=43 40s
61. Answer
var cpu = stream
|from()
.measurement('cpu')
var sys = stream
|from()
.measurement('sys')
cpu
|join('sys')
.as('cpu', 'sys')
|eval(lambda: "cpu.usage_user" / "sys.load5")
67. Question
Write a query that
computes the mean of
free field for memory
and the mean of the
usage_user field for
cpu in 20s windows
cpu,host=A usage_user=10 10s
cpu,host=A usage_user=20 20s
cpu,host=A usage_user=30 30s
cpu,host=A usage_user=50 40s
mem,host=A free=10 10s
mem,host=A free=20 20s
mem,host=A free=123 30s
mem,host=A free=43 40s
71. Making that over
time
from(bucket: "mybucket")
|> range(start: -40s)
|> filter(fn: (r) => r._measurement == "http_req")
|> filter(fn: (r) => r._field == "counter")
|> derivative(non_negative: true)
|> group(by: ["path"])
|> window(period: 20s, every:20s)
|> sum()
72. Turning it into a
function so I never
have to think
about it again
rate =(table=<-, m, field="counter", by, interval) =>
|> filter(fn: (r) => r._measurement == m)
|> filter(fn: (r) => r._field == field)
|> derivative(non_negative: true)
|> group(by: by)
|> window(period: interval, every: interval)
|> sum()
from(bucket: "mybucket")
|> range(start: -40s)
|> rate(m: "http_req",
by: ["path"],
interval: 20s)
73. Question
Write a query that
computes the ratio of
system load5 to cpu
usage_user
cpu,host=A usage_user=10 10s
cpu,host=A usage_user=20 20s
cpu,host=A usage_user=30 30s
cpu,host=A usage_user=50 40s
sys,host=A load5=10 10s
sys,host=A load5=20 20s
sys,host=A load5=34 30s
sys,host=A load5=43 40s