Flux and InfluxDB 2.0

Flux and InﬂuxDB 2.0
Paul Dix

@pauldix

paul@inﬂuxdata.com

• Data-scripting language

• Functional

• MIT Licensed

• Language & Runtime/Engine

• MIT Licensed

• Multi-tenanted

• Telegraf, InﬂuxDB, Chronograf, Kapacitor rolled into 1

• OSS single server

• Cloud usage based pricing

• Dedicated Cloud

• Enterprise on-premise

Long Live InﬂuxDB 2.0
(and Telegraf)

Consistent Documented API
Collection, Write/Query, Streaming & Batch Processing, Dashboards

Ofﬁcially Supported Client
Libraries
Go, Node.js, Ruby, Python, PHP, Java, C#, C, Kotlin

Ways to run Flux - (interpreter,
InﬂuxDB 1.7 & 2.0)

// get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")

|> range(start:-1h)
Comments

|> range(start:-1h)
Named Arguments

|> range(start:-1h)
String Literals

|> range(start:-1h)
Buckets, not DBs

|> range(start:-1h)
Duration Literal

|> range(start:2018-11-07T00:00:00Z)
Time Literal

|> range(start:-1h)
Pipe forward operator

|> range(start:-1h)
Anonymous Function

|> range(start:-1h)
|> filter(fn: (r) => (r._measurement == "cpu" or r._measurement == “cpu")
and r.host == “serverA")
Predicate Function

// variables
some_int = 23
some_float = 23.2

// variables
some_int = 23
some_float = 23.2
some_string = “cpu"

// variables
some_int = 23
some_float = 23.2
some_duration = 1h

// variables
some_int = 23
some_float = 23.2
some_duration = 1h
some_time = 2018-10-10T19:00:00

// variables
some_int = 23
some_float = 23.2
some_duration = 1h
some_time = 2018-10-10T19:00:00
some_array = [1, 6, 20, 22]

// variables
some_int = 23
some_float = 23.2
some_duration = 1h
some_time = 2018-10-10T19:00:00
some_array = [1, 6, 20, 22]
some_object = {foo: "hello" bar: 22}

Data Model & Working with
Tables

Example Series
_measurement=mem,host=A,region=west,_field=free
_measurement=mem,host=B,region=west,_field=free
_measurement=cpu,host=A,region=west,_field=usage_system
_measurement=cpu,host=A,region=west,_field=usage_user

Example Series
Measurement

Example Series
Field

Table
_measurement host region _ﬁeld _time _value
mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Column

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Record

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Group Key

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Every record has
the same value!

Table Per Series
mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 11
mem B west free 2018-06-14T09:15:00 20
mem B west free 2018-06-14T09:14:50 22
cpu A west usage_user 2018-06-14T09:15:00 45
cpu A west usage_user 2018-06-14T09:14:50 49
cpu A west usage_system 2018-06-14T09:15:00 35
cpu A west usage_system 2018-06-14T09:14:50 38

input tables -> function -> output tables

// example query
from(db:"telegraf")
|> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01)
|> filter(fn: r => r._measurement == “mem" and
r._field == “free”)
|> sum()

What to sum on?
// example query
from(db:"telegraf")
|> sum()

Default columns argument
// example query
from(db:"telegraf")
|> sum(columns: [“_value”])

_meas
ureme
host region _ﬁeld _time _valu
e
mem A west free 2018-06-
14T09:1
10
14T09:1
11
_meas
ureme
emem B west free 2018-06-
14T09:15
20
mem B west free 2018-06-
14T09:14
22
Input in table form
// example query
from(db:"telegraf")
|> sum()

_meas
ureme
e
14T09:1
10
14T09:1
11
_meas
ureme
14T09:15
20
14T09:14
22
sum()
// example query
from(db:"telegraf")
|> sum()

// example query
from(db:"telegraf")
|> sum()
_meas
ureme
e
14T09:1
10
14T09:1
11
_meas
ureme
14T09:15
20
14T09:14
22
sum()
_meas
ureme
e
14T09:1
21
_meas
ureme
e
14T09:15
42

N to N table mapping
(1 to 1 mapping)

window
// example query
from(db:"telegraf")
|> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01)
|> window(every:20s)
30s of data (4 samples)

window
// example query
from(db:"telegraf")
split into 20s windows

window
_meas host region _ﬁeld _time _valu
mem A west free …14:30 10
mem B west free …14:30 20
// example query
from(db:"telegraf")
Input

window
window(
every:20s)
// example query
from(db:"telegraf")

window
window(
every:20s)
// example query
from(db:"telegraf")
|> window(every:20s) _meas
ureme
emem A west free …14:30 10
_meas
ureme
emem B west free …14:50 23
_meas
ureme
_meas
ureme

window
window(
every:20s)
// example query
from(db:"telegraf")
|> window(every:20s) _meas
ureme
_meas
ureme
_meas
ureme
_meas
ureme
N to M tables

Window based on time
_start and _stop columns

group
// example query
from(db:"telegraf")
|> group(keys:[“region"])

group
// example query
from(db:"telegraf")
new group key

group
// example query
from(db:"telegraf")

group
group(
keys:
[“region”])
// example query
from(db:"telegraf")
_meas
ureme
N to M tables
M == cardinality(group keys)

Make Everyone a Data
Programmer!

Inputs
from, fromKafka, fromFile, fromS3, fromPrometheus, fromMySQL, etc.

import "mysql"
customers = mysql.from(connect: loadSecret(name:”mysql_prod"),
query: "select id, name from customers")
data = from(bucket: "my_data")
|> range(start: -4h)
|> filter(fn: (r) => r._measurement == “write_requests")
|> rename(columns: {customer_id: “id"})
join(tables: {customers, data}, on: ["id"])
|> yield(name: "results")

import "mysql"
customers = mysql.from(connect: loadSecret(name:"mysql_prod"),
Imports for sharing code!

import "mysql"
Pulling data from a non-InﬂuxDB source

import "mysql"
Raw query (for now)

import "mysql"
Loading Secret

import "mysql"
Renaming & Shaping Data

import "mysql"
Join on any column

Outputs
to, toKafka, toFile, toS3, toPrometheus, toMySQL, etc.

option task = {
name: “Alert on disk",
every: 5m,
}
crit = 90 // alert at this percentage
warn = 80 // warn at this percentage
data = from(bucket: "telegraf/autogen")
|> filter(fn: (r) => r._measurement == "disk" and r._field == "used_percent")
|> last()
data |> filter(fn: (r) => r._value > crit)
|> addColumn(key: "level", value: "critical")
|> addColumn(key: "alert", value: task.name)
|> to(bucket: "alerts")
data |> filter(fn: (r) => r._value > warn && r._value < crit)
|> addColumn(key: "level", value: "warn")

option task = {
every: 5m,
}
|> last()
Option syntax for tasks

option task = {
every: 5m,
}
|> last()
Get at the last value without specifying time range

option task = {
every: 5m,
}
|> last()
|> addColumn(key: "level", value: “critical")
Adding a column to decorate the data

option task = {
every: 5m,
}
|> last()
|> to(bucket: "alerts") To writes to the local InﬂuxDB

Separate Alerts From
Notiﬁcations!

option task = {name: "slack critical alerts", every: 1m}
import "slack"
lastNotificationTime = from(bucket: "notificatons")
|> filter(fn: (r) => r.level == "critical" and r._field == "alert_time")
|> group(none:true)
|> last()
|> recordValue(column:"_value")
from(bucket: "alerts")
|> range(start: lastNotificationTime)
|> filter(fn: (r) => r.level == "critical")
// shape the alert data to what we care about in notifications
|> renameColumn(from: "_time", to: "alert_time")
|> renameColumn(from: "_value", to: "used_percent")
// set the time the notification is being sent
|> addColumn(key: "_time", value: now())
// get rid of unneeded columns
|> drop(columns: ["_start", "_stop"])
// write the message
|> map(fn: (r) => r._value = "{r.host} disk usage is at {r.used_percent}%")
|> slack.to(config: loadSecret(name: “slack_alert_config”), message: “_value”)
|> to(bucket: “notifications")

import "slack"
|> group(none:true)
|> last()
|> filter(fn: (r) => r.level == “critical”)
|> slack.to(config: loadSecret(name: "slack_alert"))
We have state so we don’t resend

import "slack"
|> group(none:true)
|> last()
Use last time as argument to range

import "slack"
|> group(none:true)
|> last()
Now function for current time

import "slack"
|> group(none:true)
|> last()
Map function to iterate
over values

import "slack"
|> group(none:true)
|> last()
String interpolation

import "slack"
|> group(none:true)
|> last()
Send to Slack and
record in InﬂuxDB

option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(by: ["alert"])
|> count()
|> group(none: true)
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)

option task = {
cron: "0 5 * * 0"
}
import "smtp"
body = ""
|> count()
smtp.to(
body: message)
Cron syntax

option task = {
cron: "0 5 * * 0"
}
import "smtp"
body = ""
|> count()
smtp.to(
body: message)
Closures

Tasks run logs
(just another time series)

Bulk Import & Export
Specify bucket, range, predicate

Same API in OSS, Cloud, and
Enterprise

Thank you.
Paul Dix

@pauldix

paul@inﬂuxdata.com

Flux and InfluxDB 2.0

More Related Content

What's hot

Similar to Flux and InfluxDB 2.0

More from InfluxData

Recently uploaded

Flux and InfluxDB 2.0