Flux and InfluxDB 2.0

Flux and InﬂuxDB 2.0
Paul Dix

@pauldix

paul@inﬂuxdata.com

• Data-scripting language

• Functional

• MIT Licensed

• Language & Runtime/Engine

• MIT Licensed

• Multi-tenanted

• Telegraf, InﬂuxDB, Chronograf, Kapacitor rolled into 1

• OSS single server

• Cloud usage based pricing

• Dedicated Cloud

• Enterprise on-premise

Long Live InﬂuxDB 2.0
(and Telegraf)

Consistent Documented API
Collection, Write/Query, Streaming & Batch Processing, Dashboards

Ofﬁcially Supported Client
Libraries
Go, Node.js, Ruby, Python, PHP, Java, C#, C, Kotlin

Ways to run Flux - (interpreter,
InﬂuxDB 1.7 & 2.0)

// get all data from the telegraf db
from(bucket:”telegraf/autogen”)
// filter that by the last hour
|> range(start:-1h)
// filter further by series with a specific measurement and field
|> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")

|> range(start:-1h)
Comments

|> range(start:-1h)
Named Arguments

|> range(start:-1h)
String Literals

|> range(start:-1h)
Buckets, not DBs

|> range(start:-1h)
Duration Literal

|> range(start:2018-11-07T00:00:00Z)
Time Literal

|> range(start:-1h)
Pipe forward operator

|> range(start:-1h)
Anonymous Function

|> range(start:-1h)
|> filter(fn: (r) => (r._measurement == "cpu" or r._measurement == “cpu")
and r.host == “serverA")
Predicate Function

// variables
some_int = 23
some_float = 23.2

// variables
some_int = 23
some_float = 23.2
some_string = “cpu"

// variables
some_int = 23
some_float = 23.2
some_duration = 1h

// variables
some_int = 23
some_float = 23.2
some_duration = 1h
some_time = 2018-10-10T19:00:00

// variables
some_int = 23
some_float = 23.2
some_duration = 1h
some_time = 2018-10-10T19:00:00
some_array = [1, 6, 20, 22]

// variables
some_int = 23
some_float = 23.2
some_duration = 1h
some_time = 2018-10-10T19:00:00
some_array = [1, 6, 20, 22]
some_object = {foo: "hello" bar: 22}

Data Model & Working with
Tables

Example Series
_measurement=mem,host=A,region=west,_field=free
_measurement=mem,host=B,region=west,_field=free
_measurement=cpu,host=A,region=west,_field=usage_system
_measurement=cpu,host=A,region=west,_field=usage_user

Example Series
Measurement

Example Series
Field

Table
_measurement host region _ﬁeld _time _value
mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Column

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Record

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Group Key

mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 10
Every record has
the same value!

Table Per Series
mem A west free 2018-06-14T09:15:00 10
mem A west free 2018-06-14T09:14:50 11
mem B west free 2018-06-14T09:15:00 20
mem B west free 2018-06-14T09:14:50 22
cpu A west usage_user 2018-06-14T09:15:00 45
cpu A west usage_user 2018-06-14T09:14:50 49
cpu A west usage_system 2018-06-14T09:15:00 35
cpu A west usage_system 2018-06-14T09:14:50 38

input tables -> function -> output tables

// example query
from(db:"telegraf")
|> range(start:2018-06-14T09:14:50, start:2018-06-14T09:15:01)
|> filter(fn: r => r._measurement == “mem" and
r._field == “free”)
|> sum()

What to sum on?
// example query
from(db:"telegraf")
|> sum()

Default columns argument
// example query
from(db:"telegraf")
|> sum(columns: [“_value”])

_meas
ureme
host region _ﬁeld _time _valu
e
mem A west free 2018-06-
14T09:1
10
14T09:1
11
_meas
ureme
emem B west free 2018-06-
14T09:15
20
mem B west free 2018-06-
14T09:14
22
Input in table form
// example query
from(db:"telegraf")
|> sum()

_meas
ureme
e
14T09:1
10
14T09:1
11
_meas
ureme
14T09:15
20
14T09:14
22
sum()
// example query
from(db:"telegraf")
|> sum()

// example query
from(db:"telegraf")
|> sum()
_meas
ureme
e
14T09:1
10
14T09:1
11
_meas
ureme
14T09:15
20
14T09:14
22
sum()
_meas
ureme
e
14T09:1
21
_meas
ureme
e
14T09:15
42

N to N table mapping
(1 to 1 mapping)

window
// example query
from(db:"telegraf")
|> range(start:2018-06-14T09:14:30, end:2018-06-14T09:15:01)
|> window(every:20s)
30s of data (4 samples)

window
// example query
from(db:"telegraf")
split into 20s windows

window
_meas host region _ﬁeld _time _valu
mem A west free …14:30 10
mem B west free …14:30 20
// example query
from(db:"telegraf")
Input

window
window(
every:20s)
// example query
from(db:"telegraf")

window
window(
every:20s)
// example query
from(db:"telegraf")
|> window(every:20s) _meas
ureme
emem A west free …14:30 10
_meas
ureme
emem B west free …14:50 23
_meas
ureme
_meas
ureme

window
window(
every:20s)
// example query
from(db:"telegraf")
|> window(every:20s) _meas
ureme
_meas
ureme
_meas
ureme
_meas
ureme
N to M tables

Window based on time
_start and _stop columns

group
// example query
from(db:"telegraf")
|> group(keys:[“region"])

group
// example query
from(db:"telegraf")
new group key

group
// example query
from(db:"telegraf")

group
group(
keys:
[“region”])
// example query
from(db:"telegraf")
_meas
ureme
N to M tables
M == cardinality(group keys)

Make Everyone a Data
Programmer!

Inputs
from, fromKafka, fromFile, fromS3, fromPrometheus, fromMySQL, etc.

import "mysql"
customers = mysql.from(connect: loadSecret(name:”mysql_prod"),
query: "select id, name from customers")
data = from(bucket: "my_data")
|> range(start: -4h)
|> filter(fn: (r) => r._measurement == “write_requests")
|> rename(columns: {customer_id: “id"})
join(tables: {customers, data}, on: ["id"])
|> yield(name: "results")

import "mysql"
customers = mysql.from(connect: loadSecret(name:"mysql_prod"),
Imports for sharing code!

import "mysql"
Pulling data from a non-InﬂuxDB source

import "mysql"
Raw query (for now)

import "mysql"
Loading Secret

import "mysql"
Renaming & Shaping Data

import "mysql"
Join on any column

Outputs
to, toKafka, toFile, toS3, toPrometheus, toMySQL, etc.

option task = {
name: “Alert on disk",
every: 5m,
}
crit = 90 // alert at this percentage
warn = 80 // warn at this percentage
data = from(bucket: "telegraf/autogen")
|> filter(fn: (r) => r._measurement == "disk" and r._field == "used_percent")
|> last()
data |> filter(fn: (r) => r._value > crit)
|> addColumn(key: "level", value: "critical")
|> addColumn(key: "alert", value: task.name)
|> to(bucket: "alerts")
data |> filter(fn: (r) => r._value > warn && r._value < crit)
|> addColumn(key: "level", value: "warn")

option task = {
every: 5m,
}
|> last()
Option syntax for tasks

option task = {
every: 5m,
}
|> last()
Get at the last value without specifying time range

option task = {
every: 5m,
}
|> last()
|> addColumn(key: "level", value: “critical")
Adding a column to decorate the data

option task = {
every: 5m,
}
|> last()
|> to(bucket: "alerts") To writes to the local InﬂuxDB

Separate Alerts From
Notiﬁcations!

option task = {name: "slack critical alerts", every: 1m}
import "slack"
lastNotificationTime = from(bucket: "notificatons")
|> filter(fn: (r) => r.level == "critical" and r._field == "alert_time")
|> group(none:true)
|> last()
|> recordValue(column:"_value")
from(bucket: "alerts")
|> range(start: lastNotificationTime)
|> filter(fn: (r) => r.level == "critical")
// shape the alert data to what we care about in notifications
|> renameColumn(from: "_time", to: "alert_time")
|> renameColumn(from: "_value", to: "used_percent")
// set the time the notification is being sent
|> addColumn(key: "_time", value: now())
// get rid of unneeded columns
|> drop(columns: ["_start", "_stop"])
// write the message
|> map(fn: (r) => r._value = "{r.host} disk usage is at {r.used_percent}%")
|> slack.to(config: loadSecret(name: “slack_alert_config”), message: “_value”)
|> to(bucket: “notifications")

import "slack"
|> group(none:true)
|> last()
|> filter(fn: (r) => r.level == “critical”)
|> slack.to(config: loadSecret(name: "slack_alert"))
We have state so we don’t resend

import "slack"
|> group(none:true)
|> last()
Use last time as argument to range

import "slack"
|> group(none:true)
|> last()
Now function for current time

import "slack"
|> group(none:true)
|> last()
Map function to iterate
over values

import "slack"
|> group(none:true)
|> last()
String interpolation

import "slack"
|> group(none:true)
|> last()
Send to Slack and
record in InﬂuxDB

option task = {
name: "email alert digest",
cron: "0 5 * * 0"
}
import "smtp"
body = ""
|> filter(fn: (r) => (r.level == "warn" or r.level == "critical") and r._field == "message")
|> group(by: ["alert"])
|> count()
|> group(none: true)
|> map(fn: (r) => body = body + "Alert {r.alert} triggered {r._value} timesn")
smtp.to(
config: loadSecret(name: "smtp_digest"),
to: "alerts@influxdata.com",
title: "Alert digest for {now()}",
body: message)

option task = {
cron: "0 5 * * 0"
}
import "smtp"
body = ""
|> count()
smtp.to(
body: message)
Cron syntax

option task = {
cron: "0 5 * * 0"
}
import "smtp"
body = ""
|> count()
smtp.to(
body: message)
Closures

Tasks run logs
(just another time series)

Bulk Import & Export
Specify bucket, range, predicate

Same API in OSS, Cloud, and
Enterprise

Thank you.
Paul Dix

@pauldix

paul@inﬂuxdata.com

Flux and InfluxDB 2.0

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Flux and InfluxDB 2.0

Similar to Flux and InfluxDB 2.0 (20)

More from InfluxData

More from InfluxData (20)

Recently uploaded

Recently uploaded (20)

Flux and InfluxDB 2.0