Introduction to Flux and Functional Data Scripting

Scott Anderson
Sr. Technical Writer & Tech Lead
April 2021
Introduction to Flux &
Functional Data Scripting

© 2021 InfluxData. All rights reserved.
2
What is Flux?
A functional data scripting
language designed to query,
process, and act on data.

3
Design Goals
– Turing-complete
– Useable
– Readable
– Flexible
– Extensible
– Testable
– Contributable
– Shareable

4
How does
it work?

5 © 2021 InfluxData. All rights reserved.
5

6

7

8

9

10

11
from(bucket: "system-data")
|> range(start: -1h)
|> filter(fn: (r) =>
r._measurement == "mem" and
r._field == "used_percent"
)
|> aggregateWindow(every: 5m, fn: mean)

12
Data
Structure

13

14

15
Stream of tables

16
Table

17
version userID sessionID _field _value
Table

18
v1 ex123 ex456 example 1.2
Table

19
Table
Group key: [ ]

20
Tables
Group key: ["version", "userID" ]

21
Tables
Group key: ["version", "userID" ]

22
sql.from(
driverName: "mysql",
dataSourceName: "${username}:${password}@tcp(localhost:3306)/db",
query:"SELECT CustomerName,City FROM Customers;"
)
CustomerName City
John Doe New York
Jane Seymour London
Rufus Peck San Francisco
Group key: [ ]

23
from(bucket: "example-bucket")
|> filter(fn: (r) => r._measurement == "m1" and r.field == "f1")
_start _stop _time _measurement host _field _value
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:09:00Z m1 host1 f1 1.2
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:28:00Z m1 host1 f1 2.5
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:39:00Z m1 host1 f1 2.2
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:18:00Z m1 host2 f1 1.2
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:24:00Z m1 host2 f1 2.5
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:42:00Z m1 host2 f1 2.2
Group key: ["_start", "_stop", "_measurement", "host", "_field"]

24
Flux
Syntax

25
Pipe-forward
|>

26
Basic data types
string "example string"
bool true
float 1.20
int 123456789
uint 123456789
duration 1h
time 2019-12-01T00:00:00Z
bytes [104 101 108 108 111]

27
Composite data types
record {key1: "value1", key2: 2.0}
array [12, 34, 56]
dict [1: "foo", 2: "bar"]
function f = (x) => x

28
Variable declarations
varName = varValue
foo = "bar"
bazRecord = {this: 1, that: 8}
data = from(...)

29
Reference composite types
r = {key1: "value1", key2: 2.0}
Dot notation
r.key1
Returns "value1"
Bracket notation
r["key1"]
Returns "value1"

30
arr = [12, 34, 56]
Bracket notation
arr[0]
Returns 12

31
dictEx = [1: "foo", 2: "bar"]
Function reference
dict.get(dict: dictEx, key: 1, default: "baz")
Returns "foo"

32
dictEx = [1: "foo", 2: "bar"]
Function reference
dict.get(dict: dictEx, key: 3, default: "baz")
Returns "baz"

33
r = {key1: "value1", key2: 2.0}
key = "key1"
r.key Returns an error
r.["key"] Returns an error
r.["${key}"] Returns an error
r = ["key1": "value1", "key2": "2.0"]
key = "key1"
dict.get(
dict: r,
key: key,
default: ""
) Returns "value1"

34
Functions
f = (x, y) => x / y
f(x: 12, y: 2)
Returns 6

35
Function types
Static
f = (x, y=2) => x / y
Transformation
f = (t=<-) => t
Predicate
(r) => r.key1 == "foo"

36
Predicate functions
(r) => 1.0 == 1.0
Parameter(s) Predicate Expression

37
Predicate functions
(r) => true

38
Predicate functions
(r) => 1.0 != 1.0

39
Predicate functions
(r) => false

40
{
Predicate functions
(r) => 1.0 > 1.0
== != < > <= >= =~ !~

41
Predicate functions
(r) => 1.0 == 1.0 and "foo" != "bar"

42
Predicate functions
(r) => true and true

43
Predicate functions
(r) => true

44
Predicate functions
(r) => 1.0 > 1.0 and "foo" != "bar"

45
Predicate functions
(r) => false and true

46
Predicate functions
(r) => false

47
Predicate functions
(r) => 1.0 > 1.0 or "foo" != "bar"

48
Predicate functions
(r) => true

49
Predicate functions
(r) =>
1.0 == 1.0 and
("foo" == "bar" or "baz" != "quz")

50
Predicate functions
r = {fname: "John", lname: "Doe", age: 42}
(r) => r.fname == "John" and r.age > 40

51
Predicate functions
r = {fname: "Mike", lname: "Smith", age: 29}
(r) => r.fname == "John" and r.age > 40

52
Basic
Query

53
Basic Query
Source

54
Basic Query
sql.from(
driverName: "postgres",
dataSourceName: "...",
query: "..."
)
Source

55
Basic Query
csv.from(csv: "...")
Source

56
Basic Query
Source

57
Basic Query
r._field == "used_percent")
Source
Filter

58
Basic Query
|> group(columns: ["host"])
Source
Filter
Shape

59
Basic Query
|> mean()
Source
Filter
Shape
Process

60
Basic Query
|> mean()
Source
Filter
Shape
Process

61
Process Data
data = from(bucket: "system-data")
r._field == "used_percent"
)

62
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:09:00Z mem host1 used_percent 62.1
Process Data

63
Process Data
data
|> map(fn: (r) => ({_time: r._time, _value: r._value}))

64
_time _value
2021-01-01T00:09:00Z 62.1
2021-01-01T00:18:00Z 44.0
2021-01-01T00:24:00Z 52.4
2021-01-01T00:28:00Z 63.8
2021-01-01T00:39:00Z 65.4
2021-01-01T00:42:00Z 50.7
Process Data

65
Process Data
data
|> map(fn: (r) => ({r with _value: r._value * 0.01}))

66
Process Data

67
Process Data
data
|> map(fn: (r) => ({
r with
state: if r._value > 65.0 then "high" else "ok"
}))

68
_start _stop _time _measurement host _field _value state
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-
01T00:09:00Z
mem host1 used_percent 62.1 ok
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-
01T00:28:00Z
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:39:00Z mem host1 used_percent 65.4 high
_start _stop _time _measurement host _field _value state
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-01T00:18:00Z mem host2 used_percent 44.0 ok
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-
01T00:24:00Z
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z 2021-01-
01T00:42:00Z
Process Data

69
Process Data
data |> mean()

70
|> mean()

71
Process Data

72
_start _stop _measurement host _field _value
2021-01-01T00:00:00Z 2021-01-01T01:00:00Z mem host1 used_percent 63.8
Process Data

73
Process Data
data |> sum()

74
Process Data

75
Process Data
data |> max()

76
Process Data

77
Process Data
data |> last()

78
Process Data

79
Flux in
Practice

80
from(bucket: "market-summary")
|> range(start: -6mo)
r._measurement == "stockPrices" and
r.symbol == "GME"
)
|> holtWinters(n: 24, seasonality: 12, interval: 1w)

81
heatIndex = (t, h) => //...
from(bucket: "sensor-data")
|> range(start: -1d)
|> filter(fn: (r) => r._measurement == "sensors")
|> filter(fn: (r) => r._field == "temp" or r._field == "hum")
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> map(fn: (r) => ({ r with heatIndex: heatIndex(t: r.temp, h: r.hum) }))

82
import "sql"
sensorInfo = sql.from(
driverName: "postgres",
dataSourceName: "postgresql://localhost?sslmode=disable",
query: "SELECT * FROM sensors"
)
sensorMetrics = from(bucket: "example-bucket")
|> filter(fn: (r) => r._measurement == "airSensors")
join(tables: {metric: sensorMetrics, info: sensorInfo}, on: ["sensor_id"])
|> group(columns: ["sensor_id", "lastInspected"]),
|> aggregateWindow(
every: 5m,
fn: (tables=<-, column="error") => tables |> count(column: column)
)

We look forward to bringing together our
community of developers to learn, interact
and share tips and use cases.
10-11 May 2021
Hands-On Flux Training
18-19 May 2021
Virtual Experience
www.influxdays.com/emea-2021-virtual-experience/

Introduction to Flux and Functional Data Scripting

More Related Content

What's hot

Similar to Introduction to Flux and Functional Data Scripting

More from InfluxData

Recently uploaded

Introduction to Flux and Functional Data Scripting