Anais Dotis-Georgiou [InfluxData] | Learn Flux by Example | InfluxDays NA 2021

Anais Dotis-Georgiou
Developer Advocate, InﬂuxData
Flux by Example

© 2021 InﬂuxData Inc. All Rights Reserved.
About Me
○ Anais Jackie Dotis on LinkedIn
○ @AnaisDotis
Developer Advocate, InfluxData

https:/
/docs.inﬂuxdata.com/inﬂuxdb/v2.0/reference/sample-data/

https:/
/docs.inﬂuxdata.com/inﬂuxdb/v2.0/reference/sample-data/
• NOAA water sample dataset
import "experimental/csv"
csv.from(url: "https://influx-testdata.s3.amazonaws.com/noaa.csv")
|> to(bucket: "noaa")
• Air sensor sample dataset
import "influxdata/influxdb/sample"
sample.data(set: "airSensor")
|>to(bucket: "airSensor")

Sample datasets
• NOAA water sample dataset
The NOAA water sample dataset is static dataset extracted from NOAA
Center for Operational Oceanographic Products and Services data. The
sample dataset includes 15,258 observations of water levels (ft) collected
every six minutes at two stations (Santa Monica, CA (ID 9410840) and
Coyote Creek, CA (ID 9414575)) over the period from August 18, 2015
through September 18, 2015.
• Air sensor sample dataset
Air sensor sample data represents an “Internet of Things” (IoT) use case by
simulating temperature, humidity, and carbon monoxide levels for multiple
rooms in a building.

NOAA water sample dataset schema
● 5 measurement:
○ average_tempearture, h2o_feet, h2o_pH, h2o_quality,
h2o_temperature
● 5 ﬁelds:
○ degrees, index, level description, pH, water_level
● 2 tags: location, randtag
○ 2 location tag values
■ coyote_creek, santa_monica
○ 3 randtag tag values
■ 1, 2, 3

Air sensor sample dataset schema
● 1 measurement: airSensors
● 3 ﬁelds:
○ co
○ humidity
○ temperature
● 1 tag: sensor_id
○ 8 sensor_id tag values
■ TM0100, TM0101, TM0102, TM0103, TM0200, TM0201, TM0202,
TM0203

Agenda
• from() |> range() |> filter()
• conditional filtering
• group()
• aggregateWindow() vs window()
• selectors and aggregators
• yield()
• map()
• conditional mapping
• pivot() and schema.fieldsAsCol()
• findRecord()

Agenda–the questions we’ll be answering
1. How do I return all temp values greater than 80 for
coyote_creek?
2. How do I return all temp values greater than 80 for
coyote_creek and all temp values less than 80 for
santa_monica?
3. How do I calculate the mean() temp for coyote_creek and
santa_monica?
4. How do I calculate the mean() temp across coyote_creek
and santa_monica?
5. How do I return the values for 3. and 4. simultaneously?
6. How do I calculate the difference in temp across
coyote_creek and santa_monica?
7. How do I calculate the difference between the mean() temp
of coyote creek and the temp of coyote creek?

10
1. How do I return all
temp values greater
than 80 for
coyote_creek?

Step 1: Understanding from() |> range() |> ﬁlter()
from(bucket: "noaa")
|> range(start: rfc3339startTime, stop: rfc3339stopTime)
|> filter(fn: (r) => r["_measurement"] == "average_temperature")
|> filter(fn: (r) => r["_field"] == "degrees")
|> filter(fn: (r) => r["location"] == "coyote_creek")
|> limit(n:3)
|> yield(name: "last 3")
where rfc3339startTime, rfc3339stopTime and rfc3339timeN = 2018-10-07T19:36:05.000Z
use |> range(start: 0) to filter for all data.
use relative durations as well |> range(start: -1y)

Step 1: Understanding from() |> range() |> ﬁlter()
Not in Group
Key
In Group Key In Group Key Not in
Group
Key
In Group Key In Group Key In Group Key Not in Group
Key
table
last 3
_measurement _field _value location _start _start _time
0 average_temper
ature
temperature 82.0 coyote_creek rfc3339startTime rfc3339stopTime rfc3339time1
0 average_temper
ature
0 average_temper
ature

Step 2: Filtering on thresholds
|> limit(n:3)
|> filter(fn: (r) => r._value > 80.0)
|> yield(name: "thresholds")

Answer 1
Not in Group
Key
Group
Key
Key
table
thresholds
0 average_temper
ature
0 average_temper
ature

15
2. How do I return all temp
values greater than 80 for
coyote_creek and all temp
values less than 80 for
santa_monica?

Step 1: Filter for both locations
|> filter(fn: (r) => r["location"] == "coyote_creek" or r["location"] == "santa_monica")
|> limit(n:3)
|> yield(name: "before")

Not in
Group Key
Group
Key
In Group Key In Group Key In Group Key Not in Group Key
table
before
0 average_temperature temperature 82.0 coyote_creek rfc3339startTime rfc3339stopTime rfc3339time1
1 average_temperature temperature 85.0 santa_monica rfc3339startTime rfc3339stopTime rfc3339time1

Step 2: Conditional ﬁltering based on the
location
|> limit(n:3)
|> filter(fn: (r) =>
if r["location"] == "coyote_creek"
then r._value > 80.0
else r._value < 80.0
)
|> yield(name: "conditional filtering")

Not in
Group Key
Group
Key
table
conditional
filtering
Answer 2

20
3. How do I calculate the
mean() temp for
coyote_creek and
santa_monica?

Step 1: Apply mean()
|> limit(n:3)
|> mean()
|> yield(name: "mean")

Not in
Group Key
Group
Key
In Group Key In Group Key In Group Key
table
conditional
filtering
_measurement _field _value location _start _start
0 average_temperature temperature 80.3333 coyote_creek rfc3339startTime rfc3339stopTime
1 average_temperature temperature 79.6667 santa_monica rfc3339startTime rfc3339stopTime
Solution 3

23
mean() temp across
coyote_creek and
santa_monica?

Step 1: group() data
|> limit(n:3)
|> group(columns:["_measurement"])
|> yield(name: "group on measurement")

Not in Group
Key
In Group Key Not In Group
Key
Not in
Group
Key
Not In Group Key Not In Group Key Not In Group Key Not in Group Key
table
group on
measurement
0 average_temperatur
e
e
e
e
temperature 85.0 santa_monica rfc3339startTime rfc3339stopTime rfc3339time1
e
e

Not in
Group Key
In Group Key Not in
Group
Key
table
conditional
filtering
_measurement _value
0 average_temperature 80
Solution 4

29
4. Bonus Q: How could I
retain the other
columns?

Bonus Solution 4A: group() data and include other
columns
|> limit(n:3)
|> group(columns:["_measurement", "_field", "_start", "_stop"])
|> yield(name: "group on many")

Bonus Solution 4B: group() data and use the
except mode
|> limit(n:3)
|> group(columns:["_time", "_value", "location"], mode: "except")
|> yield(name: "group except")

Not in
Group Key
Group
Key
Not in Group Key In Group Key In Group Key Not in Group Key
table
before

Bonus Solution 4C: group() to ungroup the data
|> limit(n:3)
|> group()
|> yield(name: "ungroup")

Not in
Group Key
Not in Group Key Not in Group
Key
Not in
Group
Key
Not in Group Key Not in Group Key Not in Group Key Not in Group Key
table
before

36
5. How do I return the
values for 3. and 4.
simultaneously?

Not in
Group Key
In Group Key Not in
Group
Key
table
mean temp
across both
_measurement _value
0 average_temperature 80
Solution 5
Not in
Group Key
Group
Key
table
mean for
each
1 average_temperature temperature 79.6667 santa_monica rfc3339startTime rfc3339stopTime

39
5. Bonus Q: how do I
return the min(), max(),
and mean() of the
coyote_creek temp?

Bonus Solution 5
Not in
Group Key
Group
Key
table
mean
Not in
Group Key
Group
Key
table
min
Not in
Group Key
Group
Key
table
max

42
difference in temp
across coyote_creek
and santa_monica?

Not in
Group Key
Group
Key
Not in Group Key In Group Key In Group Key In Group Key
table
group on
time

Not in
Group Key
Group
Key
Not in Group Key In Group Key In Group Key In Group Key
table
difference
0 average_temperature temperature -3.0 coyote_creek rfc3339startTime rfc3339stopTime rfc3339time1
1 average_temperature temperature -1.0 coyote_creek rfc3339startTime rfc3339stopTime rfc3339time2

What do I do if my timestamps aren’t the same
across tags or ﬁelds?
• window() your data instead of grouping on time
• Also it’s more performant!
• truncate your timestamps to transform irregular time series
into regular ones with date.truncate()

Not in Group
Key
Group
Key
Not in Group
Key
table
window
difference
0 average_temperature temperature -3.0 coyote_creek rfc3339startTime rfc3339startTime + 6m rfc3339time1
1 average_temperature temperature -1.0 coyote_creek rfc3339startTime + 6m rfc3339stopTime + 12m rfc3339time2
2 average_temperature temperature 6.0 coyote_creek rfc3339startTime + 12m rfc3339stopTime + 18m rfc3339time3

aggregateWindow() vs window()
• aggregateWindow() applies an aggregate or selector
function (any function with a column parameter) to ﬁxed
windows of time.
• The window() function groups records based on a time
value. The function calculates time windows and stores
window bounds in the _start and _stop columns. _start
and _stop values are assigned to rows based on the _time
value.

Solution 6C: pivot() and map()
|> limit(n:3)
|> pivot(rowKey:["_time"], columnKey: ["location"], valueColumn: "_value")
|> yield(name: "pivot")
|> map(fn: (r) => ({ r with difference: r.coyote_creek - r.santa_monica }))
|> yield(name: "after map"

Not in
Group Key
In Group Key In Group Key Not in Group
Key
Not In Group
Key
In Group Key In Group Key Not in Group Key
table
pivot
_measurement _field coyote_creek santa_monica _start _start _time
0 average_temperature temperature 82.0 85.0 rfc3339startTime rfc3339stopTime rfc3339time1
Not in
Group
Key
In Group Key Not in Group
Key
Not In Group
Key
Not In Group
Key
In Group Key In Group Key Not in Group
Key
table
pivot
_measurement coyote_creek santa_monica difference _start _start _time
0 average_temper
ature
82.0 85.0 -3.0 rfc3339startTi
me
rfc3339stopTime rfc3339time1
0 average_temper
ature
73.0 74.0 -1.0 rfc3339startTi
me
0 average_temper
ature
86.0 80.0 6.0 rfc3339startTi
me

52
6. Bonus Topic: digging
deeper into pivot() and
map()

Using map() to change data types
data |> limit(n:3)
|> map(fn: (r) => ({ r with int: float(v: r._value}))
|> yield(name: "type conversion")
Especially useful for when you want to:
• perform math across columns with different data types
• group data together with different data types

Conditionally transform values
Especially useful for when you want to:
• create custom checks
• handle null values
• handle special mathematical exceptions i.e. when trying to perform
transforms like dividing by two values and handling 0 values.

Assigning a level to your data
data
|> map(fn: (r) => ({
r with
level:
if r._value >= 80.0 then "critical"
else if r._value < 80.0 and r._value >= 70.0 then "warning"
else "normal"
})
)

Handling diving by 0 or null values
data
|> map(fn: (r) => ({ r with
diving:
if not exists r.coyote_creek or r.coyote_creek == 0.0 then 0.0
else if not exists r.santa_monica or r.santa_monica == 0.0 then 0.0
else r.coyote_creek / r.coyote_creek
}))

Using pivot() on ﬁelds and schema.ﬁeldsAsCol()
data = from(bucket: "airsensor")
|> range(start: rfc3339startTime, stop: rfc3339stop Time)
|> filter(fn: (r) => r["_measurement"] == "airSensors")
|> filter(fn: (r) => r["_field"] == "co" or r["_field"] == "humidity")
|> filter(fn: (r) => r["sensor_id"] == "TLM0100")
|> limit(n:2)
|> yield(name: "before")
data
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> yield(name: "pivoted")
data
|> schema.fieldsAsCol()
|> yield(name: "fiedsAsCol")

Not in
Group Key
Group
Key
Key
table
before
_measurement _field _value sensor_id _start _start _time
0 airSensors co
0.483
TM0100 rfc3339startTime rfc3339stopTime rfc3339time1
0 airSensors co 0.478 TM0100 rfc3339startTime rfc3339stopTime rfc3339time2
Not in Group Key In Group Key Not in Group
Key
Key
table
pivoted/fieldsAsCol
_measurement co sensor_id _start _start _time
0 airSensors
0.483
0 airSensors 0.478 TM0100 rfc3339startTime rfc3339stopTime rfc3339time2

Using pivot() multiple columns
from(bucket: "airsensor")
|> range(start: rfc3339startTime, stop: rfc3339stop Time)
|> filter(fn: (r) => r["_measurement"] == "airSensors")
|> filter(fn: (r) => r["_field"] == "co" or r["_field"] == "humidity")
|> filter(fn: (r) => r["sensor_id"] == "TLM0100" r["sensor_id"] == "TLM0101" )
|> pivot(rowKey:["_time"], columnKey: ["_field", "sensor_id"], valueColumn: "_value")
|> yield(name: "mult col pivot")

Not in
Group Key
Group
Key
Key
table
before
_measurement _field _value sensor_id _start _start _time
0 airSensors co
0.483
1 airSensors co 0.478 TM0101 rfc3339startTime rfc3339stopTime rfc3339time1
2 airSensors humidity 35.12 TM0100 rfc3339startTime rfc3339stopTime rfc3339time1
3 airSensors humidity 34.86 TM0101 rfc3339startTime rfc3339stopTime rfc3339time1
Not in
Group
Key
In Group
Key
Not in
Group Key
Not in
Group Key
Not in
Group Key
Not in Group
Key
table
mult col
pivot
_measure
ment
co_TM010
0
co_TM010
1
humidty_T
M0100
humidty_TM0
101
sensor_id _start _start _time
0 airSensors
0.483 0.478 35.12 34.86

61
difference between the
mean() temp of coyote creek
and the temp of coyote
creek?

Step 1: use ﬁndRecord()
|> range(start: rfc3339startTime, stop: rfc3339stopTime,)
|> limit(n:3)
mean_coyote = data |> mean() |> findRecord(fn: (key) => true, idx: 0)
// |> findRecord(fn: (key) => key.location == "santa_monica", idx: 0)
// where mean_coyote._value = 80.3

Step 2: apply map()
data
|> map(fn: (r) => ({ r with difference: r._value - mean_coyote._value }))
|> yield( name: "diff mean and coyote creek")
import "array"
array.from(rows: [{_time: now(), _value: float(v: mean_coyote._value)}])
Bonus: use array.from() to check mean value

Not in Group Key In Group Key In Group Key Not in Group Key Not In Group Key In Group Key In Group Key Not in Group Key
table
diff mean and
coyote creek
_measurement _field _value difference _start _start _time
0 average_tempera
ture
temperature 82.0 1.667 rfc3339startTime rfc3339stopTime rfc3339time1
0 average_tempera
ture
temperature 73.0 -7.333 rfc3339startTime rfc3339stopTime rfc3339time2
0 average_tempera
ture
temperature 86.0 5.667 rfc3339startTime rfc3339stopTime rfc3339time3
Not in Group Key Not in Group Key Not in Group Key
table
_result
_value _time
0 80.333 rfc3339timeNow

Flux Proﬁler

Resources
● Top 5 Hurdles for Flux Beginners and Resources for Learning
to Use Flux
● Top 5 Hurdles for Intermediate Flux Users and Resources for
Optimizing Flux
● TL;DR InﬂuxDB Tech Tips – Optimizing Flux Performance in
InﬂuxDB Cloud

Questions?

Thank You

Anais Dotis-Georgiou [InfluxData] | Learn Flux by Example | InfluxDays NA 2021

Anais Dotis-Georgiou [InfluxData] | Learn Flux by Example | InfluxDays NA 2021

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Anais Dotis-Georgiou [InfluxData] | Learn Flux by Example | InfluxDays NA 2021

Similar to Anais Dotis-Georgiou [InfluxData] | Learn Flux by Example | InfluxDays NA 2021(20)

More from InfluxData

More from InfluxData(20)

Recently uploaded

Recently uploaded(20)

Anais Dotis-Georgiou [InfluxData] | Learn Flux by Example | InfluxDays NA 2021