Anais Dotis-Georgiou & Faith Chikwekwe [InfluxData] | Top 10 Hurdles for Flux Beginners | InfluxDays Virtual Experience NA 2020

Faith Chikwekwe, InfluxData
Anais Dotis-Georgiou, InfluxData
Top 10 Hurdles for
Flux Beginners

Introduction and Overview
Flux

© 2020 InfluxData. All rights reserved. 3
Introduction to Flux
● Functional query and
scripting language
● JavaScript-esque
● Open source
● Written in Go and Flux itself
from(bucket:"example-bucket")
|> range(start:-1h)
|> filter(fn:(r) =>
r._measurement == "cpu" and
r.cpu == "cpu-total"
)

Introduction to Flux
● Math across Measurements
● Custom Functions
● SPEC and syntax examples available in repo
github.com/influxdata/flux/SPEC.md

1.Overlooking UI tools
which facilitate writing
Flux

Using the UI–Solution
● Using the Flux Script Editor
● Using the Raw Data View

● Variables
Using the Flux Builder

● Variables
Injecting Functions

Table View vs Raw Data View

Using the Raw Data View

2. Folks misunderstand
annotated CSVs and
sometimes don’t write
them correctly.

● array.from()
● to()
● Use github.com/influxdata/flux/stdlib/universe
Annotated CSV Solutions

from(bucket: "cats-and-dogs")
|> range(start: 2020-05-15T00:00:00Z, stop: 2020-05-16T00:00:00Z)
|> filter(fn: (r) => r["_measurement"] == "cats")
|> filter(fn: (r) => r["_field"] == "adult")
|> filter(fn: (r) => r["shelter"] == "A")
|> filter(fn: (r) => r["type"] == "calico")
|> limit(n:2)
#group,false,false,true,true,false,false,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string
#default,_result,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,shelter,type
,,0,2020-05-15T00:00:00Z,2020-05-16T00:00:00Z,2020-05-15T18:50:33.262484Z,8,adult,cats,A,calico
,,0,2020-05-15T00:00:00Z,2020-05-16T00:00:00Z,2020-05-15T18:51:48.852085Z,7,adult,cats,A,calico

Explaining Annotated CSV
● #group: A boolean that indicates the column is part of the group key. A group key
is a list of columns for which every row in the table has the same value.
○ true columns: In our example query above, we’ve filtered by a single field
type, adult, a single “shelter” tag, “A”, and a single “type” tag, “calico”. These
values are constant across rows, so those columns are set to true.
○ false columns: The _time and _value columns have different values values
across rows which is why they receive a false for the value of the #group
Annotation.
● #datatype: Describes the type of data.
● #default: The value to use for rows with an empty value. So for example, if we
had assigned our query to the variable ourQuery this annotation would look like:
#default,ourQuery,,,,,,,,,

import "experimental/array"
array.from(rows: [{_measurement: "m0", mytag: "t0", _field: "f0", _value: "foo", _time: 2020-01-
01T00:00:00Z,},
{_measurement: "m0", mytag: "t0", _field: "f0", _value: "bar", _time: 2020-01-
02T00:00:00Z}])
|> to(bucket: “my-bucket”)
Using array.from()

3. Data layout design
leads to runaway
cardinality and slow Flux
queries

Influx Line Protocol

General Recommendations
● Encode meta data in tags. Measurements and tags are
indexed while field values are not. Commonly queried
metadata should be stored in tags.
● Limit the number of series or try to reduce series
cardinality.
● Keep bucket and measurement names short and simple.
● Avoid encoding data in measurement names.
● Separate data into buckets when you need to assign
different retention policies to that data or require an
authentication token.

Common Mistakes
Mistake: Making ids (such as eventid, orderid, or userid) a tag.
This is another example that can cause unbounded cardinality
if the tag values aren’t scoped.
Solution: Instead, make these metrics a field.

Schema and Flux
● Your schema affects your Flux query performance.
● Anticipate common queries and let that influence your
schema design.

4. Storing the wrong data
in InfluxDB and under-
utilizing Source Functions

● Use Source functions to pull relevant data into InfluxDB as
needed
● Work with PostgreSQL, MySQL, Snowflake, SQLite, SQL
Server, Athena, BigQuery
Source Functions – Solution

import "sql"
sql.from(
driverName: "postgres",
dataSourceName: "postgresql://user:password@localhost",
query: "SELECT * FROM example_table"
)
Source Functions – Solution

5. Confused about how to
project multiple
aggregations

● Create multiple queries with the UI and visualize them
simultaneously
● Get “Fancy” with Flux to project multiple aggregations
○ join()
○ pivot() and union()
○ reduce()
Multiple Aggregation Projections – Solutions

Multiple Aggregation with the UI

● The join() function merges two or more input streams,
whose values are equal on a set of common columns, into a
single output stream.
● The reduce() function aggregates records in each table
according to the reducer, fn, providing a way to create
custom aggregations.
Join and Reduce

6. Common Flux Packages
and Utility Functions
Not knowing beginner tips to make writing Flux easier

Common Flux Packages and Utility Functions
aggregateWindow() Create your own utility function to aggregate data
for your use case and pass it in as the `fn` param
monitor package Write your own check and notification scripts for
greater control over alerts `flux/docs/Alerts.md`
map() Replace the value in a column (for example
offsetting the `_value` column) using the with
keyword
I need more functionality for my
use case
Consider contributing a custom package to Flux
flux/stdlib/contrib/README.md

Custom Aggregate
custom_agg = …
from(bucket:"items")
|> range(start:-1h)
|> aggregateWindow(every: 1m, fn: custom_agg)
Map
from(bucket: “items”)
|> filter(fn: (r) => r._measurement == "m")
|> map(fn: (r) => ({r with _value: r._value * 2.0} ))

Median Absolute Deviation with Flux

7. Performance Gains
Being unaware of awesome performance
optimizations to make your queries faster in
InfluxDB Cloud 2.0

- Memory optimizations
- More pushdowns
- What’s a pushdown?
- “Pushing down” the work of transforming the data to be
done by the storage side
- How does it help optimize my query?
Performance Gains

Query You have some data stored in InfluxDB Cloud 2.0. You write a
query for that data. (e.g. from |> range |> filter |> group |> max)
Plan The Flux planner will check if your query matches any
existing pushdown patterns.
Rewrite If it matches a pushdown pattern, then the planner
will write the plan for how to perform your query.
Execute
The Flux executor will execute your query by
invoking a hidden operation (e.g. ReadGroupMax).
This will initiate a storage read API call.
Result
Storage will transform the data and stream
the result back to Flux via gRPC. Flux converts
the gRPC data back to Flux tables.

8. Schema Mutations
Not using schema mutators at the right time

9. Tasks
When to write your own tasks

How to write a Task

option task = {name: “rateTask”, every: 1h}
rate = from(bucket: “items”)
|> range(start: -task.every)
|> filter(fn: (r) => (r._measurement == “totalItems”))
|> filter(fn: (r) => (r._field == “items”))
|> group(columns: [“itemGroupName”])
|> aggregateWindow(fn: sum, every: task.every)
|> map(fn: (r) => {
return: _time: r._time, _stop: r._stop, _start: r._start, _measurement: “rateTask”, _field: “fruits”, _value: r._value,
itemGroupName: itemGroupName,
}
})
rate
|> to(bucket: “newItems”)
How to write a Task

from(bucket: “newItems”)
|> range(start: -1mo)
|> filter(fn: (r) => (r._measurement == “rateTask”))
|> filter(fn: (r) => (r._field == “items”))
Output Task Data to a Dashboard
Examples for how to write a check and notification script are in
flux/doc/Alerts.md.
You can even write your own alerts

10. Downsampling Tasks
Not knowing when/why to downsample data

options task = {name: “foo”, every: 1m}
from(bucket: “items”)
|> filter(fn: r._measurement == "m" and r._field == "f")
|> group(columns: ["_measurement", "_field", "_start", "_stop"])
|> aggregateWindow(fn: last, every: task.every)
|> to(bucket: "downsample_bucket")
Why Downsample?

option task = {name: "Downsampling CPU", every: 1m}
data = from(bucket: "my-bucket")
|> filter(fn: (r) => r._measurement == "my_measurement")
data
|> mean()
|> set(key: "agg_type",value: "mean_temp")
|> to(bucket: "downsampled", org: "my-org", tagColumns: ["agg_type"])
data
|> count()
|> set(key: "agg_type",value: “count_temp")
|> to(bucket: "downsampled", org: "my-org", tagColumns: ["agg_type"])
Why Downsample?

Thanks!

Anais Dotis-Georgiou & Faith Chikwekwe [InfluxData] | Top 10 Hurdles for Flux Beginners | InfluxDays Virtual Experience NA 2020

Anais Dotis-Georgiou & Faith Chikwekwe [InfluxData] | Top 10 Hurdles for Flux Beginners | InfluxDays Virtual Experience NA 2020

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Anais Dotis-Georgiou & Faith Chikwekwe [InfluxData] | Top 10 Hurdles for Flux Beginners | InfluxDays Virtual Experience NA 2020

Similar to Anais Dotis-Georgiou & Faith Chikwekwe [InfluxData] | Top 10 Hurdles for Flux Beginners | InfluxDays Virtual Experience NA 2020(20)

More from InfluxData

More from InfluxData(20)

Recently uploaded

Recently uploaded(20)

Anais Dotis-Georgiou & Faith Chikwekwe [InfluxData] | Top 10 Hurdles for Flux Beginners | InfluxDays Virtual Experience NA 2020