Best Practices: How to Analyze IoT Sensor Data with InfluxDB

| © Copyright 2022, InﬂuxData
1
Best Practices: How to
Analyze IoT Sensor Data with
InﬂuxDB
Brian Gilmore
September 2022

2
Agenda
● The basics of time series data and applications
● A platform overview — InﬂuxDB, Telegraf, and Flux
● How to start collecting data at the edge and use your
preferred IoT protocol (i.e. MQTT)

3
Brian Gilmore
Director of IoT and Emerging Technology
● Four amazing months at InﬂuxData
● Hospitals → Aquariums → Smart Buildings → Software
● Driven to democratize technology
● Passionate about diversity and inclusion
● Reach out for absolutely anything!

4
Key Drivers for Time Series Applications
Performance,
Availability, and
Security
Time Series Data
from Assets and
Applications
ACCESS ANALYZE ACT
Improve and Expand
the Business

5
Key Components of Time Series Applications
Data Source
Data Source
Data Source
Data Source
Data Source
Applications Infrastructure
Event Processing
• Detection
• Recognition
• Transformation
• Enrichment
• Routing
Database
Real-time Data
Historical Data
APIs
Automation
Customer Apps
Partner Apps
Internal Apps
• Supervisory Control
• Rules-based Exception Handling
• ML Model Distribution
• ML Training Data Distribution
Instrumentation Actuation Automation
Administration
Push Ingestion
Poll Ingestion

6 | © Copyright 2022, InﬂuxData
6
Time Series Data

7
What is Time Series Data?
• Weather conditions
• Stock prices
• CPU use
• Healthcare Metrics
• Logs
• Traces
Sensors
in the physical world
Instrumentation
of the virtual world
A sequence of data points, typically consisting of successive
measurements made from the same source over a time interval.
E X A M P L E S

8
Metrics, Events, and Traces
Usually derived through sampling, usually numeric, and
typically regular in period.
Usually emitted, on-event or on-exception. Can be either
numeric or strings. Irregular period by nature.
Bundled and uniquely labeled collections of related
metrics and events related to a speciﬁc transaction or
interaction. Irregular period and explicit duration.
Metrics
Events
Traces

9
How They Relate
Metrics (one/sec):
Events (varied):
Traces (bundled):
12s
3s
8.26s
Condition 1
Condition 2
Condition 3
Condition 4
Trace ID: 86ef2836-58fe-4a2e-ba69-3de4596072fd
duration 37 seconds
6s

10
Timestamp Precision
• Relative time - :since epoch:
• Ideal to create early and
maintain throughout pipeline
• Must consider implications of
aggregation vs truncation
• Use cases: event ordering,
correlation, time-bounded
analytics

11
Data Granularity
• Loosely coupled to precision
• Also referred to as “sample rate”
• Refers to the number of discrete
samples per series per window
of time
• Reduce granularity through
aggregation and roll-ups
• Ideally retain data shape
through advanced post
processing like SDA or
Holt-Winters or raw-retention

12
Time Series Data in InﬂuxDB

13
Functional Architecture
Sensors
Metrics
Events
Transactions
Instrumentation
InfluxDB Edge Nodes
(OSS)
Functions*:
Input configuration
Input monitoring
Timestamping
Linebreaking
Whitespace handling
In-stream processing
Line Protocol Conversion
Output configuration
Output routing
InfluxDB Clusters
(Enterprise or
Cloud)
REST API Client Libraries Data Explorer Notebooks Visualizations
Time Series Applications
Monitor, Troubleshoot, Alert, Predict, Automate
Edge Datacenter Cloud
Kafka
Kinesis
CSP Pub/Sub
Cloud Native
Telegraf

14
Line Protocol
ALL time series data is written to InfluxDB using Line Protocol, and
uses the following format:
<measurement>[,<tag-key>=<tag-value>] [<field-key>=<field-value>]
[unix-nano-timestamp]
Tag Set
hostname=server02, us_west=az
Measurement
cpu_load,
Field Set
temp=24.5, volts=7
Timestamp
1234567890000000
Where data is formatted as line protocol is user and use case specific. Telegraf provides significant
shortcuts for many popular data sources and formats, and new Cloud-Native Data Connectors
provide JSONPath and Regex parsing solutions to work with more unstructured data. Customers
often build telemetry pipelines specifically for LP delivery to InfluxDB

15
Fields and Tags
• Tags are like labels, designed to further
specify and disambiguate similar
signals. They are indexed for fast seek,
ﬁltering, and grouping.
• Fields are the primary numerical and
other values to be monitored. They are
not indexed as stored values are highly
variable.
• Tags are best for metadata,
Fields are best for samples
sample time: Friday July 15 2022 15:40:50 GMT
machine id: SN0039992 (TAG)
lab id: MEDF-MA-US01 (TAG)
patient id: 123456789A (TAG)
sample volume: 23.24 (FIELD)
sample units: ml (TAG)
cell count: 234 (FIELD)

16
Considering Data Cardinality
Cardinality: The number of unique measurement, tag set, and field key combinations
in an InfluxDB bucket or database.
Bucket Measurement SensorID SensorName Fields Series #
myBucket Temperature SN001 Discharge Temp t=71.1 1
myBucket Temperature SN002 Inlet Temp temp=71.8 2
myBucket Humidity SN003 Discharge Hum h=60.6 3
myBucket Humidity SN004 Inlet Hum h=61.2 4
Series Cardinality = 4
For example:
1. Two distinct “Measurements,” impact of (2)
2. Four distinct “SensorIDs,” impact of (2) - two
unique for Temperature, two have a 1:1
relationship with Measurement
3. Four distinct “SensorNames,” with 1:1
relationship with SensorID, impact of (0)
4. Three distinct field keys, 1:1 relationship with
SensorName and SensorID, impact of (0)
Failure to normalize measurement names, tag names, and field keys is the quickest way to increased
cardinality. In the above example, if just one of the Discharge Temp Value’s field keys were changed to
“temp,” this would increase the series cardinality to 5.
High cardinality can impact the performance of very large databases where cardinality can reach hundreds
of millions of unique series. Poor data hygiene at this scale can have significant impact on search
performance. While this is uncommon - good data hygiene is always best

17
Flux: Built for Time Series

18
Flux is a Functional Scripting Language
API CLI
/api/v2/query influx query [query literal] [flags]
Basic Examples Capability Overview
Main Features:
• Source Data
• Filter Data
• Shape Data
• Process Data
Data Model Components:
• Tables
• Records
• Columns
• Streams
About Flux
Flux is a functional data scripting language designed to unify querying, processing,
analyzing, and acting on data into a single syntax.
InﬂuxDB
Bucket
CSV File
SQL Server

19
Flux Precepts
• Functional Language
• Declarative Language
• Strongly and Statically Typed
• Objects are Immutable
• Named Parameters
• Overloaded Parameters
• Pipe-Forward (|>)
• Operates on Streaming
Tables
• Map and Reduce capable
• Extensible Packages
• Imports

20
Common Flux Functions
Function Purpose
Parameters: default
(options)
Example
group() Regroup input data by
modifying group keys in
input tables
Columns: [] (*)
mode: by (by, except)
tables: (<-)
tables
|> group(columns: ["_time", "tag"])
aggregateWindow() Groups data into fixed
windows of time and
then applies aggregator
every: NA (*)
function: NA (min, max,
etc)
tables
|> aggregateWindow(every: 20s, fn: mean)
Conditional Logic Applies if, then, else
logic
if, else if, else if r._value > 95.0000001 and r._value <= 100.0
then
"critical"
else if r._value > 85.0000001 and r._value <=
95.0 then
"warning"
else if r._value > 70.0000001 and r._value <=
85.0 then
"high"
else
"normal"

21
Flux Secret Weapons
Function Purpose
Parameters: default
(options)
Example
map() Iterates through
table rows and
applies a function
fn: NA (function)
tables: <- (input data)
|> map(fn: (r) => ({time: r._time, source: r.tag,
alert: if r._value > 10 then true else false}))
stateDuration() Calculates
cumulative duration
of a given state
fn: NA (function)
column: stateDuration (*)
timeColumn: _time (*)
unit: 1s (ns,us,ms,s,m,h,d)
tables: <- (input data)
|> stateDuration(fn: (r) => r._value < 15)
Custom
Functions
Fully customize flux functionName: NA
functionParameters: NA
functionOperators: NA
// Function definition
multByX = (tables=<-, x) => tables
|> map(fn: (r) => ({r with _value: r._value *
x}))
// Function usage
from(bucket: "example-bucket")
|> range(start: -1m)
|> filter(fn: (r) => r._measurement == "mem" and
r._field == "used_percent")
|> multByX(x: 2.0)

22
Why Use Flux?
• Works natively with InfluxDB storage and
computing
• Runs against InfluxDB buckets, 1.x databases,
.csv files and SQL compliant stores
• Unified language for querying, processing,
and task execution
• Allows developers to run time series data
processing code inside the time series
database
Deeply Integrated Highly Functional
• Join and Union support
• Over 500 included functions
• Inputs
• Outputs
• Transformations
• Dynamic Queries
• Geotemporal support
• Task and Notification support
• Type Conversions
• Timestamp handling

23
New InﬂuxDB Features for IoT

24
InfluxData Edge Data Replication (EDR)
API CLI
/api/v2/remotes influx remote [create,delete,list,update]
/api/v2/replications influx replication [create,delete,list,update]
Cloud (InfluxDB Cloud) Database(s)
Enables:
• Raw data replication
• Hybrid downsampling
• Eventual consistency
Build:
• Database distribution
• Hybrid apps
• ML pipelines
Bucket Queue
Bucket Queue
Flux
Edge (InfluxDB) Databases
Disk

25
InfluxData Native Collectors
Apps & Devices
IOT OT IT Mobile
Logs
Metrics
&
Events
Traces Transactions
Sensors Assets Products Comms
Publish
InfluxDB
1. Configure
Subscriptions &
Connections
2. Define Parsing
Rules
(LP, JSON, RegEx)
3. Configure Routing
(Buckets)
Low-code UI Wizard
Subscribe
Brokers and Streaming APIs
Topics and Queues
iotdevices/amer/r23rr23042/temperature
iotdevices/asia/tower1/floor 81/humidity
mobile/apps/myapp/errors
servermon/server1/cpu
factory1/line1/machineA/output

26
Demo - Cloud Native MQTT

27

28

29

30

31

32

33

34

InfluxDB University
Free live and self-paced
training on:
• InfluxDB
• Telegraf
• Flux
• Kapacitor
• and more
Scan to explore
the course catalog
influxdbu.com

Enroll for free at inﬂuxdbu.com
Join InﬂuxDB University

Deeply technical,
single-track InfluxDB event
Join the two-day developer conference dedicated
to building IoT, analytics, and cloud applications
with InfluxDB.
November 2-3, 2022
Register today for free: influxdays.com

Taming the Telegraf Tiger Training
November 1, 2022
Virtual - Paciﬁc Time
Free - Limit: 300
Register today for free: inﬂuxdays.com
Advanced Flux Training
November 8-9, 2022
In-Person - London, UK
£500 - Limit: 50

Taming the Telegraf Tiger Training
November 1, 2022
Virtual - Pacific Time
InfluxDays 2022 Conference
November 2-3, 2022
Virtual & In-Person
Register today for free: influxdays.com
Advanced Flux Training
November 8-9, 2022
In-Person - London, UK

40
T H A N K Y O U

Best Practices: How to Analyze IoT Sensor Data with InfluxDB

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices: How to Analyze IoT Sensor Data with InfluxDB

Similar to Best Practices: How to Analyze IoT Sensor Data with InfluxDB (20)

More from InfluxData

More from InfluxData (20)

Recently uploaded

Recently uploaded (20)

Best Practices: How to Analyze IoT Sensor Data with InfluxDB