Building Your Data Streams for all the IoT

Craig Hobbs, Solution Engineer, InfluxData
Building IoT Data Streams with
NiFi & InfluxDB

© 2019 InfluxData. All rights reserved.2
About Me
○ Craig Hobbs on Linkedin
○ @chobbs10 on Twitter
“I fight for the users”
Solution Architect, InfluxData

Agenda
• The IoT Challenge
• OPC-Based Data
• NiFi + Influx Solution
• Walk-Through of Basic Setup
• Q/A

“The World of IoT is Time Series Data”
Sensor data is all time-stamped to help you
understand how your processes and equipment are
doing over time to help further innovation and
improvement.

The IoT/IIoT Challenge
No incentive for HW vendors to provide an open communication
framework or interoperability creating vendor lock-in.
At the same time, value contained within these systems is massive,
and it’s only getting more important for enterprises to utilize this
data in process improvement, analytic models, business
automation, and performance visibility.

What is OPC UA
Based Data?

OPC UA
A “Clumsy” Definition
OPC Unified Architecture (OPC UA) is a
machine to machine communication
protocol for industrial automation focused
on communicating with industrial
equipment and systems for data collection
and control.

Why Industries use OPC UA ?
OPC UA was to provide a path forward from the original OPC
communications model that would better meet the emerging
needs of industrial automation.
By removing the barrier because of incompatibilities and
proprietary communication interfaces between different
suppliers' automation hardware and software, OPU UA makes
multi-vendor interoperability and "plug and play" connectivity a
reality.

But there are still big challenges
Even with OPC UA , existing systems that collect these data
streams were still very clunky with no real-time processing
capabilities.
The operational systems still rely on sampling or “lossy data”
approximation and limit access of the data to the local operator’s
machine.

What is
Apache NiFi?

Hello NIFI
NiFi is a Apache Software project
designed to automate the flow of data
between systems.
● Written in Java
● Flow-based programming model
● Extensible plug-in-play environment
● Drag-Drop UI, warm deploy

What is the
InfluxDB Platform?

InfluxDB Platform
● InfluxDB: Database and storage
engine purpose-built to handle
time series data:
● Chronograf: Visualization tool
with pre-canned dashboards
with the standard baseline for
metrics and event monitoring.
● OPC Unified Access data
ingest: InfluxDB actively
supports a number of open
source OPC-UA stacks that are
available to help collect data
from your PLCs.

Why NiFi and InfluxDB?
By combining the two platforms industries can establish reliable
collection streams and that make data easily accessible.
This solution enables the enterprise to securely collect, process,
monitor and share data from their facilities across all business units
supporting a wide range of business-based use-cases.

● Reliable data-stream processing
● Enterprise wide reporting and monitoring
● Ingestion of large volumes
● Data shaping for enterprise consumption
● Real-time queries on large datasets
● Storage optimizations and compression
We work well
together I know,
right!
NiFi + InfluxDB
Better Together

Other advantages of bringing these together
• Scalability
• High-Availability
• Security
• Open Source
• Community Support
• No Development or Coding
• Real-Time Collection
• Flexible Deployment
• Data integration
• No Vendor Lock-in

Basic Setup
NIFI and InfluxDB / Better Togetherp

NiFi + InfluxDB Data Collection

Demo Time!

The finally collection flow of processors and controllers enables a data stream solution that reliable
and supports large data ingest volumes.
Basic NiFi Data Collection Processor Flow

Data Exploration, Visualizations , and Alerts
Chronograf -User interface and
visualization engine makes it easy to
monitoring and alerting for your
infrastructure.
Kapacitor - Real-time data
processing for both stream and
batch data for InfluxDB. Kapacitor
lets you plug in your own custom
logic or user-defined functions to
process alerts with dynamic
thresholds, match metrics for
patterns, compute statistical
anomalies, and perform specific
actions based on these alerts,
Integration with HipChat, OpsGenie,
Alerta, Sensu, PagerDuty, Slack and
more.

Summary Review Thoughts...
Then check out the super-power
team of NiFi<>InfluxDB
So, if you are limited by the analytics capabilities of your traditional
automation systems and are looking to further leverage your big
data platform to securely collect, observe, and act-on across your
enterprise

Learning Resources and Open Source Tools
How to Bring Your Data Infrastructure into the Industry 4.0 Age
● https://www.influxdata.com/resources/how-to-bring-your-data-infrastructure-into-the-
industry-4-0-age/
NiFi Service & Processor bundle from the Tempus IIoT framework
● https://github.com/hashmapinc/nifi-opcua-bundle
FacExposing InfluxDB data over OPC-UA
● https://www.factry.io/blog/exposing-influxdb-data-opcua/
InfluxDB Processors For Apache NiFi
● https://github.com/influxdata/nifi-influxdb-bundle

Twitter: @chobbs10
Email: craig@influxdata.com

Resources:

Begin by downloading the following NiFi Service &
Processor bundle from the Tempus IIoT framework
for industrial data ingestion and analysis. These
processors and the associated controller service allow
NiFi access to OPC UA servers in a read-only fashion.
You will find 2 processors in the Tempus bundle,
GetOPCNodeList and GetOPCData. GetNodeIds
allows access to the tags that are currently in the OPC
UA server. GetOPCData takes a list of tags and
queries the OPC UA server for the values.
Get started with the NiFi-OPCUA bundle

Add the GetOPCNodeList processor to the canvas and configure it by right-clicking on the processor
and clicking configure from the context menu.
Add the GetOPCNodeList processor
Click on the SCHEDULING tab and set the timer to something like 5-10 seconds, as below. Otherwise
you will just slam your OPC server with requests to enumerate the address space.

Configuring the GetOPCNodeList processor
Next, configure the Properties by clicking on the
PROPERTIES tab. Create a new controller service by
clicking in the box to the right of the OPC UA Service
property and clicking on Create a new service from the
drop down as shown below.
When you are done configuring the processor as per
the table above, click on the little arrow to the right of
the controller service (shown below). This will allow you
to configure the controller service.

After clicking Yes, you will be take to the Configure Controller Service modal box. Click on the
PROPERTIES tab. and configure it as per the image below, replacing the Endpoint URL with your own
opc.tcp//{ipaddress}:{port} endpoint.
Configuring the controller service

Add the GetOPCData processor
Head back to the NiFi canvas now, and right-click on
the GetOPCData processor and select Configure from
the context menu to configure the processor.

Configure the GetOPCData processor
You can now finish the configuration of the processor
properties. Please review the property description to
custom configure based on your environment.
NOTE: You will want to use the same controller service
instance as created for the GetOPCNodeList processor.

Connect the GetOPCData GetOPCNodeList
processor
We are now ready to get the Node List. You can view
the list using the NiFi modal with the flow files
currently in the queue.
The output data from the GetOPCData processor will
produces CSV style data (minus the header) in the
format of tag name, time stamp, value and status.
The file should show the root of the data as being
collected as define by your Node Filter in the
GetOPCNodeList processor.

Create a relationship by dragging your mouse from
the GetOPCNodeList processor to the GetOPCData
processor and ticking the Success box.
Now your flow should look like the one here:
Connect GetOPCNodeList & GetOPCData

Now it's time to connect the InfluxDB processor to
ingest the OPC data records. To provide the best
possible ingest performance, InfluxDB has created a
new processor that is based on NiFi Record Design.
Getting started:
Download and install the InfluxDB Processors. Copy
the appropriate nar file into the lib directory of your
NiFi installation ($NiFi_HOME/lib) and restart NiFi.
https://github.com/influxdata/nifi-influxdb-bundle
InfluxDB record processor for Apache NiFi

Head back to the NiFi canvas and add the
PutInfluxDatabasesRecord processor. Connect it to
the previous GetOPCData processor.
Right click to configure the processor and select the
PROPERTIES tab. You will want to define a new
Record Reader, Controller Service, and Content
Properties to write the content of the records into
InfluxDB database.
Add the PutInfluxDatabasesRecord processor

From the service configuration window, define your
InfluxDB database, URL and access credentials.
This will create a shareable connection service for
all NiFi processors.
You can optionally add in a processor for logging
and retries. Now your flow should look like the one
here:
Configure the PutInfluxDatabasesRecord processor

Create a relationship by dragging your mouse from the GetOPCData processor to the
PutInfluxDatabaseRecord processor and ticking the Success box. Finally, flow of processors and
controllers enables a data stream solution that reliable and supports large data ingest volumes.
Basic NiFi data collection processor flow

Building Your Data Streams for all the IoT

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Your Data Streams for all the IoT

Similar to Building Your Data Streams for all the IoT (20)

More from DevOps.com

More from DevOps.com (20)

Recently uploaded

Recently uploaded (20)

Building Your Data Streams for all the IoT