Data Con LA 2018 - Streaming and IoT by Pat Alwell

1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Data Flow
Wrangling the Internet of Things
Pat Alwell – Solutions Engineer
Big Data Day Los Angeles ,CA
August 2018

whoami
Pat Alwell
Solutions Engineer Hortonworks
AWS | Spark | Hadoop Admin
Career Started at Algebraix Data
Connect with me:
GitHub  https://github.com/patalwell
Email  palwell@hortonworks.com
Goals for the Session
• Demonstrate how organizations can leverage
Hortonworks Dataflow (HDF) to wrangle the Internet
of Things

Apache NiFi Managed Dataflow
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

Nothing in HDF Makes Sense Except in Light
of Flow Based Programming
Flow
Management
Administration
HDF
Streaming
Flow-based programming is an
abstraction of information packets,
algorithmic transformations, and a
common set of connections. The flow of
data is essentially equivocal to a
production line. Raw material is pushed
or pulled into a process and transformed
to meet an end goal.

• Connection = Route between processors
 Queues that can be dynamically prioritized
• Process Group = Logical Group of processors and their connections
 Receive data via input ports, send data via output ports
 FlowFile = Unit of data moving through the system
 Content + Attributes (Metadata)
 Processor = Process data
 Transforms/Writes FlowFiles
 Creates Provenance
Nifi Terminology

What is HDF?

Flow
Management
HDF
How Can we Manage Flows with
HDF?
Apache NiFi supports powerful and scalable directed
graphs of data routing, transformation, and system
mediation logic.
Apache MiNiFi is a complementary data collection
approach that supplements the core tenets of NiFi in
dataflow management, focusing on the collection of data
at the source of its creation.

Visual Command and Control A Convenient Graphical User Interface that
supports Flow Based Programming
• Drag and drop processors to build a flow
• Start, stop, and configure components in real time
• View errors and corresponding error messages
• View statistics and health of data flow
• Create templates of common processor & connections

Visual Command and Control
Over 200 + Processors designed to help you
capture and deliver data to and from common
sources
Examples Include:
-Capturing Logs from Mobile Devices and
Sensors; formatting said logs with Regex,
pushing said logs into HDFS or S3
-Collecting sensor readings from GPIO headers
and delivering the information to an application
via Kafka
-Customer sentiment analysis by joining social
media data to customer information within Hive
or Phoenix

220+ Processors for Deeper Ecosystem Integration
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
HTTP
Syslog
Email
HTML
Image
HL7
FTP
UDP
XML
SFTP
AMQP
WebSocket

Provenance and Lineage

Payload Prioritization
• Configure a prioritizer per
connection
• Determine what is important for
your data – time based, arrival
order, importance of a data set
• Funnel many connections down to
a single connection to prioritize
across data sets
• Develop your own prioritizer if
needed

Back-Pressure

Latency vs. Throughput

• Java
• < 40MB binary distribution
• Requires JRE 1.8
• More feature complete
• Targeted for any system that can run a JVM
• Supported Processors: https://github.com/apache/nifi-
minifi/blob/6ddf8bb0ee3614320a53ce7f2e0b3950ee4d9c5f
/minifi-docs/src/main/markdown/minifi-java-agent-quick-
start.md
• C++
• Dynamic heap of ~1MB based on use-case
• Targeted for resource constrained environments
• Supported Processors: https://github.com/apache/nifi-
minifi-cpp/blob/master/PROCESSORS.md
Minifi’s Key Features
An Embedded Extension that supports Flow
Based Programming on the Edge
Agents Provide:
• Small and lightweight footprint
• Central management of agents
• Generation of data provenance
• Integration with NiFi for follow-on
dataflow management and full chain of
custody of information

How can we Administer and Secure flow
activity?
Administration
HDF
The Apache Ambari project is aimed at making Hadoop
management simpler by developing software for
provisioning, managing, and monitoring Apache Hadoop
clusters. Ambari provides an intuitive, easy-to-use
Hadoop management web UI backed by its RESTful
APIs. - https://ambari.apache.org/

Cluster Administration and Role Based Security

• NiFi Registry - sub-project of Apache NiFi
• https://github.com/apache/nifi-registry
• https://issues.apache.org/jira/projects/NIFIREG
• Complimentary application, central location for
storage/management of “versioned” resources
• Initial capability to store and retrieve “versioned
flows”
• Integration on NiFi side
• Start/Stop version control of a process group
• Change version (upgrade/downgrade)
• Import new process group from a version
Version Control for Flows

• Parameterize configuration like connection
strings, file paths, etc.
• Referenced via Expression language
• Kafka Brokers = ${kafka.brokers}
• Variables associated with a process group
• Right-click on canvas to view variables for
current process group
• Hierarchical order of precedence, resolve
closest reference to component
• Editing variables automatically restarts
any components referencing the variables!
Level 1
Level 2
Vars
Vars
Variable Registry Flow Ubiquity

⬢ Data Governance
– Centralized registry to provide reusable
schema
– Version management to define
relationship between schemas
– Validation to enable generic format
conversion and generic routing
⬢ Operational Efficiency
– Centralized registry to avoid attaching
schema to every piece of data
– Version management to enable
consumers and producers can evolve at
different rates
– Validation to ensure data quality
Schema Registry

How Can we take advantage of Streaming
computations?
HDF
Streaming

A Distributed Streaming Platform that supports
Pub-Sub Systems
• Publish and Subscribe to streams of records,
similar to a messaging queue
• Store streams of records in a fault-tolerant way
• Process Streams of records as they occur
• Topic is a partitioned Log of events
Generally used to…
• Build real-time streaming data pipelines to
reliably transfer data between systems
• Build real-time streaming applications that
transform or react to the streams of data
What is Kafka?

Streaming Messaging Manager
(*NEW)
“Kafka Blindness” – Customers who use Kafka today struggle with
monitoring and managing Kafka clusters.

What is Storm?
A Distributed Fault Tolerant Service that
processes Streams of Data• Capture data from external systems (Kafka, Hbase, Hive, HDFS, and AWS
Kinesis) Spout
• Transform and aggregate said data using filter, map, flatMap, aggregate,
reduce, count, etc. Bolt
• Write data back to an external system for storage or visualization (Hbase,
Hive, Druid) Bolt
• The chain is known as a topology. The topology is run under a master
slave type architecture.
Generally used to…
• Processing streams
• No need for intermediate queues. Continuous computation Send
data to clients continuously so they can update and show results in
real time, such as site metrics.
• Distributed remote procedure call
• Easily parallelize CPU-intensive operations.
*
* Leibiusky, Jonathan. Getting Started with Storm:
Continuous Streaming Computation with Twitter's
Cluster Technology (p. 1). O'Reilly Media. Kindle
Edition.

What does our flow look
like?

⬢ Full Q&A Platform (like StackOverflow)
⬢ Knowledge Base Articles
⬢ Code Gallery and Samples
⬢ https://community.hortonworks.com
Join Us: Hortonworks Community Connection

⬢ Download our HDF Sandbox for Docker, VMWare, or
VirtualBox:
https://hortonworks.com/downloads/#sandbox
⬢ Follow our HDF tutorials:
https://hortonworks.com/tutorial/analyze-iot-weather-
station-data-via-connected-data-architecture/
⬢ Reach out to an Enterprise Account Manager
Care to Learn More?

Questions ?

Apache NiFi / ETL Tools
NiFi
NOT schema dependent
• Dataflow management for both structured
and unstructured data, powered by
separation of metadata and payload
• Schema is not required, but you can have
schema
• Minimum modeling effort, just enough to
manage dataflows
• Do the plumbing job, maximize developers’
brainpower for creative work
⚠ Not designed to do heavy lifting transformation
work for DB tables (JOIN datasets, etc.). You
can create custom processors to do that, but
long way to go to catch up with existing ETL
tools from user experience perspective (GUI for
data wrangling, cleansing, etc.)
ETL (Informatica, etc.)
Schema dependent
• Tailored for Databases/WH
• ETL operations based on schema/data
modeling
• Highly efficient, optimized performance
⚠ Must pre-prepare your data, time consuming to
build data modeling, and maintain schemas
⚠ Not geared towards handling unstructured data,
PDF, Audio, Video, etc.
⚠ Not designed to solve dataflow problems

Apache NiFi / Integration, or ingestion, Frameworks
NiFi
End user facing dataflow management
tool
• Out of the box solution for dataflow
management
• Interactive command and control in the core,
design and deploy on the edge
• Flexible failure handling at each point of the
flow
• Visual representation of global dataflow and
connectivities
• Native cross data center communication
• Data provenance for traceability
⚠ Not a library to be embedded in other
applications
Integration framework (Spring
Integration, Camel, etc), ingestion
framework (Flume, etc)
Developer facing integration tool with a
focus on data ingestion
• A set of tools to orchestrate workflow
• A fixed design and deploy pattern
• Leverage messaging bus across
disconnected networks
⚠ Developer facing, custom coding needed to
optimize
⚠ Pre-built failure handling, lack of flexibility
⚠ No holistic view of global dataflow
⚠ No built-in data traceability

Apache NiFi / Messaging Bus Services
NiFi
Provide dataflow solution
• Centralized management, from edge to
core
• Great traceability, event level data
provenance starting when data is born
• Interactive command and control – real
time operational visibility
• Dataflow management, including
prioritization, back pressure, and edge
intelligence
• Visual representation of global dataflow
⚠ Not a messaging bus, flow maintenance
needed when you have frequent consumer
side updates
Messaging Bus (Kafka, JMS, etc.)
Provide messaging bus service
• Low latency
• Great data durability
• Decentralized management (producers &
consumers)
• Low broker maintenance for dynamic
consumer side updates
⚠ Not designed to solve dataflow problems
(prioritization, edge intelligence, etc.)
⚠ Traceability limited to in/out of topics, no lineage
⚠ Lack of global view of
components/connectivities

Apache NiFi / Processing Frameworks
NiFi
Simple event processing
• Primarily feed data into processing
frameworks, can process data, with a
focus on simple event processing
• Operate on a single piece of data, or in
correlation with an enrichment dataset
(enrichment, parsing, splitting, and
transformations)
• Can scale out, but scale up better to
take full advantage of hardware
resources, run concurrent processing
tasks/threads (processing terabytes of
data per day on a single node)
⚠ Not another distributed processing
framework, but to feed data into those
Processing Frameworks (Storm, Spark,
etc.)
Complex and distributed processing
• Complex processing from multiple streams
(JOIN operations)
• Analyzing data across time windows (rolling
window aggregation, standard deviation,
etc.)
• Scale out to thousands of nodes if needed
⚠ Not designed to collect data or manage data
flow

Data Con LA 2018 - Streaming and IoT by Pat Alwell

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Con LA 2018 - Streaming and IoT by Pat Alwell

Similar to Data Con LA 2018 - Streaming and IoT by Pat Alwell (20)

More from Data Con LA

More from Data Con LA (20)

Recently uploaded

Recently uploaded (20)

Data Con LA 2018 - Streaming and IoT by Pat Alwell