Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI

Harnessing Data-in-Motion
with Hortonworks DataFlow
Introduction to HDF 2.0
Haimo Liu
Product Manager
Aldrin Piri
Technical Staff

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda HDF 2.0: Flow Management
– NiFi basics
– NiFi use cases
– NiFi demos
HDF 2.0: Streaming Analytics

Simplistic View of Enterprise Data Flow
Data Flow
Process and Analyze
Data
Acquire Data
Store Data

Interacting with different business partners and customers
Realistic View of Enterprise Data Flow

• For agile and immediate creation, configuration, control of dataflowsVisual Command and Control
• Ensures trust of your dataData Lineage (Provenance)
• Because not all data is of equal importanceData Prioritization
• Since not all senders/receivers/connections work perfectly all the timeData Buffering/Back-Pressure
• Adapt to different situations with different requirementsControl Latency vs Throughput
• Security of data, and data accessSecure Control Plane/Data Plane
• ScalabilityScale out Clustering
• Ecosystem flexibility and growthExtensibility
Apache NiFi: Designed for 8 challenges of global enterprise dataflow

What is Apache NiFi used for?
• Reliable and secure transfer of data between systems
• Delivery of data from sources to analytic platforms
• Enrichment and preparation of data:
– Conversion between formats
– Extraction/Parsing
– Routing decisions
What is Apache NiFi NOT used for?
• Distributed Computation
• Complex Event Processing
• Joins / Complex Rolling Window Operations
Use Cases for Apache NiFi

FlowFile
• Unit of data moving through the system
• Content + Attributes (key/value pairs)
Processor
• Performs the work, can access FlowFiles
Connection
• Links between processors
• Queues that can be dynamically prioritized
Terminology

HTTP Data FlowFile
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g
Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT
Content-Type: text/html
Hello world XXXXXXXXXXXXXXXXXXXXXXXXXXXX
Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'fileSize’ Value: '23609'
Key: 'filename’ Value: '15650246997242'
Key: 'path’ Value: './’
0101010101110101010101010101 (Binary)
Header
Content
Analogy: FlowFiles are like HTTP Data

1. Drag and drop processors to build a flow
2. Start, stop, and configure components in real time
3. View errors and corresponding error messages
4. View statistics and health of data flow
5. Create templates of common processor & connections
Create, Run, View, Start, Stop, Change, Fix, Dataflows in Real-Time

Apache NiFi Demo: Tail Logs, Route on Content, Buffer in Kafka,
Deliver to HDFS

What is Data Provenance and Why is it Important?
BEGIN
END
LINEAGE
IT and Cloud Operators
• Understand traceability, lineage
• Enable recovery and replay
Compliance Regulations
• Provide an audit trail
• Remediation capabilities

Provenance Enables Easy Access and Traceability of Changes

Need Fine-Grained Security and Compliance?
Security
• Secured authentication
• Enterprise authorization services –
entitlements change often
• Encrypted content, encrypted
communications
• People and systems with different roles
require difference access levels
• Tagged/classified data

Repositories - Pass by reference

Repositories – Copy on Write

Agenda HDF 2.0 Flow Management
HDF 2.0 Platform Evolution
– Product offering
– Example use case

 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Core
Infrastructure
Hortonworks DataFlow Manages Data in Motion
Regional
InfrastructureSources

DataFlow Management and Stream Processing
Core
InfrastructureSources
 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Regional
Infrastructure

Edge Intelligence with Apache MiNiFi
 Guaranteed delivery
 Data buffering
‒ Backpressure
‒ Pressure release
 Prioritized queuing
 Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
 Data provenance
 Recovery / recording a rolling log
of fine-grained history
 Designed for extension
Different from Apache NiFi
 Design and Deploy
 Warm re-deploys
Key Features

NiFi vs. MiNiFi Java Agent
NiFi Framework
Components
MiNiFi
NiFi Framework
User Interface
Components
NiFi

Example: Company X provides alerting services when users’ resting heart rate higher
than a threshold
Real-Time Insights Require DataFlow Mgmt and Stream Processing
Acquire
Data
Company X Cloud
Instance 1
Acquire
Data
Company X Cloud
Instance 2
Acquire
Data
Company X Cloud
Instance 3
Acquire Data
Across Cloud
Instances
Parse, Filter,
Validate, Enrich
and Route
Core Data Center
Analytics/Pattern
Match
Data
Store
Alerts
Dashboards/Visualization
Flow Management Stream ProcessingLegend:

Data in Motion Needs Dataflow Management and Stream Processing
 Acquire data from various Wearable Device’s Cloud Instances
 Move Data from Customer Cloud Instances to on-premise instance
 Perform Intelligent Routing & Filtering of data. The routing and filtering rules will be often
changed at run-time.
 Deliver the data data to various downstream systems. New downstream apps should will always
appear and the data should be fed to it when it comes online.
 Parse the device data to standardized format that downstream sysem can understand
 Enrich the data with contextual information including patient/customer info (age, sex, etc..)
 Recognize the Pattern when the resting heart rate exceeds a certain threshold (the insight),
and then create an alert/notification.
 Run a Outlier detection model on streaming heart rate that comes in. If the score is above
certain threshold, alert on the heart rate.
Flow
Management
(NiFi, MiNiFi)
Stream
Processing
(Storm, Kafka)

Use Cases for Data in Motion
Use Cases for Data-in-Motion Using DataFlow Mgmt
• Data Ingestion
• Edge Intelligence
• First Mile Problem
• Physical Data Movement
• Simple event processing such as Route, Filter, Enrich,
Transform, etc.
When Only DataFlow
Management is
Required
Use Cases for Data-in-Motion Using DataFlow Mgmt and
Steam Processing
• Flow Management to deliver data for Stream Processing
• PLUS: Complex pattern matching on unbounded streams of
data.
When Both DataFlow
Management and
Stream Processing

Flow management
D A T A I N M O T I O N D A T A A T R E S T
IoT Data Sources AWS
Azure
Google Cloud
Hadoop
NiFi
Kafka
Storm
Others…
NiFi
NiFi NiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
MiNiFi
NiFi
HDF 2.0: Data-in-Motion Platform
Enterprise Services
Ambari Ranger Other services
Flow management + Stream Processing

New Stream Processing Features HDF 2.0
 New Storm Connectors
 Storm-Kafka Spout using new
client APIs
 Storm Distributed Log Search
 Storm Dynamic Worker
Profiling
 Kafka Grafana Integration
 Storm Grafana Integration
 Improved Nimbus HA
 Storm Automatic Back
Pressure
 Storm Distributed cache
 Storm Windowing and State
Management
 Storm Performance
improvements
 Improved Kafka SASL
 Storm Topology Event inspector
 Storm Resource Aware
Scheduling
 Storm Dynamic Log Levels
 Pacemaker Storm Daemon
 Kafka Rack Awareness
Developer Productivity EnterpriseReadiness Operational Simplicity

For More Info: https://community.hortonworks.com/
Hortonworks Community Connection:
Data Ingestion and Streaming

Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI

More Related Content

What's hot

Similar to Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI

Recently uploaded

Harnessing Data-in-Motion with HDF 2.0, introduction to Apache NIFI/MINIFI

Editor's Notes