More Related Content More from Hortonworks (20) Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Harnessing Data-in-Motion
with Hortonworks DataFlow
Edge intelligence for IoT with
Apache MiNiFi (subproject of Apache NiFi)
Joe Percivall
Software Engineer
Anna Yong
Product Marketing
2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
EMBRACE AN OPEN
APPROACH
MASTER THE VALUE
OF DATA
EVERY BUSINESS
IS A DATA BUSINESS
4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Constrained
High-latency
Localized context
Hybrid – cloud / on-premises
Low-latency
Global context
Core
Infrastructure
Harnessing Data in Motion
Regional
InfrastructureSources
5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow Manages Data in Motion
Core
InfrastructureSources
Constrained
High-latency
Localized context
Hybrid – cloud / on-premises
Low-latency
Global context
Regional
Infrastructure
6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Requirements for Data in Motion
Perishable Insights
ConnectivityPrioritization
Security
AdaptabilityExtensibility
Scalability
Provenance
Real-Time
7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connecting Data Between Ecosystems Without Coding: 170+ Processors
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
HL7
FTP
UDP
XML
SFTP
HTTP
Syslog
Email
HTML
Image
AMQP
MQTT
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi
Guaranteed delivery
Data buffering
‒ Backpressure
‒ Pressure release
Prioritized queuing
Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
Data provenance
Recovery / recording a rolling log
of fine-grained history
Designed for extension
Visual command and control
Flow templates
Policy based security
Clustering
Key Features
9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simplified Ingestion Example
Let’s consider the needs of a courier service
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Courier service from the perspective of NiFi
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
NiFi NiFi NiFi NiFi NiFi NiFi
On Delivery Routes
12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Courier service from the perspective of NiFi & MiNiFi
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client
Libraries
Client
Libraries
MiNiFi
MiNiFi
NiFi NiFi NiFi NiFi NiFi NiFi
Client
Libraries
On Delivery Routes
13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Created to more effectively collect
data at the edge
14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi
Guaranteed delivery
Data buffering
‒ Backpressure
‒ Pressure release
Prioritized queuing
Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
Data provenance
Recovery / recording a rolling log
of fine-grained history
Designed for extension
Visual command and control
Flow templates
Policy based security
Clustering
Key Features
15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Guaranteed delivery
Data buffering
‒ Backpressure
‒ Pressure release
Prioritized queuing
Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
Data provenance
Different from Apache NiFi
Design and Deploy
Warm re-deploys
Key Features
Apache NiFi MiNiFi
Recovery / recording a rolling log
of fine-grained history
Designed for extension
16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Command and Control
vs.
Design and Deploy
17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Different MiNiFi Distributions
Java
– <40MB binary distribution
– Requires Java 1.8
– More feature complete
– Targeted for any systems that can run a JVM (ie. Servers, Raspberry Pi)
C++
– 600KB code size and static data ~50KB
– Dynamic heap of ~1MB based on use-case
– Targeted for resource constrained environments (ie. edge IoT devices)
Both use same config format and use NiFi terminology
Different focuses depending on requirements
18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Typical Simple Example
Tail a rolling file -> Site to Site
19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Typical Simple Example
Tail a rolling file -> Site to Site
20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simple Config.yml
Tail a rolling file -> Site to Site
21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi and MiNiFi, JAVA and C++
22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs MiNiFi Java Processes
NiFi Framework
Components
MiNiFi
NiFi Framework
User Interface
Components
NiFi
23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Java Processes
Bootstrap
NiFi
UI
bootstrap.conf
nifi.properties
flow.xml.gzreads &
modifies
reads
reads
starts
NiFi MiNiFi
24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
MiNiFi Java Processes
MiNiFi
Bootstrap
Configuration
Change Notifier(s)
bootstrap.conf
nifi.properties
flow.xml.gz
reads
reads
starts
config.ymltransforms
reads
into
NiFi MiNiFi
25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Same Extensible framework (nars)
NiFi standard processors are bundled
– TailLog
– UpdateAttribute
– Route on content and attributes
– PutEmail
– ….
Allows MiNiFi Java to use NiFi processors
26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
MiNiFi C++ Features
Same design and deploy configuration
Initial set of processors
– TailFile
– GetFile
– GenerateFlowFile
– LogAttribute
– ListenSyslog
Site to Site Client implementation in C++ for talking to NiFi instances
27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simple Config.yml
Tail a rolling file -> Site to Site
29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Courier service from the perspective of NiFi & MiNiFi
Physical Store
Gateway
Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/
Deliverer: Rigo Peter, https://thenounproject.com/rigo/
Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/
Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client
Libraries
Client
Libraries
MiNiFi
MiNiFi
NiFi NiFi NiFi NiFi NiFi NiFi
Client
Libraries
On Delivery Routes
30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demo: Digital Temp & Humidity Sensor, Raspberry Pi Zero, MiNiFi
Raspberry
Pi Zero
with MiNiFi
(link)
USB hub for
power & WiFi
Digital Temp & Humidity
Measurement Sensor
31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
MiNiFi Command and Control
Design Flow at a centralized place, deploy on the edge
Version control of flows
– Align with NiFi SDLC work
Agent status monitoring
Bi-directional command and control
Currently a feature proposal, initial version being architected
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core
Infrastructure
Hortonworks DataFlow Use Cases
Regional
InfrastructureSources
Dataflow Management
• On-ramp into Hadoop
• Log Collection / Splunk Optimization
• Cyber Security
• IoT Ingestion
• Deliver data into stream processing engines
Real-time Event Processing
(Kafka, Storm)
Move data between
from on-prem and
cloud environments
35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache MiNiFi Use Case: Distributed Systems: Retail POS
100’s or 1000’s or more terminals to gather:
Real-time pricing info
Real-time inventory updates
Real-time offers
Traceable logs of transactions
Log of user-interactions for trouble-shooting
devices
Predictive help desk
36. Apache MiNiFi Use Case: Distributed Systems: Connected Car
MiNiFi
SD Card
Wi-Fi
DSRC
LTE
Raw data
via CAN,
Ethernet
Car Wireless Gateway
Buffer offline
delay-
tolerant data
1
Real-time
data
1
4
Wi-Fi AP
DSRC RSU
eNodeB
5
5
Possibly modify route
or adjust sampling rate
and prepare to flush
delay-tolerant buffers
when SDRC RSU or Wi-
Fi AP in range
2
Data Storage
Decision-Making Server
Map Database
3
Hortonworks maximizes intelligence with platforms that deliver actionable
insights from connected car data
37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
GENIVI Alliance: Open Source, In-Vehicle Infotainment Software
The GENIVI Alliance is a nonprofit industry
alliance committed to driving the broad adoption
of specified, open source, In-Vehicle Infotainment
software.
38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache MiNiFi Use Cases: Distributed Systems of Devices
Routers
Security cameras
Cable modems
ATM
Fleet of Trucks
Manufacturing Line
Security Appliance
Point of Sale
Weather detection system
Thermostats
Utility/Power meters
Fleet of Ships
Any distributed system of devices with data to be collected
40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Learn more and join us!
Apache NiFi site
http://nifi.apache.org
Subproject MiNiFi site
http://nifi.apache.org/minifi/
Subscribe to and collaborate at
dev@nifi.apache.org
users@nifi.apache.org
Submit Ideas or Issues
https://issues.apache.org/jira/browse/NIFI
https://issues.apache.org/jira/browse/MINIFI
Follow us on Twitter
@apachenifi
41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
Hortonworks Community Connection:
Data Ingestion and Streaming
https://community.hortonworks.com/
Editor's Notes Hortonworks: Powering the Future of Data Hortonworks: Powering the Future of Data Hortonworks: Powering the Future of Data Hortonworks: Powering the Future of Data