Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi

12,754 views

Published on

Apache MiNiFi is designed to make it practical to enable data collection from the second it is born, ideal for IoT scenarios where there are a large number connected devices or a need for a smaller and more streamlined footprint than Apache NiFi. Join us as we walk through how Apache MiNiFI works, and how it can enable edge data collection from the likes of connected cars, log services, Raspberry PI’s and more. On demand webinar: http://hortonworks.com/webinar/edge-intelligence-iot-apache-minifi/

Published in: Technology
  • Be the first to comment

Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi

  1. 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Harnessing Data-in-Motion with Hortonworks DataFlow Edge intelligence for IoT with Apache MiNiFi (subproject of Apache NiFi) Joe Percivall Software Engineer Anna Yong Product Marketing
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Apache NiFi Refresher • Intro to Apache MiNiFi • Demo • Next Step • How MiNiFi can be used – use cases
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved EMBRACE AN OPEN APPROACH MASTER THE VALUE OF DATA EVERY BUSINESS IS A DATA BUSINESS
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Constrained  High-latency  Localized context  Hybrid – cloud / on-premises  Low-latency  Global context Core Infrastructure Harnessing Data in Motion Regional InfrastructureSources
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks DataFlow Manages Data in Motion Core InfrastructureSources  Constrained  High-latency  Localized context  Hybrid – cloud / on-premises  Low-latency  Global context Regional Infrastructure
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Requirements for Data in Motion Perishable Insights ConnectivityPrioritization Security AdaptabilityExtensibility Scalability Provenance Real-Time
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connecting Data Between Ecosystems Without Coding: 170+ Processors Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute HL7 FTP UDP XML SFTP HTTP Syslog Email HTML Image AMQP MQTT All Apache project logos are trademarks of the ASF and the respective projects. Fetch
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi  Guaranteed delivery  Data buffering ‒ Backpressure ‒ Pressure release  Prioritized queuing  Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance  Data provenance  Recovery / recording a rolling log of fine-grained history  Designed for extension  Visual command and control  Flow templates  Policy based security  Clustering Key Features
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Apache NiFi Refresher • Intro to Apache MiNiFi • Demo • Next Step • How MiNiFi can be used – use cases
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simplified Ingestion Example Let’s consider the needs of a courier service Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ NiFi NiFi NiFi NiFi NiFi NiFi On Delivery Routes
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi & MiNiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries On Delivery Routes
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Created to more effectively collect data at the edge
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi  Guaranteed delivery  Data buffering ‒ Backpressure ‒ Pressure release  Prioritized queuing  Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance  Data provenance  Recovery / recording a rolling log of fine-grained history  Designed for extension  Visual command and control  Flow templates  Policy based security  Clustering Key Features
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Guaranteed delivery  Data buffering ‒ Backpressure ‒ Pressure release  Prioritized queuing  Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance  Data provenance Different from Apache NiFi  Design and Deploy  Warm re-deploys Key Features Apache NiFi MiNiFi  Recovery / recording a rolling log of fine-grained history  Designed for extension
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Command and Control vs. Design and Deploy
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Different MiNiFi Distributions  Java – <40MB binary distribution – Requires Java 1.8 – More feature complete – Targeted for any systems that can run a JVM (ie. Servers, Raspberry Pi)  C++ – 600KB code size and static data ~50KB – Dynamic heap of ~1MB based on use-case – Targeted for resource constrained environments (ie. edge IoT devices)  Both use same config format and use NiFi terminology Different focuses depending on requirements
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Typical Simple Example Tail a rolling file -> Site to Site
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Typical Simple Example Tail a rolling file -> Site to Site
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simple Config.yml Tail a rolling file -> Site to Site
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi and MiNiFi, JAVA and C++
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi vs MiNiFi Java Processes NiFi Framework Components MiNiFi NiFi Framework User Interface Components NiFi
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Java Processes Bootstrap NiFi UI bootstrap.conf nifi.properties flow.xml.gzreads & modifies reads reads starts NiFi MiNiFi
  24. 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi Java Processes MiNiFi Bootstrap Configuration Change Notifier(s) bootstrap.conf nifi.properties flow.xml.gz reads reads starts config.ymltransforms reads into NiFi MiNiFi
  25. 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Same Extensible framework (nars)  NiFi standard processors are bundled – TailLog – UpdateAttribute – Route on content and attributes – PutEmail – …. Allows MiNiFi Java to use NiFi processors
  26. 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi C++ Features  Same design and deploy configuration  Initial set of processors – TailFile – GetFile – GenerateFlowFile – LogAttribute – ListenSyslog  Site to Site Client implementation in C++ for talking to NiFi instances
  27. 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Apache NiFi Refresher • Intro to Apache MiNiFi • Demo • Next Step • How MiNiFi can be used – use cases
  28. 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simple Config.yml Tail a rolling file -> Site to Site
  29. 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi & MiNiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries On Delivery Routes
  30. 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Demo: Digital Temp & Humidity Sensor, Raspberry Pi Zero, MiNiFi Raspberry Pi Zero with MiNiFi (link) USB hub for power & WiFi Digital Temp & Humidity Measurement Sensor
  31. 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Apache NiFi Refresher • Intro to Apache MiNiFi • Demo • Next Step • How MiNiFi can be used – use cases
  32. 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi Command and Control  Design Flow at a centralized place, deploy on the edge  Version control of flows – Align with NiFi SDLC work  Agent status monitoring  Bi-directional command and control Currently a feature proposal, initial version being architected https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
  33. 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Apache NiFi Refresher • Intro to Apache MiNiFi • Demo • Next Step • How MiNiFi can be used – use cases
  34. 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Core Infrastructure Hortonworks DataFlow Use Cases Regional InfrastructureSources Dataflow Management • On-ramp into Hadoop • Log Collection / Splunk Optimization • Cyber Security • IoT Ingestion • Deliver data into stream processing engines Real-time Event Processing (Kafka, Storm) Move data between from on-prem and cloud environments
  35. 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache MiNiFi Use Case: Distributed Systems: Retail POS 100’s or 1000’s or more terminals to gather:  Real-time pricing info  Real-time inventory updates  Real-time offers  Traceable logs of transactions  Log of user-interactions for trouble-shooting devices  Predictive help desk
  36. 36. Apache MiNiFi Use Case: Distributed Systems: Connected Car MiNiFi SD Card Wi-Fi DSRC LTE Raw data via CAN, Ethernet Car Wireless Gateway Buffer offline delay- tolerant data 1 Real-time data 1 4 Wi-Fi AP DSRC RSU eNodeB 5 5 Possibly modify route or adjust sampling rate and prepare to flush delay-tolerant buffers when SDRC RSU or Wi- Fi AP in range 2 Data Storage Decision-Making Server Map Database 3 Hortonworks maximizes intelligence with platforms that deliver actionable insights from connected car data
  37. 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved GENIVI Alliance: Open Source, In-Vehicle Infotainment Software The GENIVI Alliance is a nonprofit industry alliance committed to driving the broad adoption of specified, open source, In-Vehicle Infotainment software.
  38. 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache MiNiFi Use Cases: Distributed Systems of Devices  Routers  Security cameras  Cable modems  ATM  Fleet of Trucks  Manufacturing Line  Security Appliance  Point of Sale  Weather detection system  Thermostats  Utility/Power meters  Fleet of Ships Any distributed system of devices with data to be collected
  39. 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions?
  40. 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn more and join us! Apache NiFi site http://nifi.apache.org Subproject MiNiFi site http://nifi.apache.org/minifi/ Subscribe to and collaborate at dev@nifi.apache.org users@nifi.apache.org Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI https://issues.apache.org/jira/browse/MINIFI Follow us on Twitter @apachenifi
  41. 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? Hortonworks Community Connection: Data Ingestion and Streaming https://community.hortonworks.com/
  42. 42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank you!

×