Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache NiFi- MiNiFi meetup Slides

88,712 views

Published on

MiNiFi is a recently started sub-project of Apache NiFi that is a complementary data collection approach which supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation. Simply, MiNiFi agents take the guiding principles of NiFi and pushes them to the edge in a purpose built design and deploy manner. This talk will focus on MiNiFi's features, go over recent developments and prospective plans, and give a live demo of MiNiFi.
The config.yml is available here: https://gist.github.com/JPercivall/f337b8abdc9019cab5ff06cb7f6ff09a

Published in: Technology
  • Be the first to comment

Apache NiFi- MiNiFi meetup Slides

  1. 1. Apache NiFi – MiNiFi Taking Dataflow Management to the Edge Joe Percivall - @JPercivall
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Me • Software Engineer at Hortonworks • Apache NiFi committer and PMC member • Github: github.com/JPercivall
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Why create MiNiFi? • MiNiFi 0.0.1-Java • Demo • Prospective plans
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Why create MiNiFi? • MiNiFi 0.0.1-Java • Demo • Prospective plans
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved • Web-based User Interface for creating, monitoring, & controlling data flows • Directed graphs of data routing and transformation • Highly configurable - modify data flow at runtime, dynamically prioritize data • Easily extensible through development of custom components • Data Provenance tracks data through entire system [1] https://nifi.apache.org/ Apache NiFi Dataflow
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine- grained history • Visual command and control • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simplified Example Let’s consider the needs of a courier service Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi & MiNiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi MiNiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Recovery/recording a rolling log of fine- grained history • Designed for extension • Design and Deploy • Warm re-deploys
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi MiNiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Recovery/recording a rolling log of fine- grained history • Designed for extension • Design and Deploy • Warm re-deploys
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Visual Command and Control vs. Design and Deploy
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Created to more effectively collect data at the edge
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Why create MiNiFi? • MiNiFi 0.0.1-Java • Demo • Prospective plans
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi vs MiNiFi Java Processes NiFi Framework Components MiNiFi NiFi Framework User Interface Components NiFi
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Java Processes Bootstrap NiFi UI bootstrap.conf nifi.properties flow.xml.gzreads & modifies reads reads starts NiFi MiNiFi
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi Java Processes MiNiFi Bootstrap Configuration Change Notifier(s) bootstrap.conf nifi.properties flow.xml.gz reads reads starts config.ymltransforms reads into NiFi MiNiFi
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Same Extensible framework (nars)  In minifi-0.0.1, the nifi-0.6.1 standard processors are bundled (~20mb) – Tailing a Log – UpdateAttribute – Routing by content or attributes – PutEmail Allows MiNiFi to use NiFi processors
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi 0.0.1-Java  Declarative configuration of processing flows through a YAML configuration file  Exporting of provenance events to another NiFi instance via a Reporting Task over Site to Site  Flow change configuration watcher implementations that provide reloading a NiFi instance when receiving an updated flow over REST or changes on a file system  Providing a mechanism to query an instance's status  <40mb binary distribution Release Notes
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simple Config.yml Tail a rolling file -> Site to Site
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved But what about the NiFi.properties values? Can omitted for default values
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Provenance Reporting  Site to Site Reporting Task  JSON formatted provenance events  Configured via config.yml  Optional “SiteToSiteProvenanceReportingTask” in NiFi 0.7.0
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved A bit more complex Config.yml Tail a rolling File -> Secure Site to Site with Provenance
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi Toolkit  Convert NiFi templates to config.yml  Validate config.yml files CLI to facilitate config.yml building
  24. 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Config Change Notifiers  Two implementations – RestChangeNotifier • Http(s) – FileChangeNotifier  Configured in bootstrap.conf
  25. 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers 1. Initial state –Both running
  26. 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers user creates new configuration 2. User sends update through notifier –HTTP(S) post request –Change watched file
  27. 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers 3. Bootstrap validation –Basic validation –Rest notifier will respond accordingly –Results logged validate new configuration user creates new configuration
  28. 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers config.yml saves new 4. Bootstrap saves and transforms –Copy old config.yml to a swap file validate new configuration user creates new configuration nifi.properties flow.xml.gz transforms into
  29. 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update MiNiFi Bootstrap Configuration Change Notifiers nifi.properties flow.xml.gz attempt restart config.yml saves new reads transforms into 5. Bootstrap attempts restart –MiNiFi reads in the new nifi.properties and flow.xml.gz validate new configuration user creates new configuration
  30. 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Change notifier update 6. Success or Fail –Successful restart continue processing –Failure, rollback to old config –Existing Data is mapped or orphaned MiNiFi Bootstrap Configuration Change Notifiers nifi.properties flow.xml.gz attempt restart config.yml saves new reads transforms into validate new configuration user creates new configuration
  31. 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved minifi.sh flowStatus  Components  Instance  System Diagnostics Get flow status at the command line
  32. 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Why create MiNiFi? • MiNiFi 0.0.1-Java • Demo • Prospective plans
  33. 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Courier service from the perspective of NiFi Physical Store Gateway Server Mobile Devices Registers Server Cluster Distribution Center Core Data Center at HQ Server Cluster On Delivery Routes Trucks Deliverers Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/ Deliverer: Rigo Peter, https://thenounproject.com/rigo/ Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/ Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/ Client Libraries Client Libraries MiNiFi MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi Client Libraries
  34. 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • Why create MiNiFi? • MiNiFi 0.0.1-Java • Demo • Prospective plans
  35. 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Prospective Plans  MiNiFi 0.0.1-Cpp – Close to a vote to release – Code itself is 1.2mb without optimization – Data size • ~20mb for dynamic RAM for heap • Static ~50kb  Configurable Status Reporters – minifi.sh flowStatus -> regular status update • MQTT? • HTTP? • S2S?  Handle component Annotation Data – UpdateAttribute advanced rules
  36. 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Major Feature - Centralized Command and Control  Design at a centralized place, deploy on the edge – Flow deployment – NAR deployment – Agent deployment  Version control of flows  Agent status monitoring  Bi-directional command and control Centralized management console with a UI
  37. 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions?
  38. 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank you!
  39. 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn more and join us! Apache NiFi site http://nifi.apache.org Subproject MiNiFi site http://nifi.apache.org/minifi/ Subscribe to and collaborate at dev@nifi.apache.org users@nifi.apache.org Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi
  40. 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Matured at NSA 2006-2014 Brief history of the Apache NiFi Community • Contributors from Government and several commercial industries • Releases on a 6-8 week schedule • Apache NiFi 1.0.0. release on the horizon • Zero-Master Clustering Code developed at NSA 2006 Today Achieved TLP status in just 7 months July 2015 Code available open source ASL v2 November 2014
  41. 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved MiNiFi differentiation  Let me get the key parts of NiFi close to where data begins and provide bidrectional communication  NiFi lives in the data center. Give it an enterprise server or a cluster of them.  MiNiFi lives close to where data is born and may be a guest on that device or system

×