Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
•
2 likes•1,354 views
Apache NiFi provided a revolutionary data flow management system with a broad range of integrations with existing data production, consumption, and analysis ecosystems, all covered with robust data delivery and provenance infrastructure. Now learn about the follow-on project which expands the reach of NiFi to the edge, Apache MiNiFi. MiNiFi is a lightweight application which can be deployed on hardware orders of magnitude smaller and less powerful than the existing standard data collection platforms. With both a JVM compatible and native agent, MiNiFi allows data collection in brand new environments — sensors with tiny footprints, distributed systems with intermittent or restricted bandwidth, and even disposable or ephemeral hardware. Not only can this data be prioritized and have some initial analysis performed at the edge, it can be encrypted and secured immediately. Local governance and regulatory policies can be applied across geopolitical boundaries to conform with legal requirements. And all of this configuration can be done from central command & control using an existing NiFi with the trusted and stable UI data flow managers already love. Expected prior knowledge / intended audience: developers and data flow managers should have passing knowledge of Apache NiFi as a platform for routing, transforming, and delivering data through systems (a brief overview will be provided). The talk will focus on extending the data collection, routing, provenance, and governance capabilities of NiFi to IoT/edge integration via MiNiFi. Speaker Andy LoPresto, Sr. Member of Technical Staff, Hortonworks
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Similar to Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
- 5. © Hortonworks Inc. 2011 – 2017. All Rights Reserved5 Let’s Connect A to B Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 7. © Hortonworks Inc. 2011 – 2017. All Rights Reserved7 Why is moving data effectively hard? ⬢ Standards ⬢ Formats ⬢ “Exactly Once” Delivery ⬢ Protocols ⬢ Veracity of Information ⬢ Validity of Information ⬢ Ensuring Security ⬢ Overcoming Security ⬢ Compliance ⬢ Schemas ⬢ Consumers Change ⬢ Credential Management ⬢ “That [person|team|group]” ⬢ Network* ⬢ “Exactly Once” Delivery
- 10. © Hortonworks Inc. 2011 – 2017. All Rights Reserved10 Apache NiFi Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine-grained history • Visual command and control • Flow templates • Pluggable, multi-tenant security • Designed for extension • Clustering
- 11. © Hortonworks Inc. 2011 – 2017. All Rights Reserved11 FlowFiles are like HTTP data HTTP Data FlowFile HTTP/1.1 200 OK Date: Sun, 10 Oct 2010 23:26:07 GMT Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT ETag: "45b6-834-49130cc1182c0" Accept-Ranges: bytes Content-Length: 13 Connection: close Content-Type: text/html Hello world! Standard FlowFile Attributes Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'fileSize’ Value: '23609' FlowFile Attribute Map Content Key: 'filename’ Value: '15650246997242' Key: 'path’ Value: './’ Binary Content * Header Content
- 12. © Hortonworks Inc. 2011 – 2017. All Rights Reserved12 User Interface Less of this… … more of this
- 15. © Hortonworks Inc. 2011 – 2017. All Rights Reserved15 Data Provenance ▪ Constrained ▪ High-latency ▪ Localized context ▪ Hybrid – cloud/on-premises ▪ Low-latency ▪ Global context Origin – attribution Replay – recovery Evolution of topologies Long retention Types of Lineage • Event • Configuration
- 17. © Hortonworks Inc. 2011 – 2017. All Rights Reserved17 IoT Challenges ⬢ Limited computing capability ⬢ Limited power/network ⬢ Restricted software library/platform availability ⬢ No UI ⬢ Physically inaccessible ⬢ Not frequently updated ⬢ Competing standards/protocols ⬢ Scalability ⬢ Privacy & Security
- 19. © Hortonworks Inc. 2011 – 2017. All Rights Reserved19 NiFi Solves Everything* ⬢ Runs on JVM ⬢ Provides UI for flow design & monitoring ⬢ Security built-in ⬢ TLS, authn/authz, encrypted data ⬢ Handles practically any format/protocol
- 21. © Hortonworks Inc. 2011 – 2017. All Rights Reserved21 So Why Do We Need A Different Solution? ⬢ NiFi is designed to “own the box” ⬢ NiFi 0.7.x started up in about 10-15 minutes on RP3 (593 MB) ⬢ NiFi 1.x started up in about 30 minutes on RP3 (760 MB) ⬢ 33 new processors ⬢ Rewrite for multi tenant authorization ⬢ Complete UI overhaul
- 23. © Hortonworks Inc. 2011 – 2017. All Rights Reserved23 Apache NiFi Subproject: MiNiFi ⬢ Get the key parts of NiFi close to where data begins and provide bidirectional communication ⬢ NiFi lives in the data center — give it an enterprise server or a cluster of them ⬢ MiNiFi lives as close to where data is born and is a guest on that device or system ⬢ IoT ⬢ Connected car ⬢ Legacy hardware
- 24. © Hortonworks Inc. 2011 – 2017. All Rights Reserved24 Why build MiNiFi? ⬢ NiFi is big ⬢ 1.3.0 release is 933 MB compressed ⬢ Can be modified to run in restricted environments, but requires manual surgery ⬢ Provides UI, provenance query, etc. ⬢ Runs on dedicated machines/clusters — “owns the box” ⬢ MiNiFi lives at the edge ⬢ No UI ⬢ 0.2.0 Java binary is 59 MB, C++ binary is 1193 KB ⬢ “Good guest”
- 25. © Hortonworks Inc. 2011 – 2017. All Rights Reserved25 How Does MiNiFi Interact With NiFi? ⬢ NiFi ⬢ Design flows ⬢ Aggregate data from many sources ⬢ Perform routing/analysis/ SEP ⬢ MiNiFi ⬢ Receive flows ⬢ Collect data ⬢ Send for processing
- 27. © Hortonworks Inc. 2011 – 2017. All Rights Reserved27 Flavors of MiNiFi ⬢ MiNiFi Java (v0.2.0) ⬢ Modified version of NiFi ⬢ No UI ⬢ YAML configuration ⬢ Reduced processor count ⬢ 110 by default, more available with additional NARs ⬢ MiNiFi C++ (v0.2.0) ⬢ Written from scratch ⬢ 10 processors by default ⬢ Bi-directional site-to-site & provenance data
- 29. © Hortonworks Inc. 2011 – 2017. All Rights Reserved29 What does MiNiFi provide? ⬢ Data tagging/provenance ⬢ Governance from edge (geopolitical restrictions) ⬢ Security (encryption, certificate-based authentication) ⬢ Low latency (immediate reactions & decision-making) Connected Car Reference Platform Box Tuner + DSRC CardConnectivity Card
- 30. © Hortonworks Inc. 2011 – 2017. All Rights Reserved30 MiNiFi Exfil ⬢ Site-to-Site ⬢ NiFi protocol ⬢ Two implementations ⬢ Raw socket ⬢ HTTP(S) (Java only) ⬢ Secured with mutual authentication TLS ⬢ HTTP(S), (S)FTP, JMS, Syslog, File, Email, Process (Java only)
- 32. © Hortonworks Inc. 2011 – 2017. All Rights Reserved32 Scenario ⬢ IoT Device generating log messages ⬢ Need to encrypt data on device ⬢ Need to prioritize some data for unreliable network connectivity ⬢ Transmit data to central node ⬢ Decrypt data and analyze ⬢ Make determinations and modify live flow
- 33. © Hortonworks Inc. 2011 – 2017. All Rights Reserved33 NiFi As Test Harness/Environment ⬢ Simulate the log generation ⬢ Schedule is customizable ⬢ Script can write to dynamic location
- 34. © Hortonworks Inc. 2011 – 2017. All Rights Reserved34 Build the MiNiFi Flow ⬢ Tails a log file ⬢ Logs the raw contents (can be multiple lines in time window) ⬢ Splits into individual lines ⬢ Filters the content ⬢ Using parity of the timestamp ⬢ Prioritizes ⬢ Encrypts using AES/GCM ⬢ Exfils to remote NiFi
- 35. © Hortonworks Inc. 2011 – 2017. All Rights Reserved35 Export from NiFi to MiNiFi ⬢ Save as template from NiFi ⬢ Run $ ./bin/config.sh transform template.xml config.yml ⬢ MiNiFi flow ready to run* *Still need to set up TLS & encrypted properties
- 37. © Hortonworks Inc. 2011 – 2017. All Rights Reserved37 If We Really Have TLS, Why Encrypt? ⬢ All data transmitted over TLS is encrypted ⬢ On NiFi, automatically decrypted ⬢ Attributes visible ⬢ Content still encrypted because of EncryptContent processor ⬢ Can serve as secure route for follow-on systems
- 38. © Hortonworks Inc. 2011 – 2017. All Rights Reserved38 Process Data In NiFi ⬢ Receive the data over S2S ⬢ Log the incoming messages ⬢ Decrypt content ⬢ Log again
- 41. © Hortonworks Inc. 2011 – 2017. All Rights Reserved41 Next Steps ⬢ Window Aggregator ⬢ If >60% odd in window, switch prioritization ⬢ Encrypt with different keys for different tags & send to different follow-on systems ⬢ Spotty network? Tell MiNiFi to cache low priority and send in batches ⬢ MiNiFi rollover & pruning of monitored log ⬢ Exfil MiNiFi provenance data to NiFi
- 44. © Hortonworks Inc. 2011 – 2017. All Rights Reserved44 Why NiFi & MiNiFi? ⬢ Moving data is multifaceted in its challenges and these are present in different contexts at varying scopes – Inter vs intra, domestically, internationally ⬢ Provide common tooling and extensions that are needed but be flexible for extension – Leverage existing libraries and expansive Java ecosystem for functionality – Allow organizations to integrate with their existing infrastructure ⬢ Empower folks managing your infrastructure to make changes and reason about issues that are occurring – Data Provenance to show context and data’s journey – User Interface/Experience a key component
- 46. © Hortonworks Inc. 2011 – 2017. All Rights Reserved46 Learn more and join us Apache NiFi site https://nifi.apache.org Subproject MiNiFi site https://nifi.apache.org/minifi/ Subscribe to and collaborate at dev@nifi.apache.org users@nifi.apache.org Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi