The presentation discusses the challenges of dataflow in the context of IoT, introducing Apache NiFi as a solution for data collection and management. It also presents Apache MiNiFi as a lightweight alternative to NiFi for edge data processing. Key features include guaranteed delivery, security functionalities, and the capability to handle various data formats and protocols.

1. © Hortonworks Inc. 2011–2018. All rights reserved1
The First Mile – Edge and IoT Data Collection
with Apache NiFi and MiNiFi
Andy LoPresto | @yolopey
Sr. Member of Technical Staff at Hortonworks, Apache NiFi PMC & Committer
11 October 2018 Dataworks Summit Singapore

2. © Hortonworks Inc. 2011–2018. All rights reserved2
Gauging Audience Familiarity with NiFi
“What’s a NeeFee?”
No experience with dataflow
No experience with NiFi
“I can pick this up pretty quickly”
Some experience with dataflow
Some experience with NiFi
“I refactored the Ambari
integration endpoint to allow
for mutual authentication
TLS during my coffee break”
Forgotten more about NiFi
than most of us will ever
know

3. © Hortonworks Inc. 2011–2018. All rights reserved3
Agenda
• What is dataflow and what are the challenges?
• Apache NiFi
• IoT Challenges
• Apache MiNiFi
• Exploration
• Community
• All slides provided online, so no need to transcribe

4. © Hortonworks Inc. 2011–2018. All rights reserved4
What Is Dataflow?

5. © Hortonworks Inc. 2011–2018. All rights reserved5
What Is Dataflow?
• Moving some content from A to B
• Content could be any bytes
• Logs
• HTTP
• XML
• CSV
• Images
• Video
• Telemetry
Producers A.K.A
Things
Anything
AND
Everything
Internet!
Consumers
• User
• Storage
• System
• …More Things

6. © Hortonworks Inc. 2011–2018. All rights reserved6
Moving Data Effectively Is Hard
“Data Pipeline” https://xkcd.com/2054/

7. © Hortonworks Inc. 2011–2018. All rights reserved7
• Standards
• Formats
• Protocols
• Veracity
• Validity
• Schemas
• Partitioning/Bun
dling
Data
Dataflow Challenges in 3 Categories
Infrastructure
• “Exactly Once”
Delivery
• Ensuring
Security
• Overcoming
Security
• Credential
Management
• Network
People
• Compliance
• “That
[person|team|g
roup]”
• Consumers
Change
• Requirements
Change
• “Exactly Once”
Delivery

8. © Hortonworks Inc. 2011–2018. All rights reserved8
Raise your hand if you want to maintain Python scripts for the rest of your life
Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs

9. © Hortonworks Inc. 2011–2018. All rights reserved9
What Is Apache NiFi?

10. © Hortonworks Inc. 2011–2018. All rights reserved10
• Guaranteed delivery
• Data buffering
• Backpressure
• Pressure release
• Prioritized queuing
• Flow specific QoS
• Latency vs. throughput
• Loss tolerance
Key Features
Apache NiFi
• Data provenance
• Supports push and pull models
• Recovery/recording
a rolling log of fine-grained history
• Visual command and control
• Flow templates
• Pluggable, multi-tenant security
• Designed for extension
• Clustering

11. © Hortonworks Inc. 2011–2018. All rights reserved11
Flowfiles Are Like HTTP Data
HTTPData FlowFile
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g
Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT
ETag: "45b6-834-49130cc1182c0"
Accept-Ranges: bytes
Content-Length: 13
Connection: close
Content-Type: text/html
Hello world!
Standard FlowFile Attributes
Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'fileSize’ Value: '23609'
FlowFile Attribute Map Content
Key: 'filename’ Value: '15650246997242'
Key: 'path’ Value: './’
Binary Content *
Header
Content

12. © Hortonworks Inc. 2011–2018. All rights reserved12
User Interface
Less of this… … more of this

13. © Hortonworks Inc. 2011–2018. All rights reserved13
Deeper Ecosystem Integration: 274+ Processors,
57 Controller Services
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
HTTP
Syslog
Email
HTML
Image
HL7
FTP
UDP
XML
SFTP
AMQP
WebSocket
Parse Records Convert Records

14. © Hortonworks Inc. 2011–2018. All rights reserved22
What Are the IoT Challenges?

15. © Hortonworks Inc. 2011–2018. All rights reserved23
IoT Challenges
• Limited computing capability
• Limited power/network
• Restricted software
library/platform availability
• No UI
• Physically inaccessible
• Not frequently updated
• Competing standards/protocols
• Scalability
• Privacy & Security
@_lennart

16. © Hortonworks Inc. 2011–2018. All rights reserved24
• When the Mirai attack has its own
Wikipedia page, that’s not good
• Hackers stole high-roller database from
casino via aquarium thermometer
connected to internet (04/2018)
Recent Examples

17. © Hortonworks Inc. 2011–2018. All rights reserved25
• Runs on JVM
• Provides UI for flow design & monitoring
• Security built-in
• TLS, authentication/authorization, encrypted data
• Handles practically any format/protocol
NiFi Solves Everything*

18. © Hortonworks Inc. 2011–2018. All rights reserved26
• NiFi supports AMQP, MQTT, UDP, TCP,
HTTP(S), CEF, JMS, (S)FTP, AWSIoT
• With a little pruning, NiFi can run on a
Raspberry Pi
NiFi for IoT

19. © Hortonworks Inc. 2011–2018. All rights reserved27
• NiFi is designed to “own the box”
• NiFi 0.7.x started up in about 10-15 minutes on RP3 (593 MB)
• NiFi 1.x started up in about 30 minutes on RP3 (760 MB)
• 33 new processors
• Rewrite for multi tenant authorization
• Complete UI overhaul
So Why Do We Need a Different Solution?

20. © Hortonworks Inc. 2011–2018. All rights reserved28
Enter Apache MiNiFi

21. © Hortonworks Inc. 2011–2018. All rights reserved29
• Get the key parts of NiFi close to where data begins and provide bidirectional
communication
• NiFi lives in the data center — give it an enterprise server or a cluster of them
• MiNiFi lives as close to where data is born and is a guest on that device or system
• IoT
• Connected car
• Legacy hardware
Apache NiFi Subproject: MiNiFi

22. © Hortonworks Inc. 2011–2018. All rights reserved30
• NiFi is big
• 1.7.1 release is 1.2 GB compressed
• Can be modified to run in restricted environments, but requires manual surgery
• Provides UI, provenance query, etc.
• Runs on dedicated machines/clusters — “owns the box”
• MiNiFi lives at the edge
• No UI
• 0.5.0 Java release is 67 MB, C++ release is 6.1 MB (0.2.0 fits on a floppy disk)
• “Good guest”
Why Build MiNiFi?

23. © Hortonworks Inc. 2011–2018. All rights reserved31
• MiNiFi Java (v0.5.0)
• Modified version of NiFi
• No UI
• YAML configuration
• Reduced processor count
• 63+ by default, more
available with
additional NARs
• MiNiFi C++ (v0.5.0)
• Written from scratch
• 33 processors by default
• Bi-directional site-to-site & provenance data
Flavors of MiNiFi

24. © Hortonworks Inc. 2011–2018. All rights reserved32
NiFi vs. MiNiFi Java Processes
NiFi Framework
Components
MiNiFi
NiFi Framework
User Interface
Components
NiFi

25. © Hortonworks Inc. 2011–2018. All rights reserved33
• NiFi
• Design flows
• Aggregate data from many
sources
• Perform routing/analysis/SEP
• MiNiFi
• Receive flows
• Collect data
• Send for processing
How Does MiNiFi Interact with NiFi?

26. © Hortonworks Inc. 2011–2018. All rights reserved34
• We’ve been imagining EDGE to CORE as a bi-directional linear system
• Let’s expand
that to the real
world
Let’s Add Dimensionality

27. © Hortonworks Inc. 2011–2018. All rights reserved35
• Data tagging/provenance
• Governance from edge (geopolitical
restrictions)
• Security (encryption, certificate-based
authentication)
• Low latency (immediate reactions &
decision-making)
What Does MiNiFi Provide? Connected Car Reference Platform Box
Tuner + DSRC CardConnectivity Card

28. © Hortonworks Inc. 2011–2018. All rights reserved36
MiNiFi on a Connected Car
Comprehension
Collection
CAN Bus
Gateway
MCU MCU MCU
Ethernet /
Ethernet AVB
Local
Interconnect
Network
Yet to be
established
protocol
Listen Ethernet Listen LINListen CAN Listen <>
Parse CAN Parse Ethernet Parse LIN Parse <>
Processing / Synthesis
Route
Transmit Execute PrioritizeFilter

29. © Hortonworks Inc. 2011–2018. All rights reserved37
MiNiFi on a Connected Car

30. © Hortonworks Inc. 2011–2018. All rights reserved38
• Site-to-Site
• NiFi protocol
• Two implementations
• Raw socket
• HTTP(S)
• Secured with mutual authentication TLS
• HTTP(S), (S)FTP, JMS, Syslog, File, Email, Process
MiNiFi Exfil

31. © Hortonworks Inc. 2011–2018. All rights reserved39
Edge Data Exploration

32. © Hortonworks Inc. 2011–2018. All rights reserved40
• IoT Device generating log messages
• Need to encrypt data on device
• Need to prioritize some data for
unreliable network connectivity
• Transmit data to central node
• Decrypt data and analyze
• Make determinations and modify live
flow
Scenario

33. © Hortonworks Inc. 2011–2018. All rights reserved41
• Simulate the log generation
• Schedule is customizable
• Script can write to dynamic location
NiFi as Test Harness/Environment

34. © Hortonworks Inc. 2011–2018. All rights reserved42
• Tails a log file
• Logs the raw contents (can be
multiple lines in time window)
• Splits into individual lines
• Filters the content
• Using parity of the timestamp
• Prioritizes
• Encrypts using AES/GCM
• Exfils to remote NiFi
Build the MiNiFi Flow

35. © Hortonworks Inc. 2011–2018. All rights reserved43
• Save as template from NiFi
• Run $ ./bin/config.sh transform
template.xml config.yml
• MiNiFi flow ready to run*
*Still need to set up TLS & encrypted properties
Export from NiFi to MiNiFi

36. © Hortonworks Inc. 2011–2018. All rights reserved44
• NiFi TLS Toolkit makes certificates &
keystores simple (and secure)
• Copy encrypted property value from
flow.xml.gz to config.yml (flow repo)
Setting up Crypto

37. © Hortonworks Inc. 2011–2018. All rights reserved45
• All data transmitted over TLS
is encrypted
• On NiFi, automatically
decrypted
• Attributes visible
• Content still encrypted
because of EncryptContent
processor
• Can serve as secure route for
follow-on systems
If We Really Have TLS, Why Encrypt?

38. © Hortonworks Inc. 2011–2018. All rights reserved46
• Receive the data over S2S
• Log the incoming messages
• Decrypt content
• Log again
Process Data In NiFi

39. © Hortonworks Inc. 2011–2018. All rights reserved47
Does It Work?

40. © Hortonworks Inc. 2011–2018. All rights reserved48
• Increase the write frequency
• Check that newer records (within tail
window) with higher priority arrive first
Prioritization?

41. © Hortonworks Inc. 2011–2018. All rights reserved49
• Window Aggregator
• If >60% odd in window, switch prioritization
• Encrypt with different keys for different tags & send to different follow-on systems
• Spotty network? Tell MiNiFi to cache low priority and send in batches
• MiNiFi rollover & pruning of monitored log
• Exfil MiNiFi provenance data to NiFi
Next Steps

42. © Hortonworks Inc. 2011–2018. All rights reserved50
Community

43. © Hortonworks Inc. 2011–2018. All rights reserved51
Community Example
• Jeremy Dyer
• Alexa + MiNiFi + Dyer 2.0
http://www.opensourcedad.com/apache/minifi-cpp/2016/12/18/poop-scale.html

44. © Hortonworks Inc. 2011–2018. All rights reserved52
What’s Next?

45. © Hortonworks Inc. 2011–2018. All rights reserved53
• NiFi 1.8.0 — … Oct 2018 (170+ Jiras)
• Jetty, DB improvements
• Auto load-balancing queues
• TLS Toolkit w/ external CA
• Record processor improvements
• MiNiFi C++ 0.5.0 — 6 June 2018
• MiNiFi Java 0.5.0 — 7 July 2018
• NiFi Registry 0.3.0 — 25 Sept 2018
Introducing Apache NiFi Registry
New Announcements

46. © Hortonworks Inc. 2011–2018. All rights reserved54
• Previously, flows were exported via XML
templates
• Didn’t contain sensitive values
• Couldn’t be updated in-place
• No tracking system
• NiFi Registry brings asset management
as first-class citizen to NiFi
• Flows can be versioned
• Flows can be promoted between
environments
Introducing Apache NiFi Registry 0.3.0
NiFi Registry for Dataflows

47. © Hortonworks Inc. 2011–2018. All rights reserved55
Community Health

48. © Hortonworks Inc. 2011–2018. All rights reserved56
Apache NiFi site
https://nifi.apache.org
Subproject MiNiFi site
https://nifi.apache.org/minifi/
Subscribe to and collaborate at
dev@nifi.apache.org
users@nifi.apache.org
Submit Ideas or Issues
https://issues.apache.org/jira/browse/NIFI
Follow us on Twitter
@apachenifi
Learn More and Join us

49. © Hortonworks Inc. 2011–2018. All rights reserved57
More NiFi Today…
Title Time Room
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi 1400 – 1440 Orchard
Dataflow Management From Edge to Core with Apache NiFi 1450 – 1330 Orchard

50. © Hortonworks Inc. 2011–2018. All rights reserved58
Thank you
alopresto@hortonworks.com | alopresto@apache.org | @yolopey
github.com/alopresto/slides