More Related Content
Similar to Apache NiFi 1.0 in Nutshell (20)
More from Koji Kawamura (13)
Apache NiFi 1.0 in Nutshell
- 5. 5 © Hortonworks Inc. 2011 –2016. All Rights Reserved
” NiFi is like digging irrigation ditches as the water flows, rather than building out a
sprinkler system in advance."
“NiFiは事前にスプリンクラーを配備するというより、
水が流れるのに合わせて用水路を整備するようなもんさ”
https://mail-archives.apache.org/mod_mbox/nifi-users/201604.mbox/%3C2FCCBD60-0A79-42F1-9F9B-A121591C826E@apache.org%3E
What’s Apache NiFi?
- 10. 10 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Deeper Ecosystem Integration – New Processors
Processor Description
Publish/ConsumeKafka Two NARs, with kafka 0.9/0.10 client libraries,respectively
JoltTransformJson ManipulateJSON data on the fly, with a preview functionality
GenerateTableFetch Incrementalfetch + parallel fetch against source table partitions
PutHiveQL Ingest to Hive tables
SelectHiveQL Select from Hive tables
PutHiveStreaming ingest streaming data to Hive, leverage Hive streaming API
CovertAvroToORC Format conversation,Avro to ORC
Publish/ConsumeMQTT MQTT is a popular protocolin IoTworld
- 13. 13 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Flow Management
Detailed Break Down of Requirements
à Req 1: Acquire data from various Wearable Device’s Cloud Instances
à Req 2: Move Data from Customer Cloud Instances to on-premise instance
à Req 3: Perform intelligent Routing & Filtering of data. The routing and filtering rules will be often changed at
run-time.
à Req 4: Deliver the data data to various downstream systems. New downstream apps should will always appear
and the data should be fed to it when it comes online.
à Req 5: Parse the device data to standardized format that downstream sysem can understand
à Req 6: Enrich the data with contextual information including patient/customer info (age, gender, etc..)
à Req 7: Recognize the pattern when the resting heart rate exceeds a certain threshold (the insight), and then
create an alert/notification.
à Req 8: Run a Outlier detection model on streaming heart rate that comes in. If the score is above certain
threshold, alert on the heart rate.
Stream Processing & Analytics
- 17. 17 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Connect Components to design your data flow
Component What for?
Processor Purpose built processing unit e.g. GetXXX, PutXXX
Input Port Receiving data endpointbtw Process Groups (local/remote)
Output Port Exposingdata endpoint btw Process Groups (local/remote)
Process Group Musthave, to design well structured data flow
Remote Process Group Enable data transfer btwNiFi deployments via Site-to-Site
Funnel Bundle multiple relationshipsinto one
Template Sharepart of data flow
Label Useful to visually group processors, and description
From left to right
- 20. 20 © Hortonworks Inc. 2011 –2016. All Rights Reserved
NiFi 0.x - Authorization Model
à Previously had role based authorization
– Dataflow Manager (DFM)
– Monitor
– Provenance
– Admin
– Proxy
– NiFi
à Limitation - All or nothing model
– DFM can change everything, Monitor can change nothing
– Can’t give a user ability to modify/view only certain components
– Would require standing up multiple NiFi instances
- 21. 21 © Hortonworks Inc. 2011 –2016. All Rights Reserved
NiFi 1.0 - Authorization Model
à NiFi 1.0 introduces a new delegated authorization model
à Authorize each request based on user identity, action, and resource
– Example for user1 modifying properties on processor1:
• User Identity: user1
• Action: WRITE
• Resource: processor1 (uuid)
à If authorizer says resource not found, parent is checked… if parent isn’t found, parent’s
parent is checked, and so on…
- 22. 22 © Hortonworks Inc. 2011 –2016. All Rights Reserved
NiFi 1.0 – NiFi Managed Authorizer vs. External Authorizer
à Managed Authorizer
– File based persistence
• Could be be extended to other persistence mechanisms
– NiFi UI to manage policies
– NiFi controls authorization logic
à External Authorizer
– Ranger integration
– Ranger UI to manage policies
– Ranger controls authorization logic
- 26. 26 © Hortonworks Inc. 2011 –2016. All Rights Reserved
NiFi 1.0 – Overriding Component Policies
à Component inherit policies
from the closest ancestor
Process Group with policies
defined
à View/Modify policies
handled independently
à Click Override to define a
new policy, then add Users
and Groups
à New Users and Groups
override the inherited
policies (whitelisting)
- 27. 27 © Hortonworks Inc. 2011 –2016. All Rights Reserved
NiFi 1.0 - Multi-Tenancy Example
à Create a Group for Team 1 and a Group for Team 2
à Give Team 1 view & modify for Process Group 1
à Give Team 2 view & modify for Process Group 2
à A user from Team 1 would see:
Can’t see the name of the group and
can’t right-click to configure the
group, but can enter the group
- 46. 46 © Hortonworks Inc. 2011 –2016. All Rights Reserved
What’s Next
à Framework extension
– Distributed data durability (HA
data)
– Configuration management flows
(SDLC)
à Enhanced User Experience
– Template/Extension Registry
– Variable Registry
à Deeper ecosystem integration
à Central Command and Control
à Native Agent (GA)
NiFi MiNiFi
https://cwiki.apache.org/confluence/display/NIFI/Product+requirements
Nifi product requirements Search!