Yifeng Jiang gives a presentation introducing Apache Nifi. He begins with an overview of himself and the agenda. He then provides an introduction to Nifi including terminology like FlowFile and Processor. Key aspects of Nifi are demonstrated including the user interface, provenance tracking, queue prioritization, cluster architecture, and a demo of real-time data processing. Example use cases are discussed like indexing JSON tweets and indexing data from a relational database. The presentation concludes that Nifi is an easy to use and powerful system for processing and distributing data with 90 built-in processors.

1. Introducing
Apache Nifi
Yifeng Jiang
Solutions Engineer, Hortonworks
© Hortonworks Inc. 2011 – 2015. All Rights Reserved

2. About Me
Yifeng Jiang
• Solutions Engineer, Hortonworks
• Apache HBase book author
• I like hiking
• Twitter: @uprush

3. Page 3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Agenda
• Introduction to Nifi
• Nifi Demo
• Nifi Use Case

4. Page 4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Introduction to Apache NiFi

5. Page 5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Nifi Overview
Nifi is an easy to use, powerful, and
reliable system to process and distribute
data.

6. Page 6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi Terminology
FlowFile
• Unit of data moving through the system
• Content + Attributes (key/value pairs)
Processor
• Performs the work, can access FlowFiles
Connection
• Links between processors
• Queues that can be dynamically prioritized
Process Group
• Set of processors and their connections
• Receive data via input ports, send data via output ports

7. Page 7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi - User Interface
• Drag and drop processors to build a flow
• Start, stop, and configure components in real time
• View errors and corresponding error messages
• View statistics and health of data flow
• Create templates of common processor & connections

8. Page 8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi - Provenance
• Tracks data at each point as it flows
through the system
• Records, indexes, and makes
events available for display
• Handles fan-in/fan-out, i.e. merging
and splitting data
• View attributes and content at given
points in time

9. Page 9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi - Queue Prioritization
• Configure a prioritizer per
connection
• Determine what is important for your
data – time based, arrival order,
importance of a data set
• Funnel many connections down to a
single connection to prioritize across
data sets
• Develop your own prioritizer if
needed

10. Page 10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi - Architecture

11. Page 11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Nifi Cluster
• Nifi Cluster Manager
• Nifi Cluster Nodes
• Primary Node
• Isolated Processor
OS/Host
JVM
Flow Controller
Web Server
Processor 1 Extension N
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
OS/Host
JVM
Flow Controller
Web Server
Processor 1 Extension N
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
OS/Host
JVM
NiFi Cluster Manager – Request Replicator
Web Server
Master
NiFi Cluster
Manager (NCM)
OS/Host
JVM
Flow Controller
Web Server
Processor 1 Extension N
FlowFile
Repository
Content
Repository
Provenance
Repository
Local Storage
Slaves
NiFi Nodes

12. Page 12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi Demo

13. Page 13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Nifi Demo
• The demo cluster: ambari deployment, NCM
• Real-time indexing in Solr & Banana
• Nifi UI
• Flow statistics
• Data provenance, event details, replay
• Add a Processor to push data to Kafka
• Nifi data on the node
• Flow file repository
• Content repository
• Provenance repository

14. Page 14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Site to Site -- Flow
Nifi Cluster A (source)
Nifi Cluster B (destination)
Site to site
Remote
Process Group
Flow file attributes transferred

15. Page 15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Site to Site – Data Provenance
Nifi Cluster A (source)
Nifi Cluster B (destination)
Event details at cluster B

16. Page 16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
NiFi Use Cases

17. Page 17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Use Cases – Index JSON
1. Pull in Tweets using Twitter API
2. Extract language and text into FlowFile
attributes
3. Get non-empty English tweets
${twitter.text:isEmpty():not():and(
${twitter.lang:equals("en")})}
4. Merge together JSON documents based on
quantity, or time
5. Use dynamic field mappings to select fields for
indexing:

18. Page 18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Use Cases – Index a Relational Database
1. GenerateFlowFile acts a timer to trigger
ExecuteSQL
(Future plans to not require in an incoming FlowFile
to ExecuteSQL NIFI-932)
2. ExecuteSQL performs a SQL query and
streams the results as an Avro datafile
Use expression language to construct a dynamic
date range:
${now():toNumber():minus(60000)
:format(‘YYYY-MM-DD’}
3. Convert Avro to JSON using built in
ConvertAvroToJSON processor
4. Stream JSON update to Solr

19. Page 19 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Built-in Processors
• 90 built-in processors
• Well-defined API
• Easy to implement

20. Page 20 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

21. Page 21 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Thank You