Apache NiFi Meetup - Princeton NJ 2016
•
3 likes•1,163 views
Timothy Spann 14-July-2016 Future of Data Meetup - Princeton, NJ
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Apache NiFi Meetup - Princeton NJ 2016
Similar to Apache NiFi Meetup - Princeton NJ 2016 (20)
More from Timothy Spann
More from Timothy Spann (20)
Recently uploaded
Recently uploaded (20)
Apache NiFi Meetup - Princeton NJ 2016
- 2. 2 © Hortonworks Inc. 2011 –2016. All Rights Reserved Agenda • 6:00pm – 6:45pm Registration and Food • 6:45pm – 7:00pm Introduction and Welcome • 7:00pm – 8:00pm Introduction to Apache NiFi by Tim Spann • What is NiFi? • Demo • Hands-On • 8:00pm – 8:45pm Creating a Custom Apache NiFi Processor by Milind Pandit • 8:45pm – 9:00pm Wrap-up, Upcoming Kafka Talk
- 4. 4 © Hortonworks Inc. 2011 –2016. All Rights Reserved Terminology FlowFile • Unit of data moving through the system • Content + Attributes (key/value pairs) Processor • Performs the work, can access FlowFiles Connection • Links between processors • Queues that can be dynamically prioritized Process Group • Set of processors and their connections • Receive data via input ports, send data via output ports
- 5. 5 © Hortonworks Inc. 2011 –2016. All Rights Reserved Installation Download the binary from here: http://hortonworks.com/downloads/#dataflow Or here: https://nifi.apache.org/download.html Or on Mac: brew install nifi https://nifi.apache.org/docs/nifi-docs/html/getting- started.html#starting-nifi bin/nifi.sh start
- 6. 6 © Hortonworks Inc. 2011 –2016. All Rights Reserved The Future of Data: Actionable Intelligence D A T A I N M O T I O N STORAGE STORAGE GROUP 2GROUP 1 GROUP 4GROUP 3 D A T A A T R E S T INTERNET OF ANYTHING Hortonworks’ unique approach to data-in-motion and data-at-rest powers Actionable Intelligence
- 10. 10 © Hortonworks Inc. 2011 –2016. All Rights Reserved Apache NiFi • Powerful and reliable system to process and distribute data • Directed graphs of data routing and transformation • Web-based User Interface for creating, monitoring, & controlling data flows • Highly configurable - modify data flow at runtime, dynamically prioritize data • Data Provenance tracks data through entire system • Easily extensible through development of custom components [1] https://nifi.apache.org/
- 11. 11 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi - Terminology à FlowFile • Unit of data moving through the system • Content + Attributes (key/value pairs) à Processor • Performs the work, can access FlowFiles à Connection • Links between processors • Queues that can be dynamically prioritized à Process Group • Set of processors and their connections • Receive data via input ports, send data via output ports
- 12. 12 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi - User Interface • Drag and drop processors to build a flow • Start, stop, and configure components in real time • View errors and corresponding error messages • View statistics and health of data flow • Create templates of common processor & connections
- 13. 13 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi - Provenance • Tracks data at each point as it flows through the system • Records, indexes, and makes events available for display • Handles fan-in/fan-out, i.e. merging and splitting data • View attributes and content at given points in time
- 14. 14 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi - Queue Prioritization • Configure a prioritizer per connection • Determine what is important for your data – time based, arrival order, importance of a data set • Funnel many connections down to a single connection to prioritize across data sets • Develop your own prioritizer if needed
- 15. 15 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi - Extensibility à Built from the ground up with extensions in mind à Service-loader pattern for… • Processors • Controller Services • Reporting Tasks • Prioritizers à Extensions packaged as NiFi Archives (NARs) • Deploy NiFi lib directory and restart • Provides ClassLoader isolation • Same model as standard components
- 16. 16 © Hortonworks Inc. 2011 –2016. All Rights Reserved NiFi - Architecture OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage OS/Host JVM NiFi Cluster Manager – Request Replicator Web Server Master NiFi Cluster Manager (NCM) OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage Slaves NiFi Nodes
- 17. 17 © Hortonworks Inc. 2011 –2016. All Rights Reserved Rapid Ecosystem Adoption: 130+ Processors HTTP Syslog Email HTML Image Hash Encrypt Extract TailMerge Evaluate Duplicate Execute Scan GeoEnrich Replace ConvertSplit Translate HL7 FTP UDP XML SFTP Route Content Route Context Route Text Control Rate Distribute Load AMQP
- 20. 20 © Hortonworks Inc. 2011 –2016. All Rights Reserved à https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/ à https://github.com/jfrazee/awesome-nifi à https://dzone.com/articles/getting-started-with-apache-nifi-and-hdf à https://nifi.apache.org/docs.html à https://community.hortonworks.com/articles/4356/getting-started-with-nifi- expression-language-and.html Learning More
- 22. 22 © Hortonworks Inc. 2011 –2016. All Rights Reserved Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
- 23. 23 © Hortonworks Inc. 2011 –2016. All Rights Reserved Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 –2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!