Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Spring with Apache NiFi
NiFi likes to move it…at least once
© Hortonworks Inc. 2011 – 2015. All Rights Reserved
Oleg Zhura...
Page2 © Hortonworks Inc. 2014-2016
Agenda
• Vision
• The Key Concepts
• Demo!
• Proposed Roadmap
Page3 © Hortonworks Inc. 2014-2016
….what many think their architecture looks like
Dataflow
Process and
Analyze Data
Acqui...
Page4 © Hortonworks Inc. 2014-2016
….what it really looks like
Page5 © Hortonworks Inc. 2014-2016
Modern data processing concerns
• Multiple sources of data
• Geo distribution
• Multipl...
Page6 © Hortonworks Inc. 2014-2016
Modern data concerns summary
• Multiple sources of data
• Geo distribution
• Multiple p...
Page7 © Hortonworks Inc. 2014-2016
So what is NiFi?
NiFi is a technology that provides the ability to
consolidate heteroge...
Page8 © Hortonworks Inc. 2014-2016
Product or Framework?
• NiFi is a product
• IDE for data flow design
• Data Flow contro...
Page9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Three key concepts
• Manage the flow of information
• Data Prove...
Page10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Apache NiFi – Key Features
• Guaranteed delivery
• Data bufferi...
Page11 © Hortonworks Inc. 2014-2016
Product or Framework?
• NiFi is a product
• UI for data flow design, control and manag...
Page12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Flow Based Programming (FBP)
FBP Term NiFi Term Description
Inf...
Page13 © Hortonworks Inc. 2014-2016
“All about that connectivity”
Could be looked as one of the extension to Application l...
Page14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Basics of Connecting Systems
For every connection,
these must a...
Page15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Extension / Integration Points
NiFi Term Description
Flow File ...
Page16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Let’s write some code!
DEMO
Page17 © Hortonworks Inc. 2014-2016
Agenda
Introducing Apache NiFi
• A Simple Vision
• The Key Concepts
• Demo!
• Proposed...
Page18 © Hortonworks Inc. 2014-2016
Proposed Roadmap (in progress within community)
NiFi
• Multi-tenant authorization
• HA...
Page19 © Hortonworks Inc. 2014-2016
Upcoming SlideShare
Loading in …5
×

Spring with ApacheNiFi

1,943 views

Published on

SpringOne Platform 2016
Speaker: Oleg Zhurakousky; Principal Architect, Hortonworks

Spring Integration has long captured the hearts and minds of the developers world wide for its emphasis on simplicity, modularity and productivity when it comes to all things related to work-flow orchestration and complex event processing and is successfully used in the variety of BigData solutions. Apache NiFi, on the other hand, is a new addition to the already rich BigData technology stack.

Can the two complement one another?

This hands-on talk, that is based on real field work and consisting of live demos and code, will provide a quick introduction to Apache NiFi, demonstrate its core features while concentrating on WHY/WHERE and HOW of integrating the two technologies

Published in: Technology
  • Be the first to comment

Spring with ApacheNiFi

  1. 1. Spring with Apache NiFi NiFi likes to move it…at least once © Hortonworks Inc. 2011 – 2015. All Rights Reserved Oleg Zhurakousky, Hortonworks; Twitter@z_oleg; GitHub@olegz
  2. 2. Page2 © Hortonworks Inc. 2014-2016 Agenda • Vision • The Key Concepts • Demo! • Proposed Roadmap
  3. 3. Page3 © Hortonworks Inc. 2014-2016 ….what many think their architecture looks like Dataflow Process and Analyze Data Acquire Data Store Data
  4. 4. Page4 © Hortonworks Inc. 2014-2016 ….what it really looks like
  5. 5. Page5 © Hortonworks Inc. 2014-2016 Modern data processing concerns • Multiple sources of data • Geo distribution • Multiple protocols for data transport • New technologies and products • New data processing paradigms • Streaming, Event Sourcing • Security and encryption • New type of users
  6. 6. Page6 © Hortonworks Inc. 2014-2016 Modern data concerns summary • Multiple sources of data • Geo distribution • Multiple protocols for data transport • New technologies and products • New data processing paradigms • Streaming, Event Sourcing • Security and encryption • New type of users Modern applications are now data-centric rather then data-source-centric
  7. 7. Page7 © Hortonworks Inc. 2014-2016 So what is NiFi? NiFi is a technology that provides the ability to consolidate heterogeneous sources of data into one cohesive data flow while addressing concerns described before.
  8. 8. Page8 © Hortonworks Inc. 2014-2016 Product or Framework? • NiFi is a product • IDE for data flow design • Data Flow control and management • Out-of-the-box support for key concepts and features of Flow-based Paradigm/Programming (FBP)
  9. 9. Page9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Three key concepts • Manage the flow of information • Data Provenance • Secure the control plane and data plane
  10. 10. Page10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Apache NiFi – Key Features • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine- grained history • Visual command and control • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering
  11. 11. Page11 © Hortonworks Inc. 2014-2016 Product or Framework? • NiFi is a product • UI for data flow design, control and management • Out-of-the-box support for key concepts of Flow- based Paradigm/Programming (FBP) • NiFi is a framework • Flow-based Programming (FBP) • Extension model • API
  12. 12. Page12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Flow Based Programming (FBP) FBP Term NiFi Term Description Information Packet FlowFile Each object moving through the system. Black Box FlowFile Processor Performs the work, doing some combination of data routing, transformation, or mediation between systems. Bounded Buffer Connection The linkage between processors, acting as queues and allowing various processes to interact at differing rates. Scheduler Flow Controller Maintains the knowledge of how processes are connected, and manages the threads and allocations thereof which all processes use. Subnet Process Group A set of processes and their connections, which can receive and send data via ports. A process group allows creation of entirely new component simply by composition of its components.
  13. 13. Page13 © Hortonworks Inc. 2014-2016 “All about that connectivity” Could be looked as one of the extension to Application layer of an OSI model: 1. Asynchronous Processes 2. Reliable Process Connectivity 3. Provenance NiFi Flow-based programming (FPB) Application Enterprise Integration Patterns (EIP) Presentation Session Transport Network Data Link Physical
  14. 14. Page14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Basics of Connecting Systems For every connection, these must agree: 1. Protocol 2. Format 3. Schema 4. Priority 5. Size of event 6. Frequency of event 7. Authorization access 8. Relevance P1 Producer C1 Consumer
  15. 15. Page15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Extension / Integration Points NiFi Term Description Flow File Processor Push/Pull behavior. Custom UI Flow File Comparator Used to establish priority of FlowFiles in a queue Reporting Task Used to push data from NiFi to some external service (metrics, provenance, etc..) Controller Service Used to enable reusable components / shared services throughout the flow The REST API Allows clients to connect to pull information, change behavior, etc.. Native Ports Input/Output ports connect NiFi flows to the outside world without explicit third party messaging layer.
  16. 16. Page16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Let’s write some code! DEMO
  17. 17. Page17 © Hortonworks Inc. 2014-2016 Agenda Introducing Apache NiFi • A Simple Vision • The Key Concepts • Demo! • Proposed roadmap
  18. 18. Page18 © Hortonworks Inc. 2014-2016 Proposed Roadmap (in progress within community) NiFi • Multi-tenant authorization • HA Data & Control • Enhanced user experience • Registry for Templates and Extensions • Version managed flows • Zero downtime upgrades “MiNiFi” • Agent model • C&C • Secure data path • Scale down • Full data provenance
  19. 19. Page19 © Hortonworks Inc. 2014-2016

×