Design a Dataflow in 7 minutes with Apache NiFi/HDF
•Download as PPTX, PDF•
16 likes•12,107 views
How to create a real-time dataflow in 7 Minutes with Hortonworks DataFlow, powered by Apache NiFi”.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Design a Dataflow in 7 minutes with Apache NiFi/HDF
Similar to Design a Dataflow in 7 minutes with Apache NiFi/HDF (20)
More from Hortonworks
More from Hortonworks (20)
Recently uploaded
Recently uploaded (20)
Design a Dataflow in 7 minutes with Apache NiFi/HDF
- 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Create a live dataflow in minutes How would that change your business?
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Add processor for data intake. Time: 1 minute 1 Drag and drop processor from top menu
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Choose the specific processor 2 Choose one of the processors – currently 170+ available
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example: Pick Twitter Processor
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Configure the processor. Time: 2 minutes 3 4 Select processor and choose option to Configure Adjust parameters as required
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Another processor for data output. Time: 1 minute 5 6 Filter for and select a “Put” processor Drag and drop processor from top menu
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Configure second processor. Time: 1 minute 7 Configure 2nd processor
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connect processors, configure connection. 2 minutes Configure Connection8 Note: Sample Flow is different from previous example of PutHDFS. This dataflow is PutFile. Same concepts apply.
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Click Start to Begin Processing. Time total: 7 minutes 9 Click start “play” to begin processing (will run continuously until you select stop)
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved See Processors Update with Real Time Changes 10 As data flows, GUI interface updates in real time. 11 If destination is stopped or unable to receive, queue builds
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Dynamically adjust and tune data flow as needed 12 Dynamically configure/ start/ stop/ tune/ reroute change/ pause dataflows as needed.
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Powerful Tools to Quickly Replicate, Group, Repurpose, Tune and Test in Real-Time 13 14 Create a new template Group multiple processes together to create a process group
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Provenance Means Real-Time Traceability of: Data Flow Data Content Data Context
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Watch Real Time Flow of Data: Data Provenance Select Data Provenance15
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Trace Lineage of a Particular Piece of Data Icon for Data Lineage16
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Every Change to Data is Tracked in Real-Time: processing, views Every event is traceable 17
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Real-Time Updates of Dataflow: Traceable Context & Content Know immediately both context and content18
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Easily access and trace changes to dataflow
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Audit trail of Hortonworks DataFlow User Actions
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? Hortonworks Community Connection: Data Ingestion and Streaming https://community.hortonworks.com/
Editor's Notes
- HDF supports over 90 difference processors to accelerate the process of ingesting and processing data. There are ready-made “off the shelf” processors for data collection, data processing. For example – in alphabetical order, not necessarily popularity: EncryptContent, ExecuteFlumeSink, ExecuteFlumeSource, ExecuteSQL, ExtractHL7, GetFTP, GetHTTP, PutKafka, MergeContent, MonitorActivity, PutEmail, PutHDFS, SpltJSON, TransformXML.
- There are many different processors, some of which are designed to simplify collection of big data from popular data sources. Twitter is one of them. Others include:
- This is a very unique capability of dataflow – the ability to see processors update in real time This gives data developers and data scientists the ability to quickly verify hypothesis and as well enable on-time decision making – within the relevant time-window.
- Once the data flow is established, it can be dynamically manipulated, replicated and transformed. This removes the need to develop code in a test environment, and then porting to a production environment. Being able to immediately test within the production environment, accelerates the time to insight. And all of this is tracked so when you get to the point of “what did I try before again”, or “what happened last time”, it is readily accessible via the GUI interface.
- HDF provides very fine-grained, high fidelity reporting about the origins of data, how it was used, who used it etc.