Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache NiFi: Ingesting Enterprise Data At Scale

Ingesting various enterprise data sources including JMS, MQTT, Sensors, JSON, XML, CSV, Text, Images, RDBMS and more. Processing, transforming and storing in HDFS, HBase, Phoenix, Hive and more. Also processing with TensorFlow. Hortonworks and TRAC Intermodal Event. Future of Data Princeton.

Related Books

Free with a 30 day trial from Scribd

See all

Apache NiFi: Ingesting Enterprise Data At Scale

  1. 1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Timothy Spann 2017 Future of Data – Princeton Meetup Hosted by TRAC Intermodal Apache NiFi: Ingesting Enterprise Data @ Scale
  2. 2. DATAWORKS SUMMIT/HADOOP SUMMIT JUNE 13–15, 2017 San Jose McHenry Convention Center REGISTER NOW AND SAVE $1,000 REGISTER NOW! > dataworkssummit.com
  3. 3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Agenda • Apache NiFi RDBMS, EDI, JSON, CSV, Sensors • EDI • https://community.hortonworks.com/content/kbentry/59975/in gesting-edi-into-hdfs-using-hdf-20.html • https://github.com/tspannhw/EnterpriseNIFI
  4. 4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
  5. 5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
  6. 6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Flow Management Flow management + Stream Processing D A T A I N M O T I O N D A T A A T R E S T IoT Data Sources AWS Azure Google Cloud Hadoop NiFi Kafka Storm Others… NiFi NiFi NiFi MiNiFi MiNiFi MiNiFi MiNiFi MiNiFi MiNiFi MiNiFi NiFi HDF 2.1 – Data in Motion Platform Enterprise Services Ambari Ranger Other services
  7. 7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Actionable Insights Architecture Ingestion Simple Event Processing Engine Complex Event Processing Destination Data Bus Build Predictive Model From Historical Data Deploy Predictive Model For Real-time Insights Perishable Insights Historical Insights
  8. 8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Actionable Intelligence Transforms Industrial, Transportation & Utilities Asset Data Customer Surveys Weather & Environmental Service Fleet GPS Data Smart Meter Streams Commodity Prices REVENUE PROTECTION SINGLE VIEW OF CUSTOMER PREDICTIVE EQUIPMENT MAINTENANCE CONSERVATION VOLTAGE REDUCTION COMMODITY TRADING Social Media GIS Data SCADA Outage Histories CIS Records EDW
  9. 9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved What is Apache NiFi? • Created to address the challenges of global enterprise dataflow • Key features: – Visual Command and Control – Data Lineage (Provenance) – Data Prioritization – Data Buffering/Back-Pressure – Control Latency vs. Throughput – Secure Control Plane / Data Plane – Scale Out Clustering – Extensibility
  10. 10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache NiFi What is Apache NiFi used for? • Reliable and secure transfer of data between systems • Delivery of data from sources to analytic platforms • Enrichment and preparation of data: – Conversion between formats – Extraction/Parsing – Routing decisions What is Apache NiFi NOT used for? • Distributed Computation • Complex Event Processing • Complex Rolling Window Operations
  11. 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved NiFi Terminology FlowFile • Unit of data moving through the system • Content + Attributes (key/value pairs) Processor • Performs the work, can access FlowFiles Connection • Links between processors • Queues that can be dynamically prioritized
  12. 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Contact: Timothy Spann @PaaSDeV www.meetup.com/futureofdata-princeton community.hortonworks.com/users/9304/tspann.html
  13. 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
  14. 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!

×