Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hortonworks Data In Motion Series Part 4

10,754 views

Published on

How real-world enterprises leverage Hortonworks DataFlow/Apache NiFi to to create real-time data flows in record time to enable new business opportunities, improve customer retention, accelerate big data projects from months to minutes through increased efficiency and reduced costs.
On-Demand webinar: http://hortonworks.com/webinar/paradigm-shift-business-usual-real-time-dataflows-record-time/

Published in: Technology
  • Be the first to comment

Hortonworks Data In Motion Series Part 4

  1. 1. Harnessing Data-in-Motion with Hortonworks DataFlow A Paradigm Shift to Business as Usual: Real World Use Cases of Real-Time DataFlows in Record Time Anna Yong Product Marketing Haimo Liu Product Manager
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Overview of Hortonworks DataFlow (HDF) How HDF transforms data movement – months to minutes HDF Use Cases Real-World HDF Use Cases Apache NiFi Case Studies from Hadoop Summit Other Apache NiFi/HDF Use Cases
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connected Data Platforms
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Payment Tracking Due Diligence Social Mapping Product Design M & A Call Analysis Machine Data Defect Detecting Factory Yields Customer Support Basket Analysis Segments Customer Retention Sentiment Analysis Optimize Inventories Supply Chain Cross- Sell Vendor Scorecards Ad Placement Cyber Security Disaster Mitigation Investment Planning Ad Placement Risk Modeling Proactive Repair Inventory Predictions Next Product Recs OPEX Reduction Historical Records Mainframe Offloads Device Data Ingest Rapid Reporting Digital Protection Data as a Service Fraud Prevention Public Data Capture INNOVATE RENOVATE E X PLO R E O PTIMIZE TR A NS FO R M ACTIVE ARCHIVE ETL ONBOARD DATA ENRICHMENT DATA DISCOVERY SINGLE VIEW PREDICTIVE ANALYTICS
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved  Constrained  High-latency  Localized context  Hybrid – cloud / on-premises  Low-latency  Global context Core Infrastructure Hortonworks DataFlow Manages Data in Motion Regional InfrastructureSources
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks DataFlow Manages Data in Motion Core InfrastructureSources  Constrained  High-latency  Localized context  Hybrid – cloud / on-premises  Low-latency  Global context Regional Infrastructure
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 451 Analyst Report on Stream Processing and Streaming Integration http://hortonworks.com/info/value-streaming-integration/
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Dataflow Management
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Problems Today: Timely Access to Data and Decisions http://diginomica.com/2016/04/22/royal-mail-starts-to-deliver-on-hortonworks-data-in-motion-promise “HDF helps us to streamline the flow of data and build models and visualisations quickly, so that my team can work iteratively with business colleagues on building solutions that work for the business.“ Royal Mail
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDP HORTONWORKS DATA PLATFORM Powered by Apache Hadoop HDF Makes Big Data Ingest Easy Complicated, messy, and takes weeks to months to move the right data into Hadoop HDP HORTONWORKS DATA PLATFORM Streamlined, Efficient, Easy HDP HORTONWORKS DATA PLATFORM Powered by Apache Hadoop
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks DataFlow, Powered by Apache NiFi. Demo Time
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Create a live dataflow in minutes How would that change your business?
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Add processor for data intake. Time: 1 minute 1 Drag and drop processor from top menu
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Choose the specific processor 2 Choose one of the processors – currently 170+ available
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example: Pick Twitter Processor
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Configure the processor. Time: 2 minutes 3 4 Select processor and choose option to Configure Adjust parameters as required
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Another processor for data output. Time: 1 minute 5 6 Filter for and select a “Put” processor Drag and drop processor from top menu
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Configure second processor. Time: 1 minute 7 Configure 2nd processor
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connect processors, configure connection. 2 minutes Configure Connection8 Note: Sample Flow is different from previous example of PutHDFS. This dataflow is PutFile. Same concepts apply.
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Click Start to Begin Processing. Time total: 7 minutes 9 Click start “play” to being processing (will run continuously until you select stop)
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDF Use Cases
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Core Infrastructure Hortonworks DataFlow Use Cases Regional InfrastructureSources Dataflow Management • On-ramp into Hadoop • Log Collection / Splunk Optimization • Cyber Security • IoT Ingestion • Deliver data into stream processing engines Real-time Event Processing (Kafka, Storm) Move data between from on-prem and cloud environments
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Optimize Log Analytics with Content Based Routing
  24. 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT Data Ingestion Constrained High-Latency Localized Context Hybrid – Cloud/On-Premise Low-Latency Global Context Resolves real world connectivity and transmission issues often overlooked by assuming connectivity is always perfect
  25. 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enterprise Data Movement and Hybrid Cloud • Seamlessly fuse dataflows between data centers • Data center to data center, • Remote location to data center, • Data center to cloud HDF Between Data Centers HDF HDF Remote to Data Center HDF HDF HDF HDF Between Data Centers & Cloud HDF
  26. 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved © Hortonworks Inc. 2011 Stream Processing Page 26 Data Acquisition Edge Processing Real Time Stream Analytics Rapid Application Development IoT ANALYTICS CLOUD
  27. 27. Real World Use Cases
  28. 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Royal Mail’s Journey Satisfied customers with better service levels & improved retention Actionable intelligence improves customer experience  Data management transformed to deliver specific, actionable insights to line of business departments  Analysis delivered within days & weeks rather than months per project deadlines  Churn modelling project identified customers at risk by vertical sector in order to take preventative action  Improved accuracy of delivery times for business customers & highlighted trends related to volumes of mail expected  Governance & compliance simplified due to central data platform Satisfied customers with better service levels & improved retention P R E D I C T I V E A N A L Y T I C S A C T I V E A R C H I V E S I N G L E V I E W A C T I V E A R C H I V E S I N G L E V I E W E T L O N B O A R D D A T A E N R I C H M E N T P R E D I C T I V E A N A L Y T I C S D A T A E N R I C H M E N T Parcel distribution Customer Acquisition S I N G L E V I E W Customer Support Inventory Predictions Investment Planning Data-as-a- Service Public data capture Rapid reporting EDW offload OPEX reduction Innovate Renovate New Data Products
  29. 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Open Energi Uses HDF for Electricity Demand Response As a result of its investment in Hortonworks Dataflow, Open Energi is Already:  Reducing costs thanks to 10-15% less data being transmitted across a mobile network  Creating a full transparent trail for data provenance that Open Energi can share with customers  Enabling line of business teams to contribute to building dataflow rules and processes  Standardizing the output of data across various end point devices Open Energi: hortonworks.com/blog/data-fuel-open-energi-virtual-power-station-hortonworks-dataflow/
  30. 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Prescient Traveler transformed the travel risk management market $0.5MM Savings in development costs due to Hortonworks HDF 700% Improvement in analyst productivity in determining actual threats 49,000 Number of data sources currently being analyzed to identify threats
  31. 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Centrica’s Journey 1.3 Million Smart Meters EDW Offload Mobile App for Customer Sites Ingest 300 GB per Day Product Cross-Sell Building a Data-Driven Energy Utility Business Self-service analytics for 3 million customers in UK & North America HDP and HDF simplify IT estate Ingest of 300 GB/day rationalizes maintenance work Personalized customer communications replaced impersonal up-sell messages Legacy EDWs decommisioned Innovate Renovate Smart, Efficient Homes D A T A D I S C O V E R Y D A T A E N R I C H M E N T P R E D I C T I V E A N A L Y T I C S S I N G L E V I E W A C T I V E A R C H I V E E T L O N B O A R D SINGLE VIEW S I N G L E V I E W P R E D I C T I V E A N A L Y T I C S On-site customer data capture Optimized engineering schedule Tailored servicing Customer sentiment
  32. 32. Apache NiFi Case Studies: Hadoop Summit San Jose
  33. 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved From Zero to DataFlow: http://www.slideshare.net/HadoopSummit/from-zero-to-data-flow-in-hours-with-apache-nifi-64032731
  34. 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Make Streaming Analytics Work For You http://www.slideshare.net/HadoopSummit/make-streaming-analytics-work-for-you-the-devil-is-in-the-details
  35. 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hadoop Summit Keynote: Apache Metron Ingest log data into their cyber security data lake https://youtu.be/Nffx8SKn7l4?t=1h37m50s
  36. 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hadoop Summit Keynote: Improving Customer Experience https://youtu.be/BY_0HB9uyXQ
  37. 37. Other HDF/Apache NiFi Use Cases
  38. 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Hacks & Demos Keynote: Retail Simulation at Hadoop Summit Live Voting, Electronic Conversation, Real-Time Facial Recognition Intro, Demo 1, Demo 2, Demo 3, Demo 4. https://www.youtube.com/watch?v=BY_0HB9uyXQ&feature=youtu.be&t=49m10s
  39. 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved GENIVI Alliance: Open Source, In-Vehicle Infotainment Software The GENIVI Alliance is a nonprofit industry alliance committed to driving the broad adoption of specified, open source, In-Vehicle Infotainment software.
  40. 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved More Use Cases Using Apache NiFi to read children’s books” https://twitter.com/KayLerch/status/721455415456882689
  41. 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Yet More Use Cases https://twitter.com/4everfusiongal/status/7351585225 39855872 https://www.linkedin.com/pulse/making-rain-apache-nifi- jeremy-dyer?trk=prof-post www.linkedin.com/hp/update/6138493082129149952 https://community.hortonworks.com/articles/30636/how-to-simulate-a- sales-executive-with-hdf.html
  42. 42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Even More Use Cases https://community.hortonworks.com/articles/47854/accessing-facebook- page-data-from-apache-nifi.html https://community.hortonworks.com/content/kbentry/32605/runn ing-nifi-on-raspberry-pi-best-practices.html Accessing Facebook Data
  43. 43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved And Even More Use Cases https://community.hortonworks.com/articles/20318/visualize-patients- complaints-to-their-doctors-usi.html Visualize patients' complaints to their doctors using NiFi and Solr/Banana http://hortonworks.com/blog/qualcomm-hortonworks-showcase- connected-car-platform-tu-automotive-detroit/ Connected Car
  44. 44. 44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? Hortonworks Community Connection: Data Ingestion and Streaming https://community.hortonworks.com/ Contact Us: http://hortonworks.com/contact-us/

×