Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Make Streaming IoT Analytics Work for You

1,742 views

Published on

Download Hortonworks DataFlow (HDF™) here - http://hortonworks.com/downloads/#dataflow. Making Streaming IoT Analytics Work For You With Apache NiFi, Storm, Raspberry Pi and more.

Published in: Data & Analytics

Make Streaming IoT Analytics Work for You

  1. 1. Make Streaming IoT Analytics Work for You Kanishk Mahajan & Dhruv Kumar Hortonworks May 2016
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT - Relevance  7.2 Billion active SIM Cards  25 Billion connected things  500 Million tweets per day – Average life of a tweet is 18 minutes  2 Zettabytes per year of Global IP Traffic – 80% is unstructured
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The Four Vs of Big Data http://www.ibmbigdatahub.com/infographic/four-vs-big-data
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 5 Attributes of a Streaming Platform  Ingest  Process  Analyze  Respond  Visualize
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Flow versus Stream Analytics for IoT  Data Flow – Ingest and route terabytes data into a ”unified firehose” – Actively performance manage the latency and quality of these data flows – • Across high variability of data formats, size of data and speed of data  Real Time Stream Analytics – Sub second event processing with linear scalability to billions of events – Predictive Analytics at Scale • Real time data aggregation across edge nodes while processing 10s of millions of events and 100s of gigabytes per second – Guaranteed no data loss and events processed in order
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 6 Key Focus Areas for Building a Streaming Platform for IoT 1. Common Abstraction Layer 2. Latency 3. Lambda Architecture 1. “Orchestrate” over static and real time data 4. Scale-out 5. Rapid Application Development 6. Data Visualization
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 1. Common Abstraction Layer – Select one or more streaming engines – Select one or more cloud providers – Select one or more resource managers – Select one or more event sources – …Future Proof
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 2. Latency – 500 millisecond or less- Dashboards, Security Incidents, Asset Performance – 20 milliseconds or less - Ad Networks, Preventive Maintenance – …
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 3. Lambda Architecture – Integrate static and real time data – Enrichment – Orchestration of Batch workflows – Predictive Classification and Scoring
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 4. Scale Out – Linear Scale out or scale down – Resource Management – Handle Transient workloads
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 5. Rapid Application Development – Industry vertical specific applications – Aggregations – Filters – Multi Stream Correlations – Splits – Joins – Normalizations – Business Rules Editor
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Focus Areas for Streaming Platform for IoT 6. Data Visualization – Time Series Visualizations – Metrics Dashboarding – Trends – Comparisons – Thresholds – Custom UI extensions
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT - Real Time Stream Analytics versus Streaming Engine  Abstracts underlying Streaming Engine- Storm, Spark Streaming, Flink..  OOTB Support for multiple (cloud) event sources - Kafka, AWS Kinesis, Azure Event Hub  Built-in Operators for Complex Event Processing  Built-in Real Time Dashboarding- Metrics and Events  PMML Support  Pluggable Workflow Management  Business Rules Editor and Rapid Application Development Framework  Cloud Deployment  Scalable Architecture  Handles different latency requirements
  14. 14. IoT: The Big Picture
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved © Hortonworks Inc. 2011 Hortonworks IoT Platform – The Big Picture Page 15 Data Acquisition Edge Processing Real Time Stream Analytics IoT Services Rapid Application Development IoT ANALYTICS CLOUD
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IoT Edge Data Acquisition & Processing Device Specific Protocol Device Data Acquisition Edge Processing IoT Analytics CloudHTTP(s) Edge Processing • Reliable Delivery • Buffering and Flow Control • Simple Event Processing • Edge Analytics High Availability • Scale out and Clustering • Multi Data Center Support Security • 2 way SSL • Data Element Masking • Flexible Routing based on Data Sensitivity Kafka Enablement • Data Ingest and Drain based on Kafka Consumer and Producer Support
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Device Management Agent • Send Diagnostics • Receive F/W updates • Connectivity Logs Enrollment & Authentication • 3-legged OAuth based flow • Encryption • Confidentiality Registry • Device Metadata Catalog User Linkage • Device - User Linkage • May be many - many between Device and User • Temporary or Persistent De- Authorization • Deregister • Unlink User • Disable
  18. 18. Use Cases - Connected Car
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connected Car - Generator of Big Data: http://www.slideshare.net/AditiTechnologies/how-internet-of-things-iot-is-reshaping-the-automotive-sector-infographic
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connected Car - Use Cases http://gelookahead.economist.com/infograph/car-os/
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reference Architeucture: Connected Car Platform and Hortonworks For more details contact sales@hortonworks.com Hortonworks Data Platform Core Data Engineers (CAD) Systems of Record (ERP) Customers (CRM)
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi Demo: https://www.youtube.com/watch?v=r4NsWbE4_-I (Download here: http://hortonworks.com/products/hdf/
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? http://community.hortonworks.com

×