1) Streaming analytics platforms for IoT need to focus on ingesting data from various sources, processing data in real-time, analyzing data, responding to events, and visualizing data. 2) Key areas for building such a platform include using a common abstraction layer, minimizing latency, integrating static and real-time data using lambda architecture, scaling out linearly, enabling rapid application development, and providing data visualization. 3) An example use case of a connected car generates large amounts of data that can be used for various purposes through a streaming analytics platform like predictive maintenance and customized experiences.

1. Make Streaming IoT
Analytics Work for You
Kanishk Mahajan & Dhruv Kumar
Hortonworks May 2016

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT - Relevance
 7.2 Billion active SIM Cards
 25 Billion connected things
 500 Million tweets per day
– Average life of a tweet is 18 minutes
 2 Zettabytes per year of Global IP Traffic
– 80% is unstructured

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The Four Vs of Big Data http://www.ibmbigdatahub.com/infographic/four-vs-big-data

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
5 Attributes of a Streaming Platform
 Ingest
 Process
 Analyze
 Respond
 Visualize

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data Flow versus Stream Analytics for IoT
 Data Flow
– Ingest and route terabytes data into a ”unified firehose”
– Actively performance manage the latency and quality of these data flows –
• Across high variability of data formats, size of data and speed of data
 Real Time Stream Analytics
– Sub second event processing with linear scalability to billions of events
– Predictive Analytics at Scale
• Real time data aggregation across edge nodes while processing 10s of millions of events and
100s of gigabytes per second
– Guaranteed no data loss and events processed in order

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
6 Key Focus Areas for Building a Streaming Platform for IoT
1. Common Abstraction Layer
2. Latency
3. Lambda Architecture
1. “Orchestrate” over static and real time data
4. Scale-out
5. Rapid Application Development
6. Data Visualization

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
1. Common Abstraction Layer
– Select one or more streaming engines
– Select one or more cloud providers
– Select one or more resource managers
– Select one or more event sources
– …Future Proof

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
2. Latency
– 500 millisecond or less- Dashboards, Security Incidents, Asset Performance
– 20 milliseconds or less - Ad Networks, Preventive Maintenance
– …

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
3. Lambda Architecture
– Integrate static and real time data
– Enrichment
– Orchestration of Batch workflows
– Predictive Classification and Scoring

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
4. Scale Out
– Linear Scale out or scale down
– Resource Management
– Handle Transient workloads

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
5. Rapid Application Development
– Industry vertical specific applications
– Aggregations
– Filters
– Multi Stream Correlations
– Splits
– Joins
– Normalizations
– Business Rules Editor

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Focus Areas for Streaming Platform for IoT
6. Data Visualization
– Time Series Visualizations
– Metrics Dashboarding
– Trends
– Comparisons
– Thresholds
– Custom UI extensions

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT - Real Time Stream Analytics versus Streaming Engine
 Abstracts underlying Streaming Engine- Storm, Spark Streaming, Flink..
 OOTB Support for multiple (cloud) event sources - Kafka, AWS Kinesis, Azure Event Hub
 Built-in Operators for Complex Event Processing
 Built-in Real Time Dashboarding- Metrics and Events
 PMML Support
 Pluggable Workflow Management
 Business Rules Editor and Rapid Application Development Framework
 Cloud Deployment
 Scalable Architecture
 Handles different latency requirements

14. IoT: The Big Picture

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
© Hortonworks Inc. 2011
Hortonworks IoT Platform – The Big Picture
Page 15
Data Acquisition
Edge Processing
Real Time Stream Analytics
IoT Services
Rapid Application Development
IoT
ANALYTICS
CLOUD

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
IoT Edge
Data Acquisition & Processing
Device Specific
Protocol
Device Data
Acquisition
Edge
Processing
IoT Analytics
CloudHTTP(s)
Edge Processing
• Reliable Delivery
• Buffering and Flow Control
• Simple Event Processing
• Edge Analytics
High Availability
• Scale out and Clustering
• Multi Data Center Support
Security
• 2 way SSL
• Data Element Masking
• Flexible Routing based on Data
Sensitivity
Kafka Enablement
• Data Ingest and Drain based on
Kafka Consumer and Producer
Support

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Device Management
Agent
• Send Diagnostics
• Receive F/W updates
• Connectivity Logs
Enrollment &
Authentication
• 3-legged OAuth
based flow
• Encryption
• Confidentiality
Registry
• Device Metadata
Catalog
User Linkage
• Device - User Linkage
• May be many - many
between Device and
User
• Temporary or
Persistent
De-
Authorization
• Deregister
• Unlink User
• Disable

18. Use Cases - Connected Car

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car - Generator of Big Data:
http://www.slideshare.net/AditiTechnologies/how-internet-of-things-iot-is-reshaping-the-automotive-sector-infographic

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected Car - Use Cases http://gelookahead.economist.com/infograph/car-os/

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reference Architeucture: Connected Car Platform and Hortonworks
For more details contact sales@hortonworks.com
Hortonworks Data Platform
Core Data
Engineers (CAD)
Systems of Record (ERP)
Customers (CRM)

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi Demo: https://www.youtube.com/watch?v=r4NsWbE4_-I
(Download here: http://hortonworks.com/products/hdf/

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
http://community.hortonworks.com