This document discusses streaming analytics for IoT applications. It begins by outlining how IoT is generating new sources of data and value. It then discusses streaming analytics as an approach to analyze data in motion from IoT sensors. The document reviews open source streaming analytics platforms and commercial solutions, highlighting vendors in the emerging streaming analytics as a service space. It concludes by offering tips for organizations to evaluate how streaming analytics can create value from their IoT data.

1. Streaming Analytics for IoT-Oriented Applications
Adrian Bowles, PhD
Founder, STORM Insights, Inc.
Lead Analyst, AI, Aragon Research
info@storminsights.com
MAY 11, 2017

2. Context - New Data, New Demands, New Opportunities
IOT: Impact, Infrastructure & Implications
Streaming Analytics - What, How, and Why Now?
Open Source Strategies
Vendor Solutions
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
AGENDA: STREAMING ANALYTICS FOR IOT-ORIENTED APPLICATIONS

3. When everything is connected…
New sources of data emerge
New sources of value emerge
Old assumptions must be challenged
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
THE IMPACT OF THE IOT

4. Copyright (c) 2014 by Umbrellium Ltd.
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
SENSORS ARE EVERYWHERE

5. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
DATA ON THE MOVE

6. Today:
Delta Airlines processes
5,000,000 business events per day
Pratt & Whitney jet engine:
5,000 sensors producing
10GB/s/per engine.
Formula 1 car sensors produce
about 1.2GB/s
and we need to predict the future…
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
AS THE SCOPE CHANGES, SO MUST THE SOLUTIONS

7. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
2-TIER IOT ARCHITECTURE
Sensors/
Devices Compute Engine
Data Center
Cloud
Cluster
Network

8. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
2-TIER IOT ARCHITECTURE
Sensors/
Devices Compute Engine
Data Center
Cloud
Cluster
Network

9. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
2-TIER IOT ARCHITECTURE
Sensors/
Devices Compute Engine
Data Center
Cloud
Cluster
Network

10. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
3-TIER IOT ARCHITECTURE
Sensors/
Devices
Compute EngineGateways
Data Center
Cloud
Cluster
Network

11. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
3-TIER IOT ARCHITECTURE ENABLES DISTRIBUTED ANALYTICS
Sensors/
Devices
Compute Engine
Data Center
Cloud
Cluster
Network
Compute
Engines

12. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
3-TIER IOT ARCHITECTURE ENABLES DISTRIBUTED INTELLIGENCE & ANALYTICS
Sensors/
Devices
Train the Deep Learning Model
Data Center
Cloud
Cluster
Network
Compress & Run
The DL Model

13. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
A ROLE FOR COLLABORATION AND STANDARDS

14. Predix
Jasper Control Center
Watson IoT
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
IOT PLATFORMS

15. Context - New Data, New Demands, New Opportunities
IOT: Impact, Infrastructure & Implications
Streaming Analytics - What, How, and Why Now?
Open Source Strategies
Vendor Solutions
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
AGENDA: STREAMING ANALYTICS FOR IOT-ORIENTED APPLICATIONS

16. PRESCRIPTIVE
ANALYTICS
DESCRIPTIVE
PREDICTIVE
WHAT TO DO
FUTURE OR MISSING VALUES
KNOWN VALUES
ANALYTICS

17. STATIC
DATA
STORED
DATA

18. STATIC
DIVERTED OR
SAMPLED
DATA
STOP AND FRISK
STORED
DATA

19. STATIC
DIVERTED OR
SAMPLED
STREAMING
DATA
IN MOTION
STOP AND FRISK
STORED
DATA

20. STREAMINGIN MOTION
DATA
“You could not step twice into the same river.”
Heraclitus 535BC-474BC
To understand the contents (analyze)…
Divert the flow?
Pool the data?
Evaluate everything without changing the flow?
Sample? (catch and release?)

21. STREAMING
ANALYTICS
DESCRIPTIVE
DATA
PREDICTIVE
STREAMING ANALYTICS: MOVE THE PROCESS TO THE DATA
STREAMING ANALYTICS: STATISTICAL ANALYSIS OF DATA IN MOTION

22. DataSources
Store
Process/Transform
Key
Data Flows on the Edges, Queries on the Vertices
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
CONVENTIONAL DATA ARCHITECTURE

23. DataSources
Store
Process/Transform
Observe
Key
Data Flows on the Edges, Queries Everywhere
Sampling vs Monitoring Everything…
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STREAMING DATA ARCHITECTURE

24. 440
880
440
880
THE PROBLEM WITH SAMPLING…

25. 440
880
440
880
440
880
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
THE PROBLEM WITH SAMPLING…

26. Context - New Data, New Demands, New Opportunities
IOT: Impact, Infrastructure & Implications
Streaming Analytics - What, How, and Why Now?
Open Source Strategies
Vendor Solutions
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
AGENDA: STREAMING ANALYTICS FOR IOT-ORIENTED APPLICATIONS

27. RELEVANT APACHE SOFTWARE FOUNDATION OPEN SOURCE PROJECTS
Apache Flink: “open-source stream processing framework for distributed, high-
performing, always-available, and accurate data streaming applications.”
Apache Samza: “a distributed stream processing framework. It uses Apache Kafka
for messaging, and Apache Hadoop YARN to provide fault tolerance, processor
isolation, security, and resource management.”
Apache Storm: “a free and open source distributed realtime computation system.
Storm makes it easy to reliably process unbounded streams of data, doing for
realtime processing what Hadoop did for batch processing.”
Apache Spark Streaming: “Spark Streaming brings Apache Spark's language-
integrated API to stream processing, letting you write streaming jobs the same way
you write batch jobs.”
Apache Apex: “Enterprise-grade unified stream and batch processing engine.”
Registered trademarks or trademarks of The Apache Software Foundation

28. Internode Communication
Distributed Cache
Indexed Results StoreProcessing
Machine Learning
Visualization
Cluster Management
Data
Collection
Data
Delivery
Development
YARN%
Courtesy of STRIIM
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
THE OPEN SOURCE ECOSYSTEM FOR ANALYTICS INFRASTRUCTURE

29. Context - New Data, New Demands, New Opportunities
IOT: Impact, Infrastructure & Implications
Streaming Analytics - What, How, and Why Now?
Open Source Strategies
Vendor Solutions
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
AGENDA: STREAMING ANALYTICS FOR IOT-ORIENTED APPLICATIONS

30. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
COMMERCIAL SOLUTIONS: PLATFORMS FOR STREAMING ANALYTICS AS A SERVICE
IBM Bluemix
Google Cloud Platform
Azure Stream Analytics
Oracle Stream Analytics

31. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Key Trend:
Streaming
Analytics for IOT
as a Service
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STREAMING ANALYTICS AS A SERVICE

32. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STREAMING ANALYTICS AS A SERVICE
Key Trend:
Streaming
Analytics for IOT
as a Service

33. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Key Trend:
Streaming
Analytics for IOT
as a Service
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STREAMING ANALYTICS AS A SERVICE

34. Copyright (c) 2016 by STORM Insights Inc. All Rights reserved.
Key Trend:
Streaming
Analytics for IOT
as a Service

35. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
COMMERCIAL SOLUTIONS
SAP Event Stream Processor
Tibco Event Analytics
Cisco Connected Streaming Analytics
SAS Event Stream Processing

36. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STREAMING ANALYTICS - EMERGING VENDORS TO WATCH
Integrate Analyze

37. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
DATATORRENT
Source: DataTorrent.com

38. Mul$ple'data''
sources'
Hadoop (HDFS)
(Existing)
ETL Jobs
Batch'/'
High5Latency'
Real'$me'/'
Low5Latency' ODS/ EDW
Real-Time
Applications
Legacy
Applications
Spark Hive
Big Data
Applications
Users'
CDC'
Flume'
KaAa'
Logs'
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STRIIM
Integration
&
Intelligence
Source: STRIIM.com

39. Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
STREAMANALYTIX
Source: streamanalytix.com

40. Data Sharpening
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.
ZOOMDATA
Source: zoomdata.com

41. Do you have or can you capture streaming data that can increase your value proposition?
Data about your product that can improve performance, reliability, predictability…
Can you create value from new analysis of open data?
Adding your own data/algorithms to open data creates value.
Tips:
Start by evaluating the emerging open source de facto standards.
Choose an infrastructure that allows you to evaluate live streaming data in the context of
relevant historical data.
It’s All About the Data
GETTING STARTED…
Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.

42. adrian@storminsights.com
adrian@aragonresearch.com
Twitter @ajbowles
Skype ajbowles
If you would like to connect on LinkedIn, please let
me know that you that you registered for the
Smart Data webinar series.
Upcoming SmartData Webinar Dates & Topics
June 8 Machine Learning Case Studies
July 13 Advances in NLP I: Understanding
August 10 Organizing Data and Knowledge:
The Role of Taxonomies and Ontologies
KEEP IN TOUCH!
New Content from Aragon Research
AragonResearch.com

43. RESOURCES
https://azure.microsoft.com/en-us/services/stream-analytics
apex.apache.org
http://flink.apache.org
https://spark.apache.org/streaming/
http://samza.apache.org
https://streamanalytix.com
Slide 4 IOT Search: https://thingful.net
https://www.ibm.com/cloud-computing/bluemix/data-analytics
https://www.zoomdata.com
http://www.striim.com
https://www.datatorrent.com
https://aws.amazon.com/kinesis/streams/