This document discusses streaming analytics and the Internet of Things. It describes some challenges of IoT such as data volume, processing speed requirements, and data storage needs. It then discusses WSO2's IoT and analytics platforms, which can perform real-time, batch, and predictive analytics on streaming IoT and other data. The analytics platform uses Siddhi to analyze event streams using techniques like filtering, windowing, pattern detection, joins, and more. Real-time analytics examples include alerts, counting, correlation, and learning models.

1. October 2016
Streaming Analytics and Internet
of Things
Geesara Prathap(geesara@wso2.com)

2. Challenges
2
• How fast do we need results?
• How much data to keep?
• Common language ?
• Do we have centralized data storage and
processing units?
• Knowledge of the past data only?

3. **
IoT is not new !!
Source: http://community.arm.com/groups/internet-of-things/blog/2014/06

4. IoT Ecosystem

5. WSO2 IoT Platform

6. Analytics Platform
6
WSO2 Analytics platform uniquely combine
simultaneous real time and batch analytics with
predictive analytics to run data from IoT, mobile, and
web apps into actionable insights.

7. 7
Analytics Platform

8. Analytics Strategy
8
Single platform to address all analytics styles.
Batch Analytics: analytics on data at-rest, running typically
every hour or every day, and focused on historical analytics
dashboards and reports
Real time Analytics: analyze event streams in real-time and
detects patterns and conditions
Predictive Analytics: leverages machine learning to create
a mathematical model allowing to predict future behavior.
Interactive Analytics: execute queries on the fly on top of
data at rest.

9. IoT / Edge Analytics
9

10. Streaming Analytics in Other Words
10
● Gather data from multiple sources
● Correlate data streams over time
● Find interesting occurrences
● Notify

11. Basic Building Blocks
11
● Receivers: Data collection point, associated to a
specific data connector
● Publishers: Data publishing point,
associated to a specific data
connector
● Event Streams: Event data flowing
through the system
● Execution Plans: Execution pipeline applied to event
streams
● Siddhi: Codename for the streaming engine
● Siddiqi: SQL-like query language

12. 12
Extensible Receiver Architecture

13. 13
Extensible Publisher Architecture

14. Event Streams
14
● Event stream is a sequence of
events
● Event streams are defined by
stream definition
● Event streams have inflows and
outflows
● Inflows can be from
○ Event receivers
○ Execution plans
● Outflows are to
○ Event publishers
○ Execution plans

15. Data Connectors
15
● The following connectors are available out of the box
Source: Email, File, JMS, Kafka, MQTT, SOAP, Websocket, Thrift,
Binary, Log and JMX receiver
Sink: RDBMS, Cassandra, SMS, Email, File, HTTP, JMS, Kafka,
MQTT, SOAP, Websocket, Thrift, Binary
● Incoming/ outgoing data can be mapped using XPath,
regular expressions, or JSON paths
● Data connectors are common across the analytics platform

16. Real-time Analytics Patterns
● Simple counting (e.g. failure count)
● Counting with Windows (e.g. failure count every hour)
● Preprocessing: filtering, transformations, (e.g data cleanup)
● Alerts, thresholds (e.g Alarm on high temperature)
● Data correlation, Detect missing events detecting erroneous
data( e.g detecting failed sensors)
● Joining event streams (e.g. detect a hit on soccer ball)
● Merge with data in a database, collect update data
conditionally

17. Real-time Analytics Patterns
● Detecting event sequence patterns( e.g. small transaction
followed by large transaction)
● Tracking - follow some related entity’s state in space, time etc.
(e.g location of airline baggage, vehicle, tracking wild life)
● Detect trends- Rise, turn, fall, outliers, Complex trends like
triple bottom etc., (e.g algorithmic trading, SLA, load
balancing)
● Learning a model (e.g. predictive maintenance)
● Predicting next value and corrective actions (e.g automated
car)

18. CEP = SQL for Real-time Analytics
● Easy to follow from SQL
● Expressive, short, and sweet
● Define core operations that covers 90% of
problems
● Let’s experts dig in when they like!
Let’s look at the core operation

19. Operators: Filters
Assume a temperature stream
Here weather: convertFtoC() is a user defined function. They are used to
extend the language
Usecases:
- Alerts, thresholds, (e.g Alarm on high temperature)
- Preprocessing: filtering, transformation (e.g data cleanup)

20. Operators: Windows and Aggregation
Support many window types
- Batch windows, Sliding windows, Custom windows
Usecases
- Simple counting ( e.g failure count)
- Counting with Windows ( e.g failure count every hour)

21. Operators: Patterns
Models a followed by relation: e.g. event AS followed by event B
Very powerful tool for tracking and detecting patterns
Usecases
- Detecting event sequence patterns
- Tracking
- Detect trends

22. Operators: Joins
Models a followed by relation: e.g. event AS followed by event B
Very powerful tool for tracking and detecting patterns
Usecases
- Detecting event sequence patterns
- Tracking
- Detect trends

23. Real-time Dashboard

24. TFL Traffic Analytics

25. CONTACT US !