The document outlines key aspects of building streaming applications using the WSO2 Stream Processor, focusing on patterns, architectures, and deployment strategies. It emphasizes the importance of streaming analytics for real-time data processing to derive insights, manage business processes, and optimize operations. Key topics include data preprocessing, integration, summarization, alert generation, event correlation, and real-time predictions utilizing machine learning models.

1. Technical Lead
Patterns for Building Streaming Apps
Mohanadarshan Vivekanandalingam

2. Goal
● Streaming analytics? Streaming apps?
● The architecture of a streaming engine
● Understanding streaming constructs
● Applying patterns when building streaming apps
● Managing streaming patterns
● Deployment patterns

3. Why Streaming ?
Real-time
Near
Real-time
Offline
Constant low
milliseconds &
under
Low milliseconds
to
seconds
10s seconds
to
minutes
● A stream is series of events
● Almost all new data is streaming
● Detects conditions quickly
Image Source : https://www.flickr.com/photos/plusbeautumeurs/33307049175

4. Why Streaming
Apps?
● Identify perishable insights
● Continuous integration
● Orchestration of business
processes
● Embedded execution of
code
● Sense, think, and act in real
time
- Forrester

5. How to Build a
Streaming App
Use a Streaming Processor
● Stream processor handles
data flow, scalability, and
failure - you have to handle
the rest
● Publish data to a topic
● Write an end-to-end flow to
consume events and processes
Code it Yourself
Use a Streaming SQL-based
Streaming Processor
● Write the queries using
streaming SQL

6. Patterns
Streaming Apps

7. ● To understand what stream
processing can do!
● Easy to solve common
problems in stream processing
● Where to use what?
● Learn best practices
Why Patterns for Streaming ?
Image Source : https://www.flickr.com/photos/laurawoodillustration/6986871419

8. 1. Streaming data preprocessing
2. Data store integration
3. Streaming data summarization
4. Interactive data search
5. KPI analysis and alerts
6. Event correlation and trend analysis
7. Real-time predictions
Streaming App Patterns

9. Stream Processor
Core
Streaming Engine
WSO2 Stream
Processor
Events
JMS, Thrift, SMTP, HTTP, MQTT, Kafka
Analytics Fabric
Complex Event
Processing
Incremental Time
Series Aggregation
Machine
Learning
Extension Store
Financialand
BankingAnalytics
RetailAnalytics
LocationAnalytics
OperationalAnalytics
SmartEnergy
Analytics
Custom Analytics
Solutions
...
Solutions
StatusMonitoring
Rule
Mgmt.
Dashboard

10. ● Lightweight, lean, and cloud native
● Easy to learn streaming SQL (Siddhi SQL)
● High performance analytics with just 2 nodes (HA)
● Native support for streaming machine learning
● Long term aggregations without batch analytics
● Highly scalable deployment with exactly-once processing
● Tools for development and monitoring
● Tools for business users to write their own rules
Overview of WSO2 Stream Processor

11. Stream Processing
With WSO2 Stream Processor
Siddhi Streaming App
- Process events in a streaming manner
- Isolated unit with a set of queries, input and
output streams
- SQL Like Query Language
from Sales#window.time(1 hour)
select region, brand, avg(quantity) as AvgQuantity
group by region, brand
insert into LastHourSales ;
Stream
Processor
Siddhi App
{ Siddhi }
Input Streams Output Streams
Filter Aggregate
JoinTransform
Pattern
Siddhi Extensions

12. We need a use case!

13. Use Case
Online Shopping Application
Place Order
Process
Order
Out for Delivery
Process
Payment
BANK

14. Use Case
• Monitor sales, supply, and delivery
• Optimize sales, supply, and delivery
• Predict demand/sales
• Recommend products when shopping
• Manage processing rules online
• Visualize real-time sales and delivery
Online Shopping Application

15. 1. Streaming Data Pre-processing
• Consume events from multiple sources
• Convert them to streams
• Filter events
• Add defaults to missing fields
• Change event stream structure
Filtering, Add Defaults, and Projection
Filter
Transform
Process
Add
Defaults

16. Define Stream
@app:name(‘Online-Shopping-Analytics’)
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);

17. Consume Events:
MQTT, HTTP, TCP, Kafka, JMS,
RabitMQ, etc.
Map Events to
Streams:
JSON, XML, Text, Binary, WSO2Event,
KeyValue, etc.
@app:name(‘Online-Shopping-Analytics’)
@source(type = http, …, @map(type = json, …))
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);

18. Writing the Query
@app:name(‘Online-Shopping-Analytics’)
@source(type = http, …, @map(type = json, …))
define stream ProductPurchaseStream
(userId string, sessionId string, productId
string, qty double, price double);
from ProductPurchaseStream
select *
insert into PossibleDiscountProductStream ;

19. Filter
@app:name(‘Online-Shopping-Analytics’)
@source(type = http, …, @map(type = json, …))
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
from ProductPurchaseStream [qty > 5 and
productId == ‘XYZ’]
select *
insert into PossibleDiscountProductStream ;

20. Transformation &
Defaults
@app:name(‘Online-Shopping-Analytics’)
@source(type = http, …, @map(type = json, …))
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
from ProductPurchaseStream [qty > 5 and
productId == ‘XYZ]
select userId, sessionId, productId, qty,
price * 1.17 as usdPrice,
‘USD’ as currency
insert into PossibleDiscountProductStream ;

21. Functions:
Inbuilt, Custom UDF or
Siddhi Extension
@app:name(‘Online-Shopping-Analytics’)
@source(type = http, …, @map(type = json, …))
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
from ProductPurchaseStream [qty > 5 and
productId == ‘XYZ]
select userId, sessionId, productId, qty,
convertToUSD(price) as usdPrice, ‘USD’ as
currency
insert into PossibleDiscountProductStream ;

22. 2. Data Store Integration
● Allows performing operations with the data store while
processing events on the fly
Store, Retrieve, Remove, and Modify
● Provides a REST endpoint to query Data Store
● Query optimizations using Primary and Indexing keys
● Search ● Insert ● Delete ● Update ● Insert/Update

23. Define Table
(In Memory)
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
define table UserPurchases(userId string,
sessionId string, amount double);

24. Primary Key &
Indexing
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
@primaryKey(‘userId’)
@Index(‘sessionId’)
define table UserPurchases(userId string,
sessionId string, amount double);

25. Table Backed by:
RDBMS, MongoDB, HBase, Cassandra, Solr,
Hazelcast, etc.
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
@store(type=‘rdbms’, … )
@primaryKey(‘userId’)
@Index(‘sessionId’)
define table UserTable(userId string,
sessionId string, amount double);

26. Insert into Table
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
@store(type=‘rdbms’, … )
@primaryKey(‘userId’)
@Index(‘sessionId’)
define table UserTable(userId string,
sessionId string, amount double);
from ShoppingCheckoutStream
select userId, sessionId, amount
insert into UserTable;

27. 3. Streaming Data Summarization
● Can perform aggregations over short and long time periods
● Support for aggregations such as:
○ Sum
○ Count
○ Min/Max
○ Avg
○ etc.
Aggregations Over Time Periods

28. Aggregations Over a
Short Time
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
from ProductPurchaseStream#window.time(1 min)
select productId, sum(qty) totalQty,
currentTimeMillis() as timestamp
group by productId
insert into LastMinPurchaseStream;
Windows Sliding and Batch for Time,
Length, etc.

29. 3. Streaming Data Summarization
Aggregations Over Long Time Periods
• Incremental aggregation for every
– second, minute, hour, day, year
• Support for out-of-order event arrival
• Fast data retrieval from memory and disk
for real time updates
Current Min
Current Hour
Sec
Min
Hour
0 - 1 - 5 ...
- 1
- 2 - 3 - 4 - 64 - 65 ...
- 2
- 124

30. Aggregations Over a
Long Time
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);
define aggregation PurchaseAggregation
from ProductPurchaseStream
select productId, sum(price * qty) as
totalAmount, sum(qty) as noOfItems
group by productId
aggregate every seconds ... years ;
Define Aggregation

31. 4. Interactive Data Search
Search Data Promptly
• Can perform data search on Data
Stores or pre-defined aggregations.
• Supports both REST and Java APIs

32. Data Retrieval
Query
from PurchaseAggregation
on productId == ‘XYZ’
within 2018-05-01 2018-06-01
per ‘day’
select productId, totalAmount, noOfItems ;

33. Dashboard for
Business Users
• Generate dashboard and
widgets
• Fine grained permissions
– Dashboard level
– Widget level
– Data level
• Localization support
• Inter widget communication
• Shareable dashboards with
widget state persistence

35. 5. KPI Analysis and Alerts
Generate Alerts Based on KPIs
• Identify KPIs using
– Filter, ifThenElse, having, etc.
• Send alerts using Sinks

36. Notify with
Event Sinks
define stream ShoppingPaymentStream(userId
string, name string, email string, sessionId
string, totalAmount double, address string,
isSuccess boolean);
@sink(type=‘email’, to=‘{{email}}’
@map(type=‘text’,
@payload(‘‘‘
Hi, {{name}}
Order placed successfully ...’’’))
define stream SuccessfulPaymentStream (userId
string,name string, email string, ...);
from ShoppingPaymentStream [isSuccess == true]
select *
insert into SuccessfulPaymentStream;

37. 6. Event Correlation & Trend Analysis
CEP for Patterns and Sequences
• Identify complex patterns
– Followed by, non-occurrence, etc.
• Identify trends
– Peek, triple bottom, etc.

38. Pattern
Detect non-occurrence
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
define stream ShoppingPaymentStream(userId
string, name string, email string, sessionId
string, totalAmount double, address string,
isSuccess boolean);
from every (e1 = ShoppingCheckoutStream)
-> not ShoppingPaymentStream
[sessionId == e1.sessionId]
for 15 min
select e1.sessionId, e1.totalAmount
insert into PaymentDelayedStream ;

39. Sequences
Identify Decreasing Trend
define stream LastMinPurchaseStream(productId
string, totalQty double, timestamp long);
partition with
(productId of LastMinPurchaseStream)
Begin
from every e1=LastMinPurchaseStream,
e2=LastMinPurchaseStream
[timestamp - e1.timestamp < 10 and
e1.totalQty > totalQty]*,
e3=LastMinPurchaseStream
[timestamp - e1.timestamp > 10 and
e2[last].totalQty > totalQty]
select e1.productId, e1.totalQty as
initialQty, e3.totalQty as finalQty
insert into ContinousSalesReductionStream ;
end;

40. 7. Real-time Predictions
Using Machine Learning
• Use pre-created machine learning
models and perform predictions.
– PMML, TensorFlow, etc.
• Streaming Machine Learning
– Clustering, Classification,
Regression
– Markov Models, Anomaly
detection, etc.
Image Source : https://www.flickr.com/photos/149823084@N08/27871878168/

41. Machine Learning
Models for
Prediction
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
from ShoppingCheckoutStream#pmml:predict
(“/home/user/ml.model”, userId)
select *
Insert into ShoppingPredictionStream ;

42. Continuous Learning
& Prediction
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
define stream ShoppingPredictionStream(userId
string, predictedAmount double);
from ShoppingCheckoutStream
#streamingml:hoeffdingTreeTrain
(‘Model’, userId, amount);
...
from ShoppingCheckoutStream
#streamingml:hoeffdingTreeClassifier
(‘Model’, userId);
Select *
Insert into ShoppingPredictionStream;
Online Training
Online Prediction

43. Managing
Streaming Patterns

44. Business
Rules for Non
Technical
Users
Define your own
business rules from
scratch
Modify templated
complex business
rules using rule
parameters

45. Developer Studio
for Streaming
Apps
Supports both drag n
drop & source editor
Editor
Debugger
Simulation
Testing

47. Graphical Query Builder

48. Deployment
Streaming Apps

49. • High performance
– Process around 100k
events/sec
– Just 2 nodes
– While most others need 5+
• Zero downtime & event Loss
• Incremental state persistence &
recovery
• Simple deployment with RDBMS
– No Zookeeper, Kafka, etc.
• Multi data center support
Minimum HA with 2 Nodes
Stream Processor
Stream Processor
Event Sources
Dashboard
Notification
Invocation
Data Source
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Event
Store

50. • Exactly-once processing
• Fault tolerance
• Highly scalable
• No back pressure
• Distributed development configurations via annotations
• Pluggable distribution options (YARN, K8, etc.)
Distributed Deployment

51. Distributed Deployment with Kafka
Data
Base
Event
Source
Event
Sink
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Kafka Topic
Kafka Topic
Kafka Topic
Kafka
Topic
Kafka
Topic

52. Monitoring
Streaming Apps

53. Status Dashboard
• Understand system performance via
– Throughput
– Latency
– CPU, memory utilizations
• Monitor in various scales
– Node level
– Siddhi app level
– Siddhi query level
Monitor Resource Nodes and Siddhi Apps

56. To Summarize

57. ● Gives you everything you need to build
streaming analytics
○ Manage data streams
○ Powerful Streaming SQL language
○ Dashboards and more
● Can provide 100K+ events per second with two node
HA (most alternatives need 5+ nodes) and can scale
more on top of Kafka
WSO2 Stream Processor

58. ● Why streaming patterns and when to use them
● How WSO2 Stream Processor can be used to build streaming
patterns
● How to develop streaming patterns with WSO2 Stream
Processor
● The business benefit of manageable rules and dashboards
● Patterns to apply for the deployment of streaming apps
Patterns for Streaming Apps

59. THANK YOU
wso2.com