1.
A Head Start on Cloud-native
Event Driven Applications
S. Suhothayan (Suho)
Director WSO2
@suhothayan

2.
About Me
Sri Lanka
Siddhi

3.
Increasing demand is causing disaggregation

4.
Characteristics of Cloud Nativeness
●
●
●
●
Other Key Characteristics
●
●
●
Source: https://medium.com/walmartlabs/cloud-native-application-architecture-a84ddf378f82

5.
Nation of Event Driven Applications
●
●
●
●
queues topics
●
●
●
●

6.
Stateless Event Driven Applications
●
●
● Highly scalable!
with HTTP/TCP
with JMS/Kafka/NATS

7.
Full size Image Area with text
Not all apps are stateless!

8.
Some Stateful Use Cases!
●
●
●
●
●

9.
Building Stateful Applications
remember previous events
● preserved during system failure
● shared with new nodes
● databases
● distributed cache
● in-memory

10.
Best Approach for Stateful Apps
High Performance
Scalability
Fault Tolerant

11.
Full size Image Area with text
There is no single tool for all cases!

12.
Solution to High Performance, Scalable, and Fault Tolerant Apps
● Keep data in memory.
● periodic state snapshots
● Use cache to preload data.

13.
Scalability mapreduce
Deployment of Scalable Stateful Apps

14.
How Checkpointing Works?

15.
How Checkpointing Works? ...

16.
How Checkpointing Works? ...

17.
How Checkpointing Works? ...

18.
How Checkpointing Works? ...

19.
How Checkpointing Works? ...

20.
Full size Image Area with text
How to implement all this?

21.
Streaming and Event Processing Frameworks

22.
Characteristics Of Typical Big Data Frameworks
●
●
●
Issues:
●
●
●

23.
Introducing A Cloud Native Stream Processor
●
● no commercial features
● Docker and Kubernetes
● CI/CD
● SQL like query language
graphical tool
●

24.
Image Area
Working with Siddhi
● Siddhi Editor.
●
Siddhi Test Framework
●
○ Java/Python
○ bare metal/VM
○ Docker
○ Kubernetes

25.
We are happy to announce Siddhi 5.1
●
●
●
●
●

26.
Streaming SQL
@app:name('Alert-Processor')
@source(type='kafka', ..., @map(type='json'))
define stream TemperatureStream (roomNo string, temp double);
@info(name='AlertQuery')
from TemperatureStream#window.time(5 min)
select roomNo, avg(temp) as avgTemp
group by roomNo
insert into AvgTemperatureStream;
Source/Sink & Streams
Window Query
with Rate Limiting

27.
Web Based Source Editor

28.
Web Based Graphical Editor

29.
CI/CD Pipeline Reference of Siddhi

30.
Full size Image Area with text
Patterns For
Event Driven Data Processing

31.
Patterns For Event Driven Data Processing

32.
Image Area
Scenario: Order Processing
Tasks:
●
●
●
●
●

33.
Full size Image Area with text
Consume and Publish Events With
Various Data Formats

34.
Consume and Publish Events With Various Data Formats
transports
●
●
●
●
●
data formats
●

35.
Consume and Publish Events With Various Data Formats
@source(type = mqtt, …, @map(type = json))
define stream OrderStream(custId string, item string, amount int);
@source(type = mqtt, …,
@map(type = json, @attribute(“$.id”,"$.itm” “$.count”)))
define stream OrderStream(custId string, item string, amount int);
{“event”:{“custId”:“15”,“item”:“shirt”,“amount”:2}}
{“id”:“15”,“itm”:“shirt”,“count”:2}

36.
Full size Image Area with text
Data Filtering and Preprocessing

37.
Data Filtering and Preprocessing
●
●
●
●
●
●
define stream OrderStream
(custId string, item string, amount int);
from OrderStream[item!=“unknown”]
select default(custId, “internal”) as custId,
item,
ifThenElse(amount<0, 0, amount) as amount,
insert into CleansedOrderStream;

38.
Full size Image Area with text
Date Transformation

39.
Date Transformation
●
●
●
● 60+ extensions
●
json:getDouble(json,"$.amount") as amount
str:concat(‘Hello ’,name) as greeting
amount * price as cost
time:extract('DAY', datetime) as day
myFunction(item, price) as discount

40.
Full size Image Area with text
Database Integration and Caching

41.
Database Integration and Caching
●
●

42.
In-memory Table
define stream CleansedOrderStream
(custId string, item string, amount int);
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable (name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
In-memory Table
Join Query

43.
Database Integration
define stream CleansedOrderStream
(custId string, item string, amount int);
@store(type=‘rdbms’, …,)
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable(name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
Table backed with DB
Join Query

44.
Database Caching
define stream CleansedOrderStream
(custId string, item string, amount int);
@store(type=‘rdbms’, …, @cache(cache.policy=‘LRU’, … ))
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable(name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
Table with Cache
Join Query

45.
Full size Image Area with text
Service Integration and Error Handling

46.
HTTP gRPC
●
●
Service Integration and Error Handling
200
4**
Handle response based on
status code

47.
SQL for HTTP Service Integration
@sink(type='http-call', publisher.url="http://mystore.com/discount",
sink.id="discount", @map(type='json'))
define stream EnrichedOrderStream (custId string, item string, price double);
@source(type='http-call-response', http.status.code="200",
sink.id="discount", @map(type='json',
@attributes(custId ="trp:custId", ..., price="$.discountedPrice")))
define stream DiscountedOrderStream (custId string, item string, price double);
Call service
Consume Response

48.
Error Handling Options
● Log and drop
● Wait and back pressure
● Divert events to another stream

49.
Events Diverted Into Error Stream
@onError(action='stream')
@sink(type='http', publisher.url = 'http://localhost:8080/logger',
on.error='stream', @map(type = 'json'))
define stream DiscountedOrderStream (custId string, item string, price double);
from !DiscountedOrderStream
select custId, item, price, _error
insert into FailedEventsTable;
Diverting connection failure
events into table.

50.
Full size Image Area with text
Data Summarization

51.
Data Summarization
●
○
○
○
●
○
○
●
●
●
●
●
●
●
●
●

52.
Summarizing Data Over Shorter Period Of Time
define stream DiscountedOrderStream (custId string, item string, price double);
from DiscountedOrderStream#window.time(10 min)
select custId, sum(price) as totalPrice
group by custId
insert into AlertStream;
Window query
with aggregation and
rate limiting

53.
Aggregation Over Multiple Time Granularities
𝝀
●
●
define aggregation OrderAggregation
from OrderStream
select custId, itemId, sum(price) as total, avg(price) as avgPrice
group by custId, itemId
aggregate every sec ... year;
Query

54.
Data Retrieval from Aggregations
from OrderAggregation
within "2019-10-06 00:00:00",
"2019-10-30 00:00:00"
per "days"
select total as orders;

55.
Full size Image Area with text
Rule Processing

56.
Rule Processing
● single event
○
● collection of events
○
○
● event occurrence order
○
○
○

57.
Alert Based On Event Occurrence Order
define stream OrderStream (custId string, orderId string, ...);
define stream PaymentStream (orderId string, ...);
from every (e1=OrderStream) ->
not PaymentStream[e1.orderId==orderId] for 15 min
select e1.custId, e1.orderId, ...
insert into PaymentDelayedStream;
Non occurrence of event

58.
Full size Image Area with text
Serving Online and Predefined
ML Models

59.
Serving Online and Predefined ML Models
●
○
●
○
○
●
○
○
○ from OrderStream
#pmml:predict(“/home/user/ml.model”,custId, itemId)
insert into RecommendationStream;
Find recommendations

60.
Full size Image Area with text
Scatter-gather and Data Pipelining

61.
Scatter-gather and Data Pipelining
{x,x,x} {x},{x},{x} {y},{y},{y} {y,y,y}

62.
●
●
●
Modularization
Siddhi Runtime
Siddhi App
for data capture
and preprocessing
Siddhi Apps
for each use case
Siddhi App
for common data
publishing logic

63.
Periodically Trigger Events
●
●
●
define trigger FiveMinTrigger at every 5 min;
define trigger WorkStartTrigger at '0 15 10 ? * MON-FRI';
define trigger InitTrigger at 'start';

64.
Full size Image Area with text
Realtime Decisions As A Service

65.
Realtime Decisions As A Service
Data Stores REST APIs
●
●
●
HTTP and gRPC REST APIs
●
●

66.
Full size Image Area with text
Kubernetes Deployment

67.
Steps On Deploying Siddhi App In Kubernetes

68.
Kubernetes Deployment

69.
Kubernetes Deployment
$ kubectl get siddhi
NAME STATUS READY AGE
Sample-app Running 2/2 5m

70.
What We Looked At
●
●
●
●
●
https://siddhi.io
suho@wso2.com