SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 30 day free trial to unlock unlimited reading.
A head start on cloud native event driven applications - bigdatadays
Characteristics of cloud native apps.
Problems in implementing event-driven stateful applications.
Siddhi: Cloud-Native Stream Processor.
Patterns of implementing event-driven applications.
Deploying event-driven applications on Kubernetes with Siddhi and NATS.
Characteristics of cloud native apps.
Problems in implementing event-driven stateful applications.
Siddhi: Cloud-Native Stream Processor.
Patterns of implementing event-driven applications.
Deploying event-driven applications on Kubernetes with Siddhi and NATS.
9.
Building Stateful Applications
remember previous events
● preserved during system failure
● shared with new nodes
● databases
● distributed cache
● in-memory
10.
Best Approach for Stateful Apps
High Performance
Scalability
Fault Tolerant
11.
Full size Image Area with text
There is no single tool for all cases!
12.
Solution to High Performance, Scalable, and Fault Tolerant Apps
● Keep data in memory.
● periodic state snapshots
● Use cache to preload data.
13.
Scalability mapreduce
Deployment of Scalable Stateful Apps
36.
Full size Image Area with text
Data Filtering and Preprocessing
37.
Data Filtering and Preprocessing
●
●
●
●
●
●
define stream OrderStream
(custId string, item string, amount int);
from OrderStream[item!=“unknown”]
select default(custId, “internal”) as custId,
item,
ifThenElse(amount<0, 0, amount) as amount,
insert into CleansedOrderStream;
38.
Full size Image Area with text
Date Transformation
39.
Date Transformation
●
●
●
● 60+ extensions
●
json:getDouble(json,"$.amount") as amount
str:concat(‘Hello ’,name) as greeting
amount * price as cost
time:extract('DAY', datetime) as day
myFunction(item, price) as discount
40.
Full size Image Area with text
Database Integration and Caching
52.
Summarizing Data Over Shorter Period Of Time
define stream DiscountedOrderStream (custId string, item string, price double);
from DiscountedOrderStream#window.time(10 min)
select custId, sum(price) as totalPrice
group by custId
insert into AlertStream;
Window query
with aggregation and
rate limiting
53.
Aggregation Over Multiple Time Granularities
𝝀
●
●
define aggregation OrderAggregation
from OrderStream
select custId, itemId, sum(price) as total, avg(price) as avgPrice
group by custId, itemId
aggregate every sec ... year;
Query
54.
Data Retrieval from Aggregations
from OrderAggregation
within "2019-10-06 00:00:00",
"2019-10-30 00:00:00"
per "days"
select total as orders;
55.
Full size Image Area with text
Rule Processing
56.
Rule Processing
● single event
○
● collection of events
○
○
● event occurrence order
○
○
○
57.
Alert Based On Event Occurrence Order
define stream OrderStream (custId string, orderId string, ...);
define stream PaymentStream (orderId string, ...);
from every (e1=OrderStream) ->
not PaymentStream[e1.orderId==orderId] for 15 min
select e1.custId, e1.orderId, ...
insert into PaymentDelayedStream;
Non occurrence of event
58.
Full size Image Area with text
Serving Online and Predefined
ML Models
59.
Serving Online and Predefined ML Models
●
○
●
○
○
●
○
○
○ from OrderStream
#pmml:predict(“/home/user/ml.model”,custId, itemId)
insert into RecommendationStream;
Find recommendations
60.
Full size Image Area with text
Scatter-gather and Data Pipelining
61.
Scatter-gather and Data Pipelining
{x,x,x} {x},{x},{x} {y},{y},{y} {y,y,y}
62.
●
●
●
Modularization
Siddhi Runtime
Siddhi App
for data capture
and preprocessing
Siddhi Apps
for each use case
Siddhi App
for common data
publishing logic
63.
Periodically Trigger Events
●
●
●
define trigger FiveMinTrigger at every 5 min;
define trigger WorkStartTrigger at '0 15 10 ? * MON-FRI';
define trigger InitTrigger at 'start';
64.
Full size Image Area with text
Realtime Decisions As A Service
65.
Realtime Decisions As A Service
Data Stores REST APIs
●
●
●
HTTP and gRPC REST APIs
●
●
66.
Full size Image Area with text
Kubernetes Deployment