A head start on cloud native event driven applications - bigdatadays

A Head Start on Cloud-native
Event Driven Applications
S. Suhothayan (Suho)
Director WSO2
@suhothayan

Increasing demand is causing disaggregation

Characteristics of Cloud Nativeness
●
●
●
●
Other Key Characteristics
●
●
●
Source: https://medium.com/walmartlabs/cloud-native-application-architecture-a84ddf378f82

Nation of Event Driven Applications
●
●
●
●
queues topics
●
●
●
●

Stateless Event Driven Applications
●
●
● Highly scalable!
with HTTP/TCP
with JMS/Kafka/NATS

Full size Image Area with text
Not all apps are stateless!

Some Stateful Use Cases!
●
●
●
●
●

Building Stateful Applications
remember previous events
● preserved during system failure
● shared with new nodes
● databases
● distributed cache
● in-memory

Best Approach for Stateful Apps
High Performance
Scalability
Fault Tolerant

There is no single tool for all cases!

Solution to High Performance, Scalable, and Fault Tolerant Apps
● Keep data in memory.
● periodic state snapshots
● Use cache to preload data.

Scalability mapreduce
Deployment of Scalable Stateful Apps

How to implement all this?

Streaming and Event Processing Frameworks

Characteristics Of Typical Big Data Frameworks
●
●
●
Issues:
●
●
●

Introducing A Cloud Native Stream Processor
●
● no commercial features
● Docker and Kubernetes
● CI/CD
● SQL like query language
graphical tool
●

Image Area
Working with Siddhi
● Siddhi Editor.
●
Siddhi Test Framework
●
○ Java/Python
○ bare metal/VM
○ Docker
○ Kubernetes

We are happy to announce Siddhi 5.1
●
●
●
●
●

Streaming SQL
@app:name('Alert-Processor')
@source(type='kafka', ..., @map(type='json'))
define stream TemperatureStream (roomNo string, temp double);
@info(name='AlertQuery')
from TemperatureStream#window.time(5 min)
select roomNo, avg(temp) as avgTemp
group by roomNo
insert into AvgTemperatureStream;
Source/Sink & Streams
Window Query
with Rate Limiting

CI/CD Pipeline Reference of Siddhi

Patterns For
Event Driven Data Processing

Patterns For Event Driven Data Processing

Image Area
Scenario: Order Processing
Tasks:
●
●
●
●
●

Consume and Publish Events With
Various Data Formats

Consume and Publish Events With Various Data Formats
transports
●
●
●
●
●
data formats
●

Consume and Publish Events With Various Data Formats
@source(type = mqtt, …, @map(type = json))
define stream OrderStream(custId string, item string, amount int);
@source(type = mqtt, …,
@map(type = json, @attribute(“$.id”,"$.itm” “$.count”)))
define stream OrderStream(custId string, item string, amount int);
{“event”:{“custId”:“15”,“item”:“shirt”,“amount”:2}}
{“id”:“15”,“itm”:“shirt”,“count”:2}

Data Filtering and Preprocessing

Data Filtering and Preprocessing
●
●
●
●
●
●
define stream OrderStream
(custId string, item string, amount int);
from OrderStream[item!=“unknown”]
select default(custId, “internal”) as custId,
item,
ifThenElse(amount<0, 0, amount) as amount,
insert into CleansedOrderStream;

Date Transformation

Date Transformation
●
●
●
● 60+ extensions
●
json:getDouble(json,"$.amount") as amount
str:concat(‘Hello ’,name) as greeting
amount * price as cost
time:extract('DAY', datetime) as day
myFunction(item, price) as discount

Database Integration and Caching

Database Integration and Caching
●
●

In-memory Table
define stream CleansedOrderStream
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable (name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
In-memory Table
Join Query

Database Integration
@store(type=‘rdbms’, …,)
define table ItemPriceTable(name string, unitPrice double);
on O.item == T.name
Table backed with DB
Join Query

Database Caching
@store(type=‘rdbms’, …, @cache(cache.policy=‘LRU’, … ))
define table ItemPriceTable(name string, unitPrice double);
on O.item == T.name
Table with Cache
Join Query

Service Integration and Error Handling

HTTP gRPC
●
●
Service Integration and Error Handling
200
4**
Handle response based on
status code

SQL for HTTP Service Integration
@sink(type='http-call', publisher.url="http://mystore.com/discount",
sink.id="discount", @map(type='json'))
define stream EnrichedOrderStream (custId string, item string, price double);
@source(type='http-call-response', http.status.code="200",
sink.id="discount", @map(type='json',
@attributes(custId ="trp:custId", ..., price="$.discountedPrice")))
define stream DiscountedOrderStream (custId string, item string, price double);
Call service
Consume Response

Error Handling Options
● Log and drop
● Wait and back pressure
● Divert events to another stream

Events Diverted Into Error Stream
@onError(action='stream')
@sink(type='http', publisher.url = 'http://localhost:8080/logger',
on.error='stream', @map(type = 'json'))
from !DiscountedOrderStream
select custId, item, price, _error
insert into FailedEventsTable;
Diverting connection failure
events into table.

Data Summarization

Data Summarization
●
○
○
○
●
○
○
●
●
●
●
●
●
●
●
●

Summarizing Data Over Shorter Period Of Time
from DiscountedOrderStream#window.time(10 min)
select custId, sum(price) as totalPrice
group by custId
insert into AlertStream;
Window query
with aggregation and
rate limiting

Aggregation Over Multiple Time Granularities
𝝀
●
●
define aggregation OrderAggregation
from OrderStream
select custId, itemId, sum(price) as total, avg(price) as avgPrice
group by custId, itemId
aggregate every sec ... year;
Query

Data Retrieval from Aggregations
from OrderAggregation
within "2019-10-06 00:00:00",
"2019-10-30 00:00:00"
per "days"
select total as orders;

Rule Processing

Rule Processing
● single event
○
● collection of events
○
○
● event occurrence order
○
○
○

Alert Based On Event Occurrence Order
define stream OrderStream (custId string, orderId string, ...);
define stream PaymentStream (orderId string, ...);
from every (e1=OrderStream) ->
not PaymentStream[e1.orderId==orderId] for 15 min
select e1.custId, e1.orderId, ...
insert into PaymentDelayedStream;
Non occurrence of event

Serving Online and Predeﬁned
ML Models

Serving Online and Predeﬁned ML Models
●
○
●
○
○
●
○
○
○ from OrderStream
#pmml:predict(“/home/user/ml.model”,custId, itemId)
insert into RecommendationStream;
Find recommendations

Scatter-gather and Data Pipelining

Scatter-gather and Data Pipelining
{x,x,x} {x},{x},{x} {y},{y},{y} {y,y,y}

●
●
●
Modularization
Siddhi Runtime
Siddhi App
for data capture
and preprocessing
Siddhi Apps
for each use case
Siddhi App
for common data
publishing logic

Periodically Trigger Events
●
●
●
define trigger FiveMinTrigger at every 5 min;
define trigger WorkStartTrigger at '0 15 10 ? * MON-FRI';
define trigger InitTrigger at 'start';

Realtime Decisions As A Service

Realtime Decisions As A Service
Data Stores REST APIs
●
●
●
HTTP and gRPC REST APIs
●
●

Kubernetes Deployment

Steps On Deploying Siddhi App In Kubernetes

Kubernetes Deployment
$ kubectl get siddhi
NAME STATUS READY AGE
Sample-app Running 2/2 5m

What We Looked At
●
●
●
●
●
https://siddhi.io
suho@wso2.com

A head start on cloud native event driven applications - bigdatadays

A head start on cloud native event driven applications - bigdatadays

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to A head start on cloud native event driven applications - bigdatadays

Similar to A head start on cloud native event driven applications - bigdatadays (20)

Recently uploaded

Recently uploaded (20)

A head start on cloud native event driven applications - bigdatadays