A head start on cloud native event driven applications - bigdatadays

S
A Head Start on Cloud-native
Event Driven Applications
S. Suhothayan (Suho)
Director WSO2
@suhothayan
About Me
Sri Lanka
Siddhi
Increasing demand is causing disaggregation
Characteristics of Cloud Nativeness
●
●
●
●
Other Key Characteristics
●
●
●
Source: https://medium.com/walmartlabs/cloud-native-application-architecture-a84ddf378f82
Nation of Event Driven Applications
●
●
●
●
queues topics
●
●
●
●
Stateless Event Driven Applications
●
●
● Highly scalable!
with HTTP/TCP
with JMS/Kafka/NATS
Full size Image Area with text
Not all apps are stateless!
Some Stateful Use Cases!
●
●
●
●
●
Building Stateful Applications
remember previous events
● preserved during system failure
● shared with new nodes
● databases
● distributed cache
● in-memory
Best Approach for Stateful Apps
High Performance
Scalability
Fault Tolerant
Full size Image Area with text
There is no single tool for all cases!
Solution to High Performance, Scalable, and Fault Tolerant Apps
● Keep data in memory.
● periodic state snapshots
● Use cache to preload data.
Scalability mapreduce
Deployment of Scalable Stateful Apps
How Checkpointing Works?
How Checkpointing Works? ...
How Checkpointing Works? ...
How Checkpointing Works? ...
How Checkpointing Works? ...
How Checkpointing Works? ...
Full size Image Area with text
How to implement all this?
Streaming and Event Processing Frameworks
Characteristics Of Typical Big Data Frameworks
●
●
●
Issues:
●
●
●
Introducing A Cloud Native Stream Processor
●
● no commercial features
● Docker and Kubernetes
● CI/CD
● SQL like query language
graphical tool
●
Image Area
Working with Siddhi
● Siddhi Editor.
●
Siddhi Test Framework
●
○ Java/Python
○ bare metal/VM
○ Docker
○ Kubernetes
We are happy to announce Siddhi 5.1
●
●
●
●
●
Streaming SQL
@app:name('Alert-Processor')
@source(type='kafka', ..., @map(type='json'))
define stream TemperatureStream (roomNo string, temp double);
@info(name='AlertQuery')
from TemperatureStream#window.time(5 min)
select roomNo, avg(temp) as avgTemp
group by roomNo
insert into AvgTemperatureStream;
Source/Sink & Streams
Window Query
with Rate Limiting
Web Based Source Editor
Web Based Graphical Editor
CI/CD Pipeline Reference of Siddhi
Full size Image Area with text
Patterns For
Event Driven Data Processing
Patterns For Event Driven Data Processing
Image Area
Scenario: Order Processing
Tasks:
●
●
●
●
●
Full size Image Area with text
Consume and Publish Events With
Various Data Formats
Consume and Publish Events With Various Data Formats
transports
●
●
●
●
●
data formats
●
Consume and Publish Events With Various Data Formats
@source(type = mqtt, …, @map(type = json))
define stream OrderStream(custId string, item string, amount int);
@source(type = mqtt, …,
@map(type = json, @attribute(“$.id”,"$.itm” “$.count”)))
define stream OrderStream(custId string, item string, amount int);
{“event”:{“custId”:“15”,“item”:“shirt”,“amount”:2}}
{“id”:“15”,“itm”:“shirt”,“count”:2}
Full size Image Area with text
Data Filtering and Preprocessing
Data Filtering and Preprocessing
●
●
●
●
●
●
define stream OrderStream
(custId string, item string, amount int);
from OrderStream[item!=“unknown”]
select default(custId, “internal”) as custId,
item,
ifThenElse(amount<0, 0, amount) as amount,
insert into CleansedOrderStream;
Full size Image Area with text
Date Transformation
Date Transformation
●
●
●
● 60+ extensions
●
json:getDouble(json,"$.amount") as amount
str:concat(‘Hello ’,name) as greeting
amount * price as cost
time:extract('DAY', datetime) as day
myFunction(item, price) as discount
Full size Image Area with text
Database Integration and Caching
Database Integration and Caching
●
●
In-memory Table
define stream CleansedOrderStream
(custId string, item string, amount int);
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable (name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
In-memory Table
Join Query
Database Integration
define stream CleansedOrderStream
(custId string, item string, amount int);
@store(type=‘rdbms’, …,)
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable(name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
Table backed with DB
Join Query
Database Caching
define stream CleansedOrderStream
(custId string, item string, amount int);
@store(type=‘rdbms’, …, @cache(cache.policy=‘LRU’, … ))
@primaryKey(‘name’)
@index(‘unitPrice’)
define table ItemPriceTable(name string, unitPrice double);
from CleansedOrderStream as O join ItemPriceTable as T
on O.item == T.name
select O.custId, O.item, O.amount * T.unitPrice as price
insert into EnrichedOrderStream;
Table with Cache
Join Query
Full size Image Area with text
Service Integration and Error Handling
HTTP gRPC
●
●
Service Integration and Error Handling
200
4**
Handle response based on
status code
SQL for HTTP Service Integration
@sink(type='http-call', publisher.url="http://mystore.com/discount",
sink.id="discount", @map(type='json'))
define stream EnrichedOrderStream (custId string, item string, price double);
@source(type='http-call-response', http.status.code="200",
sink.id="discount", @map(type='json',
@attributes(custId ="trp:custId", ..., price="$.discountedPrice")))
define stream DiscountedOrderStream (custId string, item string, price double);
Call service
Consume Response
Error Handling Options
● Log and drop
● Wait and back pressure
● Divert events to another stream
Events Diverted Into Error Stream
@onError(action='stream')
@sink(type='http', publisher.url = 'http://localhost:8080/logger',
on.error='stream', @map(type = 'json'))
define stream DiscountedOrderStream (custId string, item string, price double);
from !DiscountedOrderStream
select custId, item, price, _error
insert into FailedEventsTable;
Diverting connection failure
events into table.
Full size Image Area with text
Data Summarization
Data Summarization
●
○
○
○
●
○
○
●
●
●
●
●
●
●
●
●
Summarizing Data Over Shorter Period Of Time
define stream DiscountedOrderStream (custId string, item string, price double);
from DiscountedOrderStream#window.time(10 min)
select custId, sum(price) as totalPrice
group by custId
insert into AlertStream;
Window query
with aggregation and
rate limiting
Aggregation Over Multiple Time Granularities
𝝀
●
●
define aggregation OrderAggregation
from OrderStream
select custId, itemId, sum(price) as total, avg(price) as avgPrice
group by custId, itemId
aggregate every sec ... year;
Query
Data Retrieval from Aggregations
from OrderAggregation
within "2019-10-06 00:00:00",
"2019-10-30 00:00:00"
per "days"
select total as orders;
Full size Image Area with text
Rule Processing
Rule Processing
● single event
○
● collection of events
○
○
● event occurrence order
○
○
○
Alert Based On Event Occurrence Order
define stream OrderStream (custId string, orderId string, ...);
define stream PaymentStream (orderId string, ...);
from every (e1=OrderStream) ->
not PaymentStream[e1.orderId==orderId] for 15 min
select e1.custId, e1.orderId, ...
insert into PaymentDelayedStream;
Non occurrence of event
Full size Image Area with text
Serving Online and Predefined
ML Models
Serving Online and Predefined ML Models
●
○
●
○
○
●
○
○
○ from OrderStream
#pmml:predict(“/home/user/ml.model”,custId, itemId)
insert into RecommendationStream;
Find recommendations
Full size Image Area with text
Scatter-gather and Data Pipelining
Scatter-gather and Data Pipelining
{x,x,x} {x},{x},{x} {y},{y},{y} {y,y,y}
●
●
●
Modularization
Siddhi Runtime
Siddhi App
for data capture
and preprocessing
Siddhi Apps
for each use case
Siddhi App
for common data
publishing logic
Periodically Trigger Events
●
●
●
define trigger FiveMinTrigger at every 5 min;
define trigger WorkStartTrigger at '0 15 10 ? * MON-FRI';
define trigger InitTrigger at 'start';
Full size Image Area with text
Realtime Decisions As A Service
Realtime Decisions As A Service
Data Stores REST APIs
●
●
●
HTTP and gRPC REST APIs
●
●
Full size Image Area with text
Kubernetes Deployment
Steps On Deploying Siddhi App In Kubernetes
Kubernetes Deployment
Kubernetes Deployment
$ kubectl get siddhi
NAME STATUS READY AGE
Sample-app Running 2/2 5m
What We Looked At
●
●
●
●
●
https://siddhi.io
suho@wso2.com
A head start on cloud native event driven applications - bigdatadays
1 of 71

Recommended

Stream Processing with Ballerina by
Stream Processing with BallerinaStream Processing with Ballerina
Stream Processing with BallerinaSriskandarajah Suhothayan
164 views23 slides
Siddhi - cloud-native stream processor by
Siddhi - cloud-native stream processorSiddhi - cloud-native stream processor
Siddhi - cloud-native stream processorSriskandarajah Suhothayan
161 views63 slides
The Rise of Streaming SQL by
The Rise of Streaming SQLThe Rise of Streaming SQL
The Rise of Streaming SQLSriskandarajah Suhothayan
135 views31 slides
WSO2 Stream Processor: Graphical Editor, HTTP & Message Trace Analytics and m... by
WSO2 Stream Processor: Graphical Editor, HTTP & Message Trace Analytics and m...WSO2 Stream Processor: Graphical Editor, HTTP & Message Trace Analytics and m...
WSO2 Stream Processor: Graphical Editor, HTTP & Message Trace Analytics and m...Sriskandarajah Suhothayan
403 views38 slides
Make it fast for everyone - performance and middleware design by
Make it fast for everyone - performance and middleware designMake it fast for everyone - performance and middleware design
Make it fast for everyone - performance and middleware designSriskandarajah Suhothayan
3.1K views27 slides
WSO2 Analytics Platform: The one stop shop for all your data needs by
WSO2 Analytics Platform: The one stop shop for all your data needsWSO2 Analytics Platform: The one stop shop for all your data needs
WSO2 Analytics Platform: The one stop shop for all your data needsSriskandarajah Suhothayan
755 views60 slides

More Related Content

What's hot

Druid by
DruidDruid
DruidDori Waldman
1.6K views45 slides
Андрей Козлов (Altoros): Оптимизация производительности Cassandra by
Андрей Козлов (Altoros): Оптимизация производительности CassandraАндрей Козлов (Altoros): Оптимизация производительности Cassandra
Андрей Козлов (Altoros): Оптимизация производительности CassandraOlga Lavrentieva
5.3K views24 slides
Google Cloud Spanner Preview by
Google Cloud Spanner PreviewGoogle Cloud Spanner Preview
Google Cloud Spanner PreviewDoiT International
4.9K views22 slides
Webinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries by
Webinar: MongoDB Use Cases within the Oil, Gas, and Energy IndustriesWebinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries
Webinar: MongoDB Use Cases within the Oil, Gas, and Energy IndustriesMongoDB
6K views37 slides
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management by
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor ManagementMongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor ManagementMongoDB
35.7K views44 slides
Using MongoDB As a Tick Database by
Using MongoDB As a Tick DatabaseUsing MongoDB As a Tick Database
Using MongoDB As a Tick DatabaseMongoDB
19.3K views34 slides

What's hot(20)

Андрей Козлов (Altoros): Оптимизация производительности Cassandra by Olga Lavrentieva
Андрей Козлов (Altoros): Оптимизация производительности CassandraАндрей Козлов (Altoros): Оптимизация производительности Cassandra
Андрей Козлов (Altoros): Оптимизация производительности Cassandra
Olga Lavrentieva5.3K views
Webinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries by MongoDB
Webinar: MongoDB Use Cases within the Oil, Gas, and Energy IndustriesWebinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries
Webinar: MongoDB Use Cases within the Oil, Gas, and Energy Industries
MongoDB6K views
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management by MongoDB
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor ManagementMongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
MongoDB35.7K views
Using MongoDB As a Tick Database by MongoDB
Using MongoDB As a Tick DatabaseUsing MongoDB As a Tick Database
Using MongoDB As a Tick Database
MongoDB19.3K views
MongoDB Tick Data Presentation by MongoDB
MongoDB Tick Data PresentationMongoDB Tick Data Presentation
MongoDB Tick Data Presentation
MongoDB5.9K views
Cassandra as event sourced journal for big data analytics by Anirvan Chakraborty
Cassandra as event sourced journal for big data analyticsCassandra as event sourced journal for big data analytics
Cassandra as event sourced journal for big data analytics
Anirvan Chakraborty1.2K views
Joins and Other MongoDB 3.2 Aggregation Enhancements by Andrew Morgan
Joins and Other MongoDB 3.2 Aggregation EnhancementsJoins and Other MongoDB 3.2 Aggregation Enhancements
Joins and Other MongoDB 3.2 Aggregation Enhancements
Andrew Morgan10.4K views
Implementing and Visualizing Clickstream data with MongoDB by MongoDB
Implementing and Visualizing Clickstream data with MongoDBImplementing and Visualizing Clickstream data with MongoDB
Implementing and Visualizing Clickstream data with MongoDB
MongoDB4.7K views
Aggregated queries with Druid on terrabytes and petabytes of data by Rostislav Pashuto
Aggregated queries with Druid on terrabytes and petabytes of dataAggregated queries with Druid on terrabytes and petabytes of data
Aggregated queries with Druid on terrabytes and petabytes of data
Rostislav Pashuto9.7K views
codecentric AG: CQRS and Event Sourcing Applications with Cassandra by DataStax Academy
codecentric AG: CQRS and Event Sourcing Applications with Cassandracodecentric AG: CQRS and Event Sourcing Applications with Cassandra
codecentric AG: CQRS and Event Sourcing Applications with Cassandra
DataStax Academy14.8K views
Streaming Operational Data with MariaDB MaxScale by MariaDB plc
Streaming Operational Data with MariaDB MaxScaleStreaming Operational Data with MariaDB MaxScale
Streaming Operational Data with MariaDB MaxScale
MariaDB plc1.2K views
AWS Big Data Demystified #4 data governance demystified [security, networ... by Omid Vahdaty
AWS Big Data Demystified #4   data governance demystified   [security, networ...AWS Big Data Demystified #4   data governance demystified   [security, networ...
AWS Big Data Demystified #4 data governance demystified [security, networ...
Omid Vahdaty642 views
MongoDB and the Internet of Things by MongoDB
MongoDB and the Internet of ThingsMongoDB and the Internet of Things
MongoDB and the Internet of Things
MongoDB4.6K views
Cassandra at Finn.io — May 30th 2013 by DataStax Academy
Cassandra at Finn.io — May 30th 2013Cassandra at Finn.io — May 30th 2013
Cassandra at Finn.io — May 30th 2013
DataStax Academy780 views
Story of migrating event pipeline from batch to streaming by lohitvijayarenu
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaming
lohitvijayarenu82 views
MongoDB for Time Series Data by MongoDB
MongoDB for Time Series DataMongoDB for Time Series Data
MongoDB for Time Series Data
MongoDB20K views
Gregorry Letribot - Druid at Criteo - NoSQL matters 2015 by NoSQLmatters
Gregorry Letribot - Druid at Criteo - NoSQL matters 2015Gregorry Letribot - Druid at Criteo - NoSQL matters 2015
Gregorry Letribot - Druid at Criteo - NoSQL matters 2015
NoSQLmatters2.4K views

Similar to A head start on cloud native event driven applications - bigdatadays

[WSO2Con EU 2018] Patterns for Building Streaming Apps by
[WSO2Con EU 2018] Patterns for Building Streaming Apps[WSO2Con EU 2018] Patterns for Building Streaming Apps
[WSO2Con EU 2018] Patterns for Building Streaming AppsWSO2
620 views63 slides
[WSO2Con Asia 2018] Patterns for Building Streaming Apps by
[WSO2Con Asia 2018] Patterns for Building Streaming Apps[WSO2Con Asia 2018] Patterns for Building Streaming Apps
[WSO2Con Asia 2018] Patterns for Building Streaming AppsWSO2
224 views54 slides
Scaling Experimentation & Data Capture at Grab by
Scaling Experimentation & Data Capture at GrabScaling Experimentation & Data Capture at Grab
Scaling Experimentation & Data Capture at GrabRoman
458 views36 slides
Streaming Solr - Activate 2018 talk by
Streaming Solr - Activate 2018 talkStreaming Solr - Activate 2018 talk
Streaming Solr - Activate 2018 talkAmrit Sarkar
155 views27 slides
Building Analytics Applications with Streaming Expressions in Apache Solr - A... by
Building Analytics Applications with Streaming Expressions in Apache Solr - A...Building Analytics Applications with Streaming Expressions in Apache Solr - A...
Building Analytics Applications with Streaming Expressions in Apache Solr - A...Lucidworks
715 views27 slides
A federated information infrastructure that works by
A federated information infrastructure that works A federated information infrastructure that works
A federated information infrastructure that works Stratebi
575 views34 slides

Similar to A head start on cloud native event driven applications - bigdatadays(20)

[WSO2Con EU 2018] Patterns for Building Streaming Apps by WSO2
[WSO2Con EU 2018] Patterns for Building Streaming Apps[WSO2Con EU 2018] Patterns for Building Streaming Apps
[WSO2Con EU 2018] Patterns for Building Streaming Apps
WSO2620 views
[WSO2Con Asia 2018] Patterns for Building Streaming Apps by WSO2
[WSO2Con Asia 2018] Patterns for Building Streaming Apps[WSO2Con Asia 2018] Patterns for Building Streaming Apps
[WSO2Con Asia 2018] Patterns for Building Streaming Apps
WSO2224 views
Scaling Experimentation & Data Capture at Grab by Roman
Scaling Experimentation & Data Capture at GrabScaling Experimentation & Data Capture at Grab
Scaling Experimentation & Data Capture at Grab
Roman 458 views
Streaming Solr - Activate 2018 talk by Amrit Sarkar
Streaming Solr - Activate 2018 talkStreaming Solr - Activate 2018 talk
Streaming Solr - Activate 2018 talk
Amrit Sarkar155 views
Building Analytics Applications with Streaming Expressions in Apache Solr - A... by Lucidworks
Building Analytics Applications with Streaming Expressions in Apache Solr - A...Building Analytics Applications with Streaming Expressions in Apache Solr - A...
Building Analytics Applications with Streaming Expressions in Apache Solr - A...
Lucidworks715 views
A federated information infrastructure that works by Stratebi
A federated information infrastructure that works A federated information infrastructure that works
A federated information infrastructure that works
Stratebi575 views
Big Query - Women Techmarkers (Ukraine - March 2014) by Ido Green
Big Query - Women Techmarkers (Ukraine - March 2014)Big Query - Women Techmarkers (Ukraine - March 2014)
Big Query - Women Techmarkers (Ukraine - March 2014)
Ido Green757 views
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da... by WSO2
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2Con USA 2015: WSO2 Analytics Platform - The One Stop Shop for All Your Da...
WSO2641 views
[WSO2Con USA 2018] Patterns for Building Streaming Apps by WSO2
[WSO2Con USA 2018] Patterns for Building Streaming Apps[WSO2Con USA 2018] Patterns for Building Streaming Apps
[WSO2Con USA 2018] Patterns for Building Streaming Apps
WSO2240 views
GraphQL Summit 2019 - Configuration Driven Data as a Service Gateway with Gra... by Noriaki Tatsumi
GraphQL Summit 2019 - Configuration Driven Data as a Service Gateway with Gra...GraphQL Summit 2019 - Configuration Driven Data as a Service Gateway with Gra...
GraphQL Summit 2019 - Configuration Driven Data as a Service Gateway with Gra...
Noriaki Tatsumi539 views
[WSO2Con EU 2017] Streaming Analytics Patterns for Your Digital Enterprise by WSO2
[WSO2Con EU 2017] Streaming Analytics Patterns for Your Digital Enterprise[WSO2Con EU 2017] Streaming Analytics Patterns for Your Digital Enterprise
[WSO2Con EU 2017] Streaming Analytics Patterns for Your Digital Enterprise
WSO2466 views
Account Performance and Optimization by marcwan
Account Performance and OptimizationAccount Performance and Optimization
Account Performance and Optimization
marcwan1.5K views
How to separate frontend from a highload python project with no problems - Py... by Oleksandr Tarasenko
How to separate frontend from a highload python project with no problems - Py...How to separate frontend from a highload python project with no problems - Py...
How to separate frontend from a highload python project with no problems - Py...
ML Infra for Netflix Recommendations - AI NEXTCon talk by Faisal Siddiqi
ML Infra for Netflix Recommendations - AI NEXTCon talkML Infra for Netflix Recommendations - AI NEXTCon talk
ML Infra for Netflix Recommendations - AI NEXTCon talk
Faisal Siddiqi13.5K views
Netflix Machine Learning Infra for Recommendations - 2018 by Karthik Murugesan
Netflix Machine Learning Infra for Recommendations - 2018Netflix Machine Learning Infra for Recommendations - 2018
Netflix Machine Learning Infra for Recommendations - 2018
Karthik Murugesan391 views

Recently uploaded

Business Analyst Series 2023 - Week 3 Session 5 by
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5DianaGray10
237 views20 slides
Top 10 Strategic Technologies in 2024: AI and Automation by
Top 10 Strategic Technologies in 2024: AI and AutomationTop 10 Strategic Technologies in 2024: AI and Automation
Top 10 Strategic Technologies in 2024: AI and AutomationAutomationEdge Technologies
18 views14 slides
Attacking IoT Devices from a Web Perspective - Linux Day by
Attacking IoT Devices from a Web Perspective - Linux Day Attacking IoT Devices from a Web Perspective - Linux Day
Attacking IoT Devices from a Web Perspective - Linux Day Simone Onofri
15 views68 slides
PharoJS - Zürich Smalltalk Group Meetup November 2023 by
PharoJS - Zürich Smalltalk Group Meetup November 2023PharoJS - Zürich Smalltalk Group Meetup November 2023
PharoJS - Zürich Smalltalk Group Meetup November 2023Noury Bouraqadi
126 views17 slides
Lilypad @ Labweek, Istanbul, 2023.pdf by
Lilypad @ Labweek, Istanbul, 2023.pdfLilypad @ Labweek, Istanbul, 2023.pdf
Lilypad @ Labweek, Istanbul, 2023.pdfAlly339821
9 views45 slides
Microsoft Power Platform.pptx by
Microsoft Power Platform.pptxMicrosoft Power Platform.pptx
Microsoft Power Platform.pptxUni Systems S.M.S.A.
52 views38 slides

Recently uploaded(20)

Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10237 views
Attacking IoT Devices from a Web Perspective - Linux Day by Simone Onofri
Attacking IoT Devices from a Web Perspective - Linux Day Attacking IoT Devices from a Web Perspective - Linux Day
Attacking IoT Devices from a Web Perspective - Linux Day
Simone Onofri15 views
PharoJS - Zürich Smalltalk Group Meetup November 2023 by Noury Bouraqadi
PharoJS - Zürich Smalltalk Group Meetup November 2023PharoJS - Zürich Smalltalk Group Meetup November 2023
PharoJS - Zürich Smalltalk Group Meetup November 2023
Noury Bouraqadi126 views
Lilypad @ Labweek, Istanbul, 2023.pdf by Ally339821
Lilypad @ Labweek, Istanbul, 2023.pdfLilypad @ Labweek, Istanbul, 2023.pdf
Lilypad @ Labweek, Istanbul, 2023.pdf
Ally3398219 views
AMAZON PRODUCT RESEARCH.pdf by JerikkLaureta
AMAZON PRODUCT RESEARCH.pdfAMAZON PRODUCT RESEARCH.pdf
AMAZON PRODUCT RESEARCH.pdf
JerikkLaureta19 views
From chaos to control: Managing migrations and Microsoft 365 with ShareGate! by sammart93
From chaos to control: Managing migrations and Microsoft 365 with ShareGate!From chaos to control: Managing migrations and Microsoft 365 with ShareGate!
From chaos to control: Managing migrations and Microsoft 365 with ShareGate!
sammart939 views
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software257 views
Web Dev - 1 PPT.pdf by gdsczhcet
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet60 views
Automating a World-Class Technology Conference; Behind the Scenes of CiscoLive by Network Automation Forum
Automating a World-Class Technology Conference; Behind the Scenes of CiscoLiveAutomating a World-Class Technology Conference; Behind the Scenes of CiscoLive
Automating a World-Class Technology Conference; Behind the Scenes of CiscoLive
handbook for web 3 adoption.pdf by Liveplex
handbook for web 3 adoption.pdfhandbook for web 3 adoption.pdf
handbook for web 3 adoption.pdf
Liveplex22 views

A head start on cloud native event driven applications - bigdatadays

  • 1. A Head Start on Cloud-native Event Driven Applications S. Suhothayan (Suho) Director WSO2 @suhothayan
  • 3. Increasing demand is causing disaggregation
  • 4. Characteristics of Cloud Nativeness ● ● ● ● Other Key Characteristics ● ● ● Source: https://medium.com/walmartlabs/cloud-native-application-architecture-a84ddf378f82
  • 5. Nation of Event Driven Applications ● ● ● ● queues topics ● ● ● ●
  • 6. Stateless Event Driven Applications ● ● ● Highly scalable! with HTTP/TCP with JMS/Kafka/NATS
  • 7. Full size Image Area with text Not all apps are stateless!
  • 8. Some Stateful Use Cases! ● ● ● ● ●
  • 9. Building Stateful Applications remember previous events ● preserved during system failure ● shared with new nodes ● databases ● distributed cache ● in-memory
  • 10. Best Approach for Stateful Apps High Performance Scalability Fault Tolerant
  • 11. Full size Image Area with text There is no single tool for all cases!
  • 12. Solution to High Performance, Scalable, and Fault Tolerant Apps ● Keep data in memory. ● periodic state snapshots ● Use cache to preload data.
  • 13. Scalability mapreduce Deployment of Scalable Stateful Apps
  • 20. Full size Image Area with text How to implement all this?
  • 21. Streaming and Event Processing Frameworks
  • 22. Characteristics Of Typical Big Data Frameworks ● ● ● Issues: ● ● ●
  • 23. Introducing A Cloud Native Stream Processor ● ● no commercial features ● Docker and Kubernetes ● CI/CD ● SQL like query language graphical tool ●
  • 24. Image Area Working with Siddhi ● Siddhi Editor. ● Siddhi Test Framework ● ○ Java/Python ○ bare metal/VM ○ Docker ○ Kubernetes
  • 25. We are happy to announce Siddhi 5.1 ● ● ● ● ●
  • 26. Streaming SQL @app:name('Alert-Processor') @source(type='kafka', ..., @map(type='json')) define stream TemperatureStream (roomNo string, temp double); @info(name='AlertQuery') from TemperatureStream#window.time(5 min) select roomNo, avg(temp) as avgTemp group by roomNo insert into AvgTemperatureStream; Source/Sink & Streams Window Query with Rate Limiting
  • 30. Full size Image Area with text Patterns For Event Driven Data Processing
  • 31. Patterns For Event Driven Data Processing
  • 32. Image Area Scenario: Order Processing Tasks: ● ● ● ● ●
  • 33. Full size Image Area with text Consume and Publish Events With Various Data Formats
  • 34. Consume and Publish Events With Various Data Formats transports ● ● ● ● ● data formats ●
  • 35. Consume and Publish Events With Various Data Formats @source(type = mqtt, …, @map(type = json)) define stream OrderStream(custId string, item string, amount int); @source(type = mqtt, …, @map(type = json, @attribute(“$.id”,"$.itm” “$.count”))) define stream OrderStream(custId string, item string, amount int); {“event”:{“custId”:“15”,“item”:“shirt”,“amount”:2}} {“id”:“15”,“itm”:“shirt”,“count”:2}
  • 36. Full size Image Area with text Data Filtering and Preprocessing
  • 37. Data Filtering and Preprocessing ● ● ● ● ● ● define stream OrderStream (custId string, item string, amount int); from OrderStream[item!=“unknown”] select default(custId, “internal”) as custId, item, ifThenElse(amount<0, 0, amount) as amount, insert into CleansedOrderStream;
  • 38. Full size Image Area with text Date Transformation
  • 39. Date Transformation ● ● ● ● 60+ extensions ● json:getDouble(json,"$.amount") as amount str:concat(‘Hello ’,name) as greeting amount * price as cost time:extract('DAY', datetime) as day myFunction(item, price) as discount
  • 40. Full size Image Area with text Database Integration and Caching
  • 41. Database Integration and Caching ● ●
  • 42. In-memory Table define stream CleansedOrderStream (custId string, item string, amount int); @primaryKey(‘name’) @index(‘unitPrice’) define table ItemPriceTable (name string, unitPrice double); from CleansedOrderStream as O join ItemPriceTable as T on O.item == T.name select O.custId, O.item, O.amount * T.unitPrice as price insert into EnrichedOrderStream; In-memory Table Join Query
  • 43. Database Integration define stream CleansedOrderStream (custId string, item string, amount int); @store(type=‘rdbms’, …,) @primaryKey(‘name’) @index(‘unitPrice’) define table ItemPriceTable(name string, unitPrice double); from CleansedOrderStream as O join ItemPriceTable as T on O.item == T.name select O.custId, O.item, O.amount * T.unitPrice as price insert into EnrichedOrderStream; Table backed with DB Join Query
  • 44. Database Caching define stream CleansedOrderStream (custId string, item string, amount int); @store(type=‘rdbms’, …, @cache(cache.policy=‘LRU’, … )) @primaryKey(‘name’) @index(‘unitPrice’) define table ItemPriceTable(name string, unitPrice double); from CleansedOrderStream as O join ItemPriceTable as T on O.item == T.name select O.custId, O.item, O.amount * T.unitPrice as price insert into EnrichedOrderStream; Table with Cache Join Query
  • 45. Full size Image Area with text Service Integration and Error Handling
  • 46. HTTP gRPC ● ● Service Integration and Error Handling 200 4** Handle response based on status code
  • 47. SQL for HTTP Service Integration @sink(type='http-call', publisher.url="http://mystore.com/discount", sink.id="discount", @map(type='json')) define stream EnrichedOrderStream (custId string, item string, price double); @source(type='http-call-response', http.status.code="200", sink.id="discount", @map(type='json', @attributes(custId ="trp:custId", ..., price="$.discountedPrice"))) define stream DiscountedOrderStream (custId string, item string, price double); Call service Consume Response
  • 48. Error Handling Options ● Log and drop ● Wait and back pressure ● Divert events to another stream
  • 49. Events Diverted Into Error Stream @onError(action='stream') @sink(type='http', publisher.url = 'http://localhost:8080/logger', on.error='stream', @map(type = 'json')) define stream DiscountedOrderStream (custId string, item string, price double); from !DiscountedOrderStream select custId, item, price, _error insert into FailedEventsTable; Diverting connection failure events into table.
  • 50. Full size Image Area with text Data Summarization
  • 52. Summarizing Data Over Shorter Period Of Time define stream DiscountedOrderStream (custId string, item string, price double); from DiscountedOrderStream#window.time(10 min) select custId, sum(price) as totalPrice group by custId insert into AlertStream; Window query with aggregation and rate limiting
  • 53. Aggregation Over Multiple Time Granularities 𝝀 ● ● define aggregation OrderAggregation from OrderStream select custId, itemId, sum(price) as total, avg(price) as avgPrice group by custId, itemId aggregate every sec ... year; Query
  • 54. Data Retrieval from Aggregations from OrderAggregation within "2019-10-06 00:00:00", "2019-10-30 00:00:00" per "days" select total as orders;
  • 55. Full size Image Area with text Rule Processing
  • 56. Rule Processing ● single event ○ ● collection of events ○ ○ ● event occurrence order ○ ○ ○
  • 57. Alert Based On Event Occurrence Order define stream OrderStream (custId string, orderId string, ...); define stream PaymentStream (orderId string, ...); from every (e1=OrderStream) -> not PaymentStream[e1.orderId==orderId] for 15 min select e1.custId, e1.orderId, ... insert into PaymentDelayedStream; Non occurrence of event
  • 58. Full size Image Area with text Serving Online and Predefined ML Models
  • 59. Serving Online and Predefined ML Models ● ○ ● ○ ○ ● ○ ○ ○ from OrderStream #pmml:predict(“/home/user/ml.model”,custId, itemId) insert into RecommendationStream; Find recommendations
  • 60. Full size Image Area with text Scatter-gather and Data Pipelining
  • 61. Scatter-gather and Data Pipelining {x,x,x} {x},{x},{x} {y},{y},{y} {y,y,y}
  • 62. ● ● ● Modularization Siddhi Runtime Siddhi App for data capture and preprocessing Siddhi Apps for each use case Siddhi App for common data publishing logic
  • 63. Periodically Trigger Events ● ● ● define trigger FiveMinTrigger at every 5 min; define trigger WorkStartTrigger at '0 15 10 ? * MON-FRI'; define trigger InitTrigger at 'start';
  • 64. Full size Image Area with text Realtime Decisions As A Service
  • 65. Realtime Decisions As A Service Data Stores REST APIs ● ● ● HTTP and gRPC REST APIs ● ●
  • 66. Full size Image Area with text Kubernetes Deployment
  • 67. Steps On Deploying Siddhi App In Kubernetes
  • 69. Kubernetes Deployment $ kubectl get siddhi NAME STATUS READY AGE Sample-app Running 2/2 5m
  • 70. What We Looked At ● ● ● ● ● https://siddhi.io suho@wso2.com