Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A head start on cloud native event driven applications - bigdatadays

135 views

Published on

Characteristics of cloud native apps.
Problems in implementing event-driven stateful applications.
Siddhi: Cloud-Native Stream Processor.
Patterns of implementing event-driven applications.
Deploying event-driven applications on Kubernetes with Siddhi and NATS.

Published in: Technology
  • Be the first to comment

A head start on cloud native event driven applications - bigdatadays

  1. 1. A Head Start on Cloud-native Event Driven Applications S. Suhothayan (Suho) Director WSO2 @suhothayan
  2. 2. About Me Sri Lanka Siddhi
  3. 3. Increasing demand is causing disaggregation
  4. 4. Characteristics of Cloud Nativeness ● ● ● ● Other Key Characteristics ● ● ● Source: https://medium.com/walmartlabs/cloud-native-application-architecture-a84ddf378f82
  5. 5. Nation of Event Driven Applications ● ● ● ● queues topics ● ● ● ●
  6. 6. Stateless Event Driven Applications ● ● ● Highly scalable! with HTTP/TCP with JMS/Kafka/NATS
  7. 7. Full size Image Area with text Not all apps are stateless!
  8. 8. Some Stateful Use Cases! ● ● ● ● ●
  9. 9. Building Stateful Applications remember previous events ● preserved during system failure ● shared with new nodes ● databases ● distributed cache ● in-memory
  10. 10. Best Approach for Stateful Apps High Performance Scalability Fault Tolerant
  11. 11. Full size Image Area with text There is no single tool for all cases!
  12. 12. Solution to High Performance, Scalable, and Fault Tolerant Apps ● Keep data in memory. ● periodic state snapshots ● Use cache to preload data.
  13. 13. Scalability mapreduce Deployment of Scalable Stateful Apps
  14. 14. How Checkpointing Works?
  15. 15. How Checkpointing Works? ...
  16. 16. How Checkpointing Works? ...
  17. 17. How Checkpointing Works? ...
  18. 18. How Checkpointing Works? ...
  19. 19. How Checkpointing Works? ...
  20. 20. Full size Image Area with text How to implement all this?
  21. 21. Streaming and Event Processing Frameworks
  22. 22. Characteristics Of Typical Big Data Frameworks ● ● ● Issues: ● ● ●
  23. 23. Introducing A Cloud Native Stream Processor ● ● no commercial features ● Docker and Kubernetes ● CI/CD ● SQL like query language graphical tool ●
  24. 24. Image Area Working with Siddhi ● Siddhi Editor. ● Siddhi Test Framework ● ○ Java/Python ○ bare metal/VM ○ Docker ○ Kubernetes
  25. 25. We are happy to announce Siddhi 5.1 ● ● ● ● ●
  26. 26. Streaming SQL @app:name('Alert-Processor') @source(type='kafka', ..., @map(type='json')) define stream TemperatureStream (roomNo string, temp double); @info(name='AlertQuery') from TemperatureStream#window.time(5 min) select roomNo, avg(temp) as avgTemp group by roomNo insert into AvgTemperatureStream; Source/Sink & Streams Window Query with Rate Limiting
  27. 27. Web Based Source Editor
  28. 28. Web Based Graphical Editor
  29. 29. CI/CD Pipeline Reference of Siddhi
  30. 30. Full size Image Area with text Patterns For Event Driven Data Processing
  31. 31. Patterns For Event Driven Data Processing
  32. 32. Image Area Scenario: Order Processing Tasks: ● ● ● ● ●
  33. 33. Full size Image Area with text Consume and Publish Events With Various Data Formats
  34. 34. Consume and Publish Events With Various Data Formats transports ● ● ● ● ● data formats ●
  35. 35. Consume and Publish Events With Various Data Formats @source(type = mqtt, …, @map(type = json)) define stream OrderStream(custId string, item string, amount int); @source(type = mqtt, …, @map(type = json, @attribute(“$.id”,"$.itm” “$.count”))) define stream OrderStream(custId string, item string, amount int); {“event”:{“custId”:“15”,“item”:“shirt”,“amount”:2}} {“id”:“15”,“itm”:“shirt”,“count”:2}
  36. 36. Full size Image Area with text Data Filtering and Preprocessing
  37. 37. Data Filtering and Preprocessing ● ● ● ● ● ● define stream OrderStream (custId string, item string, amount int); from OrderStream[item!=“unknown”] select default(custId, “internal”) as custId, item, ifThenElse(amount<0, 0, amount) as amount, insert into CleansedOrderStream;
  38. 38. Full size Image Area with text Date Transformation
  39. 39. Date Transformation ● ● ● ● 60+ extensions ● json:getDouble(json,"$.amount") as amount str:concat(‘Hello ’,name) as greeting amount * price as cost time:extract('DAY', datetime) as day myFunction(item, price) as discount
  40. 40. Full size Image Area with text Database Integration and Caching
  41. 41. Database Integration and Caching ● ●
  42. 42. In-memory Table define stream CleansedOrderStream (custId string, item string, amount int); @primaryKey(‘name’) @index(‘unitPrice’) define table ItemPriceTable (name string, unitPrice double); from CleansedOrderStream as O join ItemPriceTable as T on O.item == T.name select O.custId, O.item, O.amount * T.unitPrice as price insert into EnrichedOrderStream; In-memory Table Join Query
  43. 43. Database Integration define stream CleansedOrderStream (custId string, item string, amount int); @store(type=‘rdbms’, …,) @primaryKey(‘name’) @index(‘unitPrice’) define table ItemPriceTable(name string, unitPrice double); from CleansedOrderStream as O join ItemPriceTable as T on O.item == T.name select O.custId, O.item, O.amount * T.unitPrice as price insert into EnrichedOrderStream; Table backed with DB Join Query
  44. 44. Database Caching define stream CleansedOrderStream (custId string, item string, amount int); @store(type=‘rdbms’, …, @cache(cache.policy=‘LRU’, … )) @primaryKey(‘name’) @index(‘unitPrice’) define table ItemPriceTable(name string, unitPrice double); from CleansedOrderStream as O join ItemPriceTable as T on O.item == T.name select O.custId, O.item, O.amount * T.unitPrice as price insert into EnrichedOrderStream; Table with Cache Join Query
  45. 45. Full size Image Area with text Service Integration and Error Handling
  46. 46. HTTP gRPC ● ● Service Integration and Error Handling 200 4** Handle response based on status code
  47. 47. SQL for HTTP Service Integration @sink(type='http-call', publisher.url="http://mystore.com/discount", sink.id="discount", @map(type='json')) define stream EnrichedOrderStream (custId string, item string, price double); @source(type='http-call-response', http.status.code="200", sink.id="discount", @map(type='json', @attributes(custId ="trp:custId", ..., price="$.discountedPrice"))) define stream DiscountedOrderStream (custId string, item string, price double); Call service Consume Response
  48. 48. Error Handling Options ● Log and drop ● Wait and back pressure ● Divert events to another stream
  49. 49. Events Diverted Into Error Stream @onError(action='stream') @sink(type='http', publisher.url = 'http://localhost:8080/logger', on.error='stream', @map(type = 'json')) define stream DiscountedOrderStream (custId string, item string, price double); from !DiscountedOrderStream select custId, item, price, _error insert into FailedEventsTable; Diverting connection failure events into table.
  50. 50. Full size Image Area with text Data Summarization
  51. 51. Data Summarization ● ○ ○ ○ ● ○ ○ ● ● ● ● ● ● ● ● ●
  52. 52. Summarizing Data Over Shorter Period Of Time define stream DiscountedOrderStream (custId string, item string, price double); from DiscountedOrderStream#window.time(10 min) select custId, sum(price) as totalPrice group by custId insert into AlertStream; Window query with aggregation and rate limiting
  53. 53. Aggregation Over Multiple Time Granularities 𝝀 ● ● define aggregation OrderAggregation from OrderStream select custId, itemId, sum(price) as total, avg(price) as avgPrice group by custId, itemId aggregate every sec ... year; Query
  54. 54. Data Retrieval from Aggregations from OrderAggregation within "2019-10-06 00:00:00", "2019-10-30 00:00:00" per "days" select total as orders;
  55. 55. Full size Image Area with text Rule Processing
  56. 56. Rule Processing ● single event ○ ● collection of events ○ ○ ● event occurrence order ○ ○ ○
  57. 57. Alert Based On Event Occurrence Order define stream OrderStream (custId string, orderId string, ...); define stream PaymentStream (orderId string, ...); from every (e1=OrderStream) -> not PaymentStream[e1.orderId==orderId] for 15 min select e1.custId, e1.orderId, ... insert into PaymentDelayedStream; Non occurrence of event
  58. 58. Full size Image Area with text Serving Online and Predefined ML Models
  59. 59. Serving Online and Predefined ML Models ● ○ ● ○ ○ ● ○ ○ ○ from OrderStream #pmml:predict(“/home/user/ml.model”,custId, itemId) insert into RecommendationStream; Find recommendations
  60. 60. Full size Image Area with text Scatter-gather and Data Pipelining
  61. 61. Scatter-gather and Data Pipelining {x,x,x} {x},{x},{x} {y},{y},{y} {y,y,y}
  62. 62. ● ● ● Modularization Siddhi Runtime Siddhi App for data capture and preprocessing Siddhi Apps for each use case Siddhi App for common data publishing logic
  63. 63. Periodically Trigger Events ● ● ● define trigger FiveMinTrigger at every 5 min; define trigger WorkStartTrigger at '0 15 10 ? * MON-FRI'; define trigger InitTrigger at 'start';
  64. 64. Full size Image Area with text Realtime Decisions As A Service
  65. 65. Realtime Decisions As A Service Data Stores REST APIs ● ● ● HTTP and gRPC REST APIs ● ●
  66. 66. Full size Image Area with text Kubernetes Deployment
  67. 67. Steps On Deploying Siddhi App In Kubernetes
  68. 68. Kubernetes Deployment
  69. 69. Kubernetes Deployment $ kubectl get siddhi NAME STATUS READY AGE Sample-app Running 2/2 5m
  70. 70. What We Looked At ● ● ● ● ● https://siddhi.io suho@wso2.com

×