Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Stream Processing with Ballerina

39 views

Published on

Organizations that can make sense out of massive amounts of data produced by systems, customers, or partners will have a competitive edge. Ballerina Stream Processing provides real-time event stream processing capabilities to microservices, with intuitive SQL queries allowing users to filter, aggregate and correlate data to make sense, take decisions and act in real-time in a distributed manner.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Stream Processing with Ballerina

  1. 1. Stream Processing with Ballerina S. Suhothayan Director - Engineering, WSO2
  2. 2. The Problem ○ Integration is not always about request-response. ○ Highly scalable systems use Event Driven Architecture to asynchronously communicate between multiple processing units. ○ Processing events from Webhooks, CDC, Realtime ETLs, and notification systems fall into asynchronous event driven systems.
  3. 3. What is a Stream? An unbounded continuous flow of records (having the same format) E.g., sensor events, triggers from Webhooks, messages from MQ
  4. 4. Why Stream Processing ? Doing continuous processing on the data forever ! Such as : ○ Monitor and detect anomalies ○ Real-time ETL ○ Streaming aggregations (e.g., average service response time in last 5 minutes) ○ Join/correlate multiple data streams ○ Detecting complex event patterns or trends
  5. 5. Stream Processing Constructs ○ Projection ○ Modifying the structure of the stream ○ Filter ○ Windows & Aggregations ○ Collection of streaming events over a time or length duration (last 5 min or last 50 events) ○ Viewed in a sliding or tumbling manner ○ Aggregated over window (e.g., sum, count, min, max, avg, etc) ○ Joins ○ Joining multiple streams ○ Detecting Patterns ○ Trends, non-occurrence of events
  6. 6. How to write Stream Processing Logic? Use language libraries : ○ Have different functions for each stream processor construct. ○ Pros: You can use the same language for implementation. ○ Cons: Quickly becomes very complex and messy. User SQL dialog : ○ Use easy-to-use SQL to script the logic ○ Pros: Compact and easy to write the logic. ○ Cons: Need to write UDFs, which SQL does not support.
  7. 7. Solution for Programing Streaming Efficiently Merging SQL and native programing 1. Consuming events to Ballerina using standard language constructs ○ Via HTTP, HTTP2, WebSocket, JMS and more. 2. Generate streams out of consumed data ○ Map JSON/XML/text messages into a record. 3. Define SQL to manipulate and process data in real time ○ If needed, use Ballerina functions within SQL 4. Generate output streams 5. Use standard language constructs to handle the output or send to an endpoint
  8. 8. “ Having lots of sensors, among all valid sensors, detect the sensors that have sent sensor readings greater than 100 in total within the last minute. A Use Case
  9. 9. Let’s see some beautiful code!
  10. 10. Ballerina Stream Processing
  11. 11. Ballerina Stream Processing type Alert record { string name; int total; }; type SensorData record { string name; int reading; }; Define input and output record types
  12. 12. Ballerina Stream Processing type Alert record { string name; int total; }; type SensorData record { string name; int reading; }; function alertQuery( stream<SensorData> sensorDataStream, stream<Alert> alertStream) { } Define input and output record types Function with input/output Streams
  13. 13. Ballerina Stream Processing type Alert record { string name; int total; }; type SensorData record { string name; int reading; }; function alertQuery( stream<SensorData> sensorDataStream, stream<Alert> alertStream) { forever { } } Define input and output record types Function with input/output Streams Forever block
  14. 14. Ballerina Stream Processing type Alert record { string name; int total; }; type SensorData record { string name; int reading; }; function alertQuery( stream<SensorData> sensorDataStream, stream<Alert> alertStream) { forever { from sensorDataStream where reading > 0 window time(60000) select name, sum(reading) as total group by name having total > 100 } } Define input and output record types Function with input/output Streams Forever block Among all valid sensors, select ones having greater than 100 reading in total within the last minute
  15. 15. Ballerina Stream Processing type Alert record { string name; int total; }; type SensorData record { string name; int reading; }; function alertQuery( stream<SensorData> sensorDataStream, stream<Alert> alertStream) { forever { from sensorDataStream where reading > 0 window time(60000) select name, sum(reading) as total group by name having total > 100 => (Alert[] alerts) { alertStream.publish(alerts); } } } Define input and output record types Function with input/output Streams Forever block Among all valid sensors, select ones having greater than 100 reading in total within the last minute Send Alert
  16. 16. Some more queries
  17. 17. Joining Two Streams Over Time // Detect raw material input falls below 5% of the rate of production consumption forever { from productionInputStream window time(10000) as p join rawMaterialStream window time(10000) as r on r.name == p.name select r.name, sum(r.amount) as totalRawMaterial, sum(p.amount) as totalConsumed group by r.name having ((totalRawMaterial - totalConsumed) * 100.0 / totalRawMaterial) < 5 => (MaterialUsage[] materialUsages) { materialUsageStream.publish(materialUsages); } }
  18. 18. Detecting Patterns Within Streams // Detect small purchase transaction followed by a huge purchase transaction // from the same card within a day forever { from every PurchaseStream where price < 20 as e1 followed by PurchaseStream where price > 200 && e1.id == id as e2 within 1 day select e1.id as cardId, e1.price as initialPayment, e2.price as finalPayment => (Alert[] alerts) { alertStream.publish(alerts); } }
  19. 19. Building Autonomous Services ○ Process incoming messages or locally produced events ○ Process events at the receiving node without sending to centralised system ○ Services can monitor themselves throw inbuilt matric streams producing events locally ○ Do local optimizations and take actions autonomously
  20. 20. Stream Processing at the Edge ○ Support microservices architecture ○ Summarize data at the edge. ○ When possible, take localized decisions. ○ Reduce the amount of data transferred to the central node. ○ Ability to run independently ○ Highly scalable
  21. 21. The Roadmap ○ Support stream processing to incorporate Ballerina’s custom functions. ○ Building Ballerina Stream Processing using Ballerina. ○ Support streams joining with tables. ○ Improve query language. ○ Support State Recovery. ○ Support High Availability.
  22. 22. Q & A
  23. 23. THANK YOU
  24. 24. Ballerina Stream Processing type Alert record { string name; int total; }; type SensorData record { string name; int reading; }; function alertQuery( stream<SensorData> sensorDataStream, stream<Alert> alertStream) { forever { from sensorDataStream where reading > 0 window time(60000) select name, sum(reading) as total group by name having total > 100 => (Alert[] alerts) { alertStream.publish(alerts); } } } Define input and output record types Function with input/output Streams Forever block Among all valid sensors, select ones having greater than 100 reading in total within the last minute Send Alert

×