If an Event is Published to a Topic and No One is Around to Consume it, Does it Make a Sound? (Bernd Ruecker, Camunda) Kafka Summit 2020
•
0 likes•1,281 views
For quite some time, I had a fuzzy feeling that I didn’t really understand event streaming architectures and how they fit more broadly into the modern software architecture puzzle. Then I saw a concrete, real-life example from an airplane maintenance use case, where billions of sensor data points come in via Kafka and must be transformed into insights that occasionally lead to important actions a mechanic needs to take. This story led to a personal revelation: Data-streams are passive in nature. On their own, they do not lead to any action. But at some point in time, actions must be taken. The action might be carried out by a human looking at data and reacting to it, or an external service that’s called, or a ""traditional"" database that’s updated, or a workflow that’s started. If there’s never any action, your stream is kind of useless. Now, the transition from a passive stream to an active component reacting to an event in the stream is very interesting. It raises a lot of questions about idempotency, scalability, and the capability to replay streams with changed logic. For example, in the project mentioned above, we developed our own stateful connector that starts a workflow for a mechanic only once for every new insight, but can also inform that workflow if the problem no longer exists. Replaying streams with historic data did not lead to any new workflows created. In this talk, I’ll walk you through the aircraft maintenance case study in as much detail as I can share, along with my personal discovery process, which I hope might guide you on your own streaming adventures.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to If an Event is Published to a Topic and No One is Around to Consume it, Does it Make a Sound? (Bernd Ruecker, Camunda) Kafka Summit 2020
Similar to If an Event is Published to a Topic and No One is Around to Consume it, Does it Make a Sound? (Bernd Ruecker, Camunda) Kafka Summit 2020 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
If an Event is Published to a Topic and No One is Around to Consume it, Does it Make a Sound? (Bernd Ruecker, Camunda) Kafka Summit 2020
- 1. If an event is published to a topic and no one is around to consume it, does it make a sound? Bernd Ruecker Co-Founder and Chief Technologist of Camunda @berndruecker
- 2. Warning: This talk might contain nonsense @berndruecker
- 3. People that understand functional programming The others @berndruecker
- 4. SELECT name FROM people WHERE city = `Berlin´ reader = new BufferedReader(new FileReader("people.txt")); while ((row = reader.readLine()) != null) { String[] data = row.split(","); if ("Berlin".equals(data[3])) { // city System.out.println(data[1]); // name } } reader.close(); @berndruecker
- 7. Functional programming, reactive & streaming = the greatest thing since sliced bread! … the term Event Stream becomes widely adopted and promotes a new mental model of how the systems are implemented. … the main abstraction in use is a composition of [stateless] functions performing transformation on immutable events. The output of an overall system can be modeled as the application of those composed functions on the input stream. https://medium.com/holisticon-consultants/from-stream-to-state-eb1cade5d403 @berndruecker
- 8. Functions Immutable Events Event Streams State Input / Output …and actions in the real world… @berndruecker
- 9. Connor Riley #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. @berndruecker
- 10. #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. Event goes into the stream Stateless Stream Processor Filter all tweets with #cisco Stateless Stream Processor Find offensive tweets Tim Levad (Cisco employee) Who is the hiring manager? I’m sure they would love to know that you will hate the work. We here at Cisco are versed in the Web. Action Input Output → Job offer revoked @berndruecker
- 13. #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. Event goes into the stream Stateless Stream Processor Filter all tweets with #cisco Refine / change logic Find offensive tweets Tim Levad (Cisco employee) Who is the hiring manager? I’m sure they would love to know that you will hate the work. We here at Cisco are versed in the Web. Action Input Output @berndruecker
- 14. #Cisco just offered me a job! Connor Riley #Cisco just offered me a job! Now I have to weigh the utility of a fatty paycheck against the daily commute to San Jose and hating the work. Event goes into the stream Stateless Stream Processor Filter all tweets with #cisco Find offensive tweets Tim Levad (Cisco employee) Who is the hiring manager? I’m sure they would love to know that you will hate the work. We here at Cisco are versed in the Web. Action Input Output @berndruecker
- 16. #Cisco just offered me a job! Stateless Stream Processor Filter all tweets with #cisco tweets Filter tweets with similar content within 15 minutes time Stateful Stream Processor State is in the API! @berndruecker
- 17. #Cisco just offered me a job! Stateless Stream Processor Filter all tweets with #cisco tweets Filter tweets from our top community members Stateful Component (as stateless stream processor) State is in the component @berndruecker
- 18. #Cisco just offered me a job! tweets Enrich tweet with personal information of sender lookup Bad, because - Load on API - Availability - Other disadvantages of synchronous remote communication @berndruecker
- 19. #Cisco just offered me a job! tweets Person stream Stateful Stream Processor Enrich tweet @berndruecker
- 20. #Cisco just offered me a job! tweets Stateful Stream Processor Persons that tweeted Enrich tweet Gather personal info @berndruecker
- 21. #Cisco just offered me a job! tweets Stateful Stream Processor THIS is workflow logic! Workflow engines can hold state Enrich tweet Gather personal info @berndruecker Persons that tweeted
- 22. #Cisco just offered me a job! tweets Stateful Stream Processor Workflow engines can hold state Enrich tweet Gather personal info Publish / subscribe Mitigates problems of synchronous communication! @berndruecker
- 23. Example workflowClient.newWorker() .jobType("person-lookup") .handler((jobClient, job) -> { System.out.println("Gather personal information"); jobClient.newCompleteCommand(job.getKey()).send().join(); }).open(); publish / subscribe @berndruecker client.newCreateInstanceCommand() .bpmnProcessId("enrich-tweet") .variables(...) .latestVersion() .send().join(); client.newCreateInstanceCommand() .bpmnProcessId("order-process") .latestVersion() .send() .join(); Or use the Zeebe Kafka Connector
- 24. Declarative Thinking Imperative Thinking Or is it just me? @berndruecker
- 25. Berlin, Germany http://berndruecker.io/ mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
- 26. Where do we stand now? State goes into Stream Processor API Component Workflow Engine* *helps to implement common patterns in stateful components @berndruecker
- 27. Where do we stand? State goes into Stream Processor API Component Workflow Engine This translation requires additional thought! @berndruecker
- 28. Let‘s do another example: Vehicle Maintenance
- 29. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Call driver to stop and inspect … @berndruecker
- 30. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor @berndruecker
- 31. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision Custom Connector with own state WorkflowsEvent Streams @berndruecker
- 32. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Custom Connector @berndruecker
- 33. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Deduplicate Messages Custom Connector @berndruecker
- 34. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Event ID: 666555444333222 Asset: Bus_4736_Motor Time: 1593611320 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Semantic Aggregation Custom Connector @berndruecker
- 35. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical_for_too_long CorrelationKey: Bus_4736_Motor_oil_pressure_... Custom Connector Time windowing is done in stream processing @berndruecker
- 36. Connecting two worlds Event ID: 555444333222111 Asset: Bus_4736_Motor Time: 1593685892 Insight: oil_pressure_critical CorrelationKey: Bus_4736_Motor_oil_pressure_critical Event ID: 777666555444333 Asset: Bus_4736_Motor Time: 1593900001 Insight: oil_pressure_normal CorrelationKey: Bus_4736_Motor_oil_pressure_normal Semantic Aggregation STARTNEW WORKFLOW STARTOR KEEP WORKFLOW CLOSE WORKFLOW Custom Connector @berndruecker
- 38. ActionInsightsMeasurements generates Leads to oil pressure is 80 psi oil pressure is critically high Schedule maintenance Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision WorkflowsEvent Streams @berndruecker Custom Connector
- 39. ActionInsightsMeasurements generates Leads to Kafka Topic Measures Kafka Topic Insights Camunda Workflow Maintenance Stateful Stream Processor Decision Custom Connector WorkflowsEvent Streams This regularly changed and the stream of events was replayed @berndruecker
- 40. Connecting two worldsCustom Connector - Deduplication - Semantic Aggregation - Start, correlate to or cancel workflow instances - Handle replays - … Streaming Workflow @berndruecker
- 42. Summary • Streams need to lead to actions outside the control of the streams architecture (otherwise nothing has really happened) • Where to handle state is a difficult question • Streams vs. Workflow might be about declarative vs. imperative thinking (what does your team understand?) • But streams and workflows also go hand in hand. Getting from stream to workflow involves a translation component @berndruecker