Workflow Engines & Event Streaming Brokers - Can they work together? [Current 2023]
Report
Share
•0 likes•196 views
![Workflow Engines & Event Streaming Brokers @NSilnitsky
public class CartServiceCompositionLayer {
public static void init(String[] args) {
// Create a new worker factory
WorkerFactory factory = WorkerFactory.newInstance(... WorkflowClient ...);
// Create a new worker that listens on the specified task queue
Worker worker = factory.newWorker("MY_TASK_QUEUE");
// Register your workflow and activity implementations
worker.registerWorkflowImplementationTypes(CheckoutWorkflowImpl.class);
worker.registerActivitiesImplementations(new CartActivityImpl());
// Start listening for tasks
factory.start();
}
}](https://image.slidesharecdn.com/finalworkflowengineseventstreamingbrokers-230926150504-afd3ed7b/75/workflow-engines-event-streaming-brokers-can-they-work-together-current-2023-32-2048.jpg?cb=1695741162)
Workflow Engines & Event Streaming Brokers - Can they work together? [Current 2023]
•0 likes•196 views
Report
Share
Download to read offline
Workflow engines and event streaming brokers offer very different solutions to the same requirement - an optimal implementation of microservices communication. At Wix, we have a good experience with event-driven architecture for our 2500 microservices using Apache Kafka. Apache Kafka provides: * support for very high throughput * Fault tolerance * very loose coupling * Huge connectors eco-system Temporal workflow orchestration has interesting features: * Support for long running tasks * business flows visual tracking * Easy to follow imperative style programming In this talk we will learn about the tradeoffs between the two technologies and how to implement various use cases in each architecture, including those that need a little more work.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Workflow Engines & Event Streaming Brokers - Can they work together? [Current 2023]
Similar to Workflow Engines & Event Streaming Brokers - Can they work together? [Current 2023](20)
More from Natan Silnitsky
More from Natan Silnitsky(20)
Recently uploaded
Recently uploaded(20)
Workflow Engines & Event Streaming Brokers - Can they work together? [Current 2023]
- 1. September 2023 Workflow Engines & Event Streaming Brokers Can they work together? Natan Silnitsky, Backend Infra TL @Wix natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
- 2. Workflow Engines & Event Streaming Brokers @NSilnitsky Would you implement a complex business flow with Kafka and Event Driven Architecture? A B C D E
- 3. Workflow Engines & Event Streaming Brokers @NSilnitsky Hi, I’m Natan → → → Backend Infra Tech Lead @Wix Yoga enthusiast Speaker Blogger →
- 4. Workflow Engines & Event Streaming Brokers @NSilnitsky
- 5. Workflow Engines & Event Streaming Brokers @NSilnitsky ~1B Unique visitors use Wix platform every month ~3.5B Daily HTTP Transactions ~70B Kafka messages a day >600 GAs every day 3000 Microservices in production
- 6. Workflow Engines & Event Streaming Brokers @NSilnitsky @Wix Is great for ● Event Streaming @Scale ● Asynchronous Messaging Pub/Sub ● Decoupling & Extensibility ● Optimize Queries - Materialized views
- 7. Workflow Engines & Event Streaming Brokers @NSilnitsky Gaps @Wix ● Complex business flows - Comprehension & Changes ● Long running tasks - Internal job processing
- 8. Workflow Engines & Event Streaming Brokers @NSilnitsky Gaps @Wix
- 9. Workflow Engines & Event Streaming Brokers @NSilnitsky Battle of the Microservices Coordination Strategies
- 10. Workflow Engines & Event Streaming Brokers @NSilnitsky ● Simplifies complex orchestration ● Fault-tolerant & resilient ● Boosts developer productivity
- 11. Workflow Engines & Event Streaming Brokers @NSilnitsky What’s now Complex business flow Temporal Overview Long running tasks Integrating Workflow Engines into Wix → → → →
- 12. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow Inventory Service Payments Service Order Service Create Order Reserve Inventory process payment Cart Service
- 13. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow Inventory Service Payments Service Order Service Create Order Reserve Inventory process payment Cart Service
- 14. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow Inventory Service Payments Service Order Service Create Order Reserve Inventory process payment Cancel reservation Cart Service
- 15. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow Payments Service Order Service Create Order Reserve Inventory process payment Cancel order Cancel reservation Inventory Service Cart Service
- 16. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow
- 17. Workflow Engines & Event Streaming Brokers @NSilnitsky How would you do it with Kafka events Inventory Service Payments Service Order Service Checkout Started Order Created Inventory reserved Cart Service
- 18. Workflow Engines & Event Streaming Brokers @NSilnitsky Inventory Service Payments Service Order Service Checkout Started Order Created Inventory reserved Cart Service Reservation canceled Payment failed Order canceled How would you do it with Kafka events
- 19. Workflow Engines & Event Streaming Brokers @NSilnitsky Order Service Cart Service function checkout(): summarize cart // publish CheckoutStarted event listen to checkoutStarted event: create order publish orderCreated event listen to inventoryCancelled event: cancel order publish orderCancelled event How would you do it with Kafka / EDA
- 20. Workflow Engines & Event Streaming Brokers @NSilnitsky listen to orderCreated event: reserve inventory if reservation successful: publish inventoryReserved event else: publish inventoryCancelled event listen to paymentCancelled event: release reserved inventory listen to inventoryReserved event: process payment if payment successful: publish paymentProcessed event else: publish paymentFailed event Inventory Service Payments Service How would you do it with Kafka events * only once, changing order
- 21. Workflow Engines & Event Streaming Brokers @NSilnitsky https://cdn.confluent.io/wp-content/uploads/stream-lineage-data-flow-stock-events.png
- 22. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow
- 23. Workflow Engines & Event Streaming Brokers @NSilnitsky How would you do it with Workflow engine? Inventory Service Payments Service Order Service Cart Service function checkout(): cart = summarizeCart() order = createOrder(cart) result = reserveInventory(order) if result != "Success": cancelOrder(order) # Rollback order creation return "Inventory reservation failed" result = processPayment(order) if result != "Success": releaseInventory(order) cancelOrder(order) return "Payment processing failed" completeOrder(order) return "Order placed successfully"
- 24. Workflow Engines & Event Streaming Brokers @NSilnitsky How would you do it with Workflow engine?
- 25. Workflow Engines & Event Streaming Brokers @NSilnitsky Complex business flow Seeing the Big Picture beats* Production Monitoring and Debugging Changing the sequence of the business flow
- 26. Workflow Engines & Event Streaming Brokers @NSilnitsky What’s now Complex business flow Temporal Overview Long running tasks Integrating Workflow Engines into Wix → → → →
- 27. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Workflow Order Service Cart Service Summarize Cart Activity Checkout Workflow Order creation Activity Logical definition
- 28. Workflow Engines & Event Streaming Brokers @NSilnitsky public interface CartActivity { @ActivityMethod CartSummary summarizeCart(Cart cart); } public interface CheckoutWorkflow { @WorkflowMethod void processCheckout(Cart cart); } public interface OrderActivity { @ActivityMethod Order createOrder(CartSummary cartSummary); }
- 29. Workflow Engines & Event Streaming Brokers @NSilnitsky public class CheckoutWorkflowImpl implements CheckoutWorkflow { CartActivity cartActivity = Workflow.newActivityStub(CartActivity.class); @Override public void processCheckout(Cart cart) { // Summarize the cart CartSummary summary = cartActivity.summarizeCart(cart); // rest of the checkout process here... } }
- 30. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Workflow Temporal Server Cart Service Summarize Cart Activity Worker Checkout Workflow Worker Workers Order creation Activity Worker Order Service
- 31. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Workflow Temporal Server Cart Service Summarize Cart Activity Worker Checkout Workflow Worker Task Queues Order creation Activity Worker Order Service
- 32. Workflow Engines & Event Streaming Brokers @NSilnitsky public class CartServiceCompositionLayer { public static void init(String[] args) { // Create a new worker factory WorkerFactory factory = WorkerFactory.newInstance(... WorkflowClient ...); // Create a new worker that listens on the specified task queue Worker worker = factory.newWorker("MY_TASK_QUEUE"); // Register your workflow and activity implementations worker.registerWorkflowImplementationTypes(CheckoutWorkflowImpl.class); worker.registerActivitiesImplementations(new CartActivityImpl()); // Start listening for tasks factory.start(); } }
- 33. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Server Temporal Workflow Cart Service Summarize Cart Activity Worker Checkout Workflow Worker Retries Order creation Activity Worker Order Service * Wix infra
- 34. Workflow Engines & Event Streaming Brokers @NSilnitsky public class CheckoutWorkflowImpl implements CheckoutWorkflow { public CheckoutWorkflowImpl() { ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(10)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofSeconds(1)) .setMaximumInterval(Duration.ofSeconds(100)) .setBackoffCoefficient(2) .setMaximumAttempts(3) .build() ) .build(); CartActivity cartActivity = Workflow.newActivityStub( CartActivity.class, options); } }
- 35. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Supports Temporal Server Cart Service Summarize Cart Activity Worker Checkout Workflow Worker Major languages Order creation Activity Worker Order Service Service written with Python Service written with JS&TS * Wix infra
- 36. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Workflow Debugging
- 37. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Workflow Debugging
- 38. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal disadvantages ● workflow code has to be deterministic ● Activity Input and Output must be serializable ● No support for Very low Latency
- 39. Workflow Engines & Event Streaming Brokers @NSilnitsky public class CheckoutWorkflowImpl implements CheckoutWorkflow { public String processCheckout() { Cart cart = cartActivity.summarizeCart(); Order order = orderActivity.createOrder(cart); String result = inventoryActivity.reserveInventory(order); if (!"Success".equals(result)) { orderActivity.cancelOrder(order); return "Inventory reservation failed"; } result = paymentActivity.processPayment(order); if (!"Success".equals(result)) { inventoryActivity.releaseInventory(order); orderActivity.cancelOrder(order); return "Payment processing failed"; } orderActivity.completeOrder(order); return "Order placed successfully"; } } Non-deterministic code has to be inside activity. Parameters must be serializable!
- 40. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Server Temporal Workflow Cart Service Summarize Cart Activity Worker Checkout Workflow Worker Serializable Order creation Activity Worker Order Service Order must be serializable Cart must be serializable
- 41. Workflow Engines & Event Streaming Brokers @NSilnitsky What’s now Complex business flow Temporal Overview Long running tasks Integrating Workflow Engines into Wix → → → →
- 42. Workflow Engines & Event Streaming Brokers @NSilnitsky Long Running tasks
- 43. Workflow Engines & Event Streaming Brokers @NSilnitsky Kafka Broker renew-sub-topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Kafka Consumer Greyhound Consumer Payment Service Long Running tasks
- 44. Workflow Engines & Event Streaming Brokers @NSilnitsky Kafka Broker renew-sub-topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Kafka Consumer Greyhound Consumer Partition Revoked Message Poll Timeout! Payment Service Long Running tasks Kafka Consumer Greyhound Consumer
- 45. Workflow Engines & Event Streaming Brokers @NSilnitsky Temporal Server Cart Service Summarize Cart Activity Worker Checkout Workflow Worker Product Info Activity Worker Order Service Long Running tasks
- 46. Workflow Engines & Event Streaming Brokers @NSilnitsky public class PaymentActivityImpl implements PaymentActivity { public processPayment(order) { ... ExecutionContext executionContext = Activity.getExecutionContext; executionContext.heartbeat("waiting for bank."); ... } }
- 47. Workflow Engines & Event Streaming Brokers @NSilnitsky What’s now Complex business flow Temporal Overview Long running tasks Integrating Workflow Engines into Wix → → → →
- 48. Workflow Engines & Event Streaming Brokers @NSilnitsky Wix has built an Open Platform Wix Orders service Wix Inventory service 3rd party Checkout app 3rd party Analytics app 3rd party Tax Calculator SPI API Produce order created
- 49. Workflow Engines & Event Streaming Brokers @NSilnitsky Can Temporal do Pub/Sub messaging? Checkout Started json Cart Service Temporal Signal Checkout workflow1 Checkout workflow2 Checkout workflow3 Order Service * Not design high throu, for same exec, or for distributing work
- 50. Workflow Engines & Event Streaming Brokers @NSilnitsky public interface CheckoutWorkflow { @SignalMethod void checkoutStarted(Cart cart); } CheckoutWorkflow workflow1 = client.newWorkflowStub(...); … workflow1.checkoutStarted(...);
- 51. Workflow Engines & Event Streaming Brokers @NSilnitsky public class CheckoutWorkflowImpl implements CheckoutWorkflow { public String processCheckout() { Cart cart = cartActivity.summarizeCart(); Order order = orderActivity.createOrder(cart); String result = inventoryActivity.reserveInventory(order); if (!"Success".equals(result)) { orderActivity.cancelOrder(order); return "Inventory reservation failed"; } result = paymentActivity.processPayment(order); if (!"Success".equals(result)) { inventoryActivity.releaseInventory(order); orderActivity.cancelOrder(order); return "Payment processing failed"; } orderActivity.completeOrder(order); return "Order placed successfully"; } } Coupled orchestration
- 52. Workflow Engines & Event Streaming Brokers @NSilnitsky Where is Wix thinking of utilizing Temporal
- 53. Workflow Engines & Event Streaming Brokers @NSilnitsky Where Wix is thinking of utilizing Temporal long running processes Cron jobs Internal microservice jobs Dual use
- 54. Workflow Engines & Event Streaming Brokers @NSilnitsky Fitting Temporal in Wix- cron jobs Cron Job
- 55. Workflow Engines & Event Streaming Brokers @NSilnitsky client.newWorkflowStub( ..., WorkflowOptions.newBuilder() .setCronSchedule("* * * * *") .build()); );
- 56. Workflow Engines & Event Streaming Brokers @NSilnitsky Where Wix is thinking of utilizing Temporal long running processes Cron jobs Internal microservice jobs Dual use
- 57. Workflow Engines & Event Streaming Brokers @NSilnitsky Cart Service Order Service 3rd party HTTP WebHook Fitting Temporal in Wix Option 1 - intra service jobs gRPC HTTP
- 58. Workflow Engines & Event Streaming Brokers @NSilnitsky Option 2 - dual use Cart Service Order Service 3rd party HTTP WebHook Fitting Temporal in Wix gRPC HTTP
- 59. Workflow Engines & Event Streaming Brokers @NSilnitsky Low Latency Long Running Tasks vs Customized Decoupled Logic Standalone Business Processes Summary - Microservices Coordination
- 60. Workflow Engines & Event Streaming Brokers @NSilnitsky Summary - Wix plans Wix has a rich and diverse distributed system and open platform mindset Kafka and Wix Infra allow us to have flexibility, extensibility and resiliency built into the system Wix is considering to inject Temporal in strategic places in order to increase dev velocity * things to consider. Some companies. Can together?
- 61. Workflow Engines & Event Streaming Brokers @NSilnitsky github.com/wix/greyhound
- 62. Workflow Engines & Event Streaming Brokers @NSilnitsky Lessons Learned From Working with 2000 Event-Driven Microservices The Next Step https://www.youtube.com/watch?v= H4OSmOYTCkM Scan the code
- 63. Thank You! September 2023 natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky