DevSum - Lessons Learned from 2000 microservices

1. Lessons Learned from 2000 event-driven microservices natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil Natan Silnitsky Backend Infra TL, Wix May 2023

2. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky

3. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Unique visitors use Wix platform every month ~1B

4. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Unique visitors use Wix platform every month ~1B Daily HTTP Transactions ~500B Kafka messages a day ~70B

5. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Unique visitors use Wix platform every month ~1B Daily HTTP Transactions ~500B Kafka messages a day ~70B GAs every day > 600 Microservices in production 2500 * scale, resilience. issues

6. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Challenges of event-driven architecture, that we’ve bumped into 1 Producing message failures Processing out-of-order & duplicates 2 4 Troubleshooting production 3 Sending large payloads * success, tools, faster

7. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky How Event-driven Architecture Works

8. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Service-to-Service Communication Cart Service User Service Inventory Service Catalog Service

9. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Request-Reply Communication HTTP RPC HTTP RPC HTTP RPC Cart Service User Service Inventory Service Catalog Service * issue scale

10. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky slow Cart Service * slow, bottleneck, cache HTTP RPC HTTP RPC HTTP RPC

11. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky unreliable Cart Service * unreliable, cascade, retr HTTP RPC HTTP RPC HTTP RPC

12. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Event-driven Communication Producer Broker Product Updated Topic Event * improve, broker, scale Catalog Service Kafka Azure Service Bus Azure Event Hubs RabbitMQ

13. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Broker more robust * DB, decoupling, no impact Cart Service Producer Consumer Kafka Catalog Service Product Updated Topic

14. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Broker Event processing is guaranteed Producer Consumer Kafka Catalog Service Cart Service Product Updated Topic

15. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky The following is based on a true story *Dates and products were changed for clarity :) * ecom simple linear

16. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2016 Wix starts using event-driven We can work event-driven!!

17. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky It all began when Ecom experienced data issues Data does NOT reflect actual catalog Risk: show wrong prices in cart Cart DB

18. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2. Produce “Product Updated” Event Broker Cart Service 4. Show updated prices in cart 3. Update Product Price Catalog Service 1. Update status After investigating Cart DB

19. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Challenge #1 Producing message failure Kafka

20. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Broker Cart Service Catalog Service Make DB Update & Event Producing Atomic

21. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Produce event to S3 Broker Catalog Service Resilient Producer Catch Unsent Events

22. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Produce event to S3 Broker Produce to Kafka Healer Service Catalog Service Poll Resilient Producer Fallback to S3 and Heal

23. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Kafka Broker Service A Service B Greyhound Producer Kafka Producer Greyhound Consumer Kafka Consumer Wrap Kafka with Greyhound* * Open source: https://github.com/wix/greyhound

24. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky ➔ Resilient Producer ➔ Parallel Consumption ➔ Batch Consumer ➔ Consumer Retry Strategies ➔ Context Propagation ➔ Metrics reporting Developer Self-Service: Wrap Kafka with Greyhound* * Open source: https://github.com/wix/greyhound

25. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2016 Wix starts using event-driven 2018 Greyhound Resilient producer & Consumer retries

26. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Produce event to S3 Broker Produce to Kafka Healer Service Catalog Service Poll Resilient Producer Fallback to S3 and Heal

27. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Broker Catalog Service Healer Service Remove Discount Introduce Discount Then ‘out-of-order’ happened Cart Service

28. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Challenge #2 Out-of-order & duplicates processing Kafka

29. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Catalog Service Broker Healer Service Introduce Discount Mitigating out-of-order with revision ID # 10 # 9 Cart Service

30. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Catalog Service Broker Healer Service Remove Discount Introduce Discount Mitigating out-of-order with revision ID # 11 # 10 # 9 Cart Service * item itself

31. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Scan the binlog. For each entry produce a ‘status update’ event Cart Service Broker Catalog Service Mitigating out-of-order with Debezium connector

32. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky More Ecom data issues Data does NOT reflect actual inventory Risk: lose potential customers Inventory DB

33. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Item 2 Item 1 Broker Payments Service Investigation leads to duplicate processing Payment for: Inventory Service Retry Item 2 5 → 3 Item 1 9 → 7 * not idempotent

34. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Item 2 5 → 4 Item 1 9 → 8 Item 2 Item 1 Payment for: Broker txnId - a7g45 Mitigating duplicates with Transaction ID Payments Service Inventory Service txnId - a7g45 txnId - a7g45

35. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2016 Wix starts using event-driven 2018 Greyhound Resilient producer & Consumer retries 2019 Revisions & Transaction IDs

36. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Broker Product Catalog Service Product Update event Cart Service “Dude, I can’t produce large payloads” ... "description": "An apple mobile which is nothing like apple", ...

37. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky * 1MB Challenge #3 Failure to send large payloads Broker ... "description": "An apple mobile which is nothing like apple", ...

38. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Large Payloads Remedy I Compression → Try several compression types (lz4, snappy, etc.) → Compression on Kafka level is usually better than application level, as payloads can be compressed in batches

39. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Large Payloads Remedy II Chunking Broker 1. Split to chunks & produce 2. Consume & reassemble Product Catalog Service Cart Service

40. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Large Payloads Remedy III Reference to Object Store 2. Produce with S3 URL 3. Consume & download from S3 1. Upload to S3 Product Catalog Service Cart Service Broker

41. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2016 Wix starts using event-driven 2018 Greyhound Resilient producer & Consumer retries 2019 We use IDs for ooo & duplicates 2020 Added compression by default

42. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky * bottlenecks Challenge #4 It’s hard for developers to debug and maintain event-driven microservices at scale in production

43. Our team

44. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Stream events with various filters

45. How do I investigate this lag? Our team

46. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Investigate consumer lag per partition

47. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky View a “stuck” event in some partition

48. How come this side-effect didn’t happen? Our team

49. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Orders Service Propagate the Context Broker Payments Topic Orders Topic Inventory Topic requestId userId Event Header 1. Greyhound produce * monitoring infra

50. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2. Greyhound consume Propagate the Context Payments Service Broker Payments Topic Orders Topic Inventory Topic 3. produce

51. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 4. Greyhound consume Propagate the Context Inventory Service Broker Payments Topic Orders Topic Inventory Topic

52. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky So developers can track events’ route

53. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky View event details

54. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky 2016 Wix starts using event-driven 2018 We open source Greyhound 2019 We use IDs for ooo & duplicates 2020 Added compression by default 2021-22 Tools in Production

55. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Wix developers have embraced event-driven architecture.

56. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Meeting these challenges made our microservices more decoupled, resilient and scalable, while keeping complexity low and data consistent.

57. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky The Blog Post https://medium.com/wix-engineerin g/event-driven-architecture-5-pitfalls-t o-avoid-b3ebf885bdb1

58. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky How to migrate 2000 microservices to Multi Cluster Managed Kafka with 0 Downtime The Next Step https://www.youtube.com/watch?v= XKbG8a-9NRE

59. Lessons Learned from 2000 Event-driven Microservices @NSilnitsky Greyhound github.com/wix/greyhound

60. Thank You! natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions?

DevSum - Lessons Learned from 2000 microservices

DevSum - Lessons Learned from 2000 microservices

Recommended

More Related Content

Similar to DevSum - Lessons Learned from 2000 microservices

Similar to DevSum - Lessons Learned from 2000 microservices(20)

More from Natan Silnitsky

More from Natan Silnitsky(20)

Recently uploaded

Recently uploaded(20)

DevSum - Lessons Learned from 2000 microservices