Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Grokking Techtalk #39: How to build an event driven architecture with Kafka & Kafka Connect

148 views

Published on

Bài talk chia sẻ về quá trình 2 năm ứng dụng Kafka và Kafka Connect để chuyển đổi mô hình hệ thống của Vexere từ Monolithic thành Microservice, Event Driven:
+ Event driven architecture là gì?
+ Làm thế nào để xây dựng 1 hệ thống event driven architecture một cách hiệu qủa bằng Kafka và Kafka Connect
+ Các use case hữu ích với Kafka & Kafka Connect
+ Kinh nghiệm thực tế và các bài học rút ra

- Về diễn giả: Anh Nguyễn Văn Lợi là một kỹ sư phần mềm với hơn 9 năm kinh nghiệm thực tế từ các công ty có hệ thống lớn trong các mảng VoIP, Ecommerce, Big Data, Logistics. Tại Vexere, anh luôn đề cao tinh thần tự học hỏi, phát triển và chia sẻ để team member liên tục tích lũy kiến thức, kỹ năng, nhằm tăng hiệu quả công việc và mang lại sản phẩm có trải nghiệm tốt nhất cho người dùng.

Published in: Engineering
  • Be the first to comment

Grokking Techtalk #39: How to build an event driven architecture with Kafka & Kafka Connect

  1. 1. How to build an event driven architecture with Kafka & Kafka Connect Nov 12, 2020 Lợi Nguyễn - Technical Architect @ VeXeRe 1
  2. 2. Vietnam’s largest online bus booking systemvexere.com Name: Nguyễn Văn Lợi Company: ● Vexere - #1 Saas based bus ticket platform in Vietnam ● Chotot - #1 Classified Marketplace in Vietnam ● Blue Orchid - A start-up founded by ex-Grab CTO ● Softfoundry - VoIP product 2
  3. 3. Vietnam’s largest online bus booking systemvexere.com VeXeRe.com is a Vietnamese online bus ticket booking system that operates through many transportation companies. 3
  4. 4. Vietnam’s largest online bus booking systemvexere.com 4
  5. 5. Vietnam’s largest online bus booking systemvexere.com 5
  6. 6. Vietnam’s largest online bus booking systemvexere.com ● Why event driven architecture? ● What is “Event Driven” architecture? ○ Event-carried State Transfer ○ Event Sourcing ● Event Sourcing in real world ○ What is 2 phase write? ○ MSSQL / transaction log ○ Postgresql / WAL ● What is Kafka & Kafka Connect? ○ Connector/Task/Worker ■ MSSQL Source Connector ○ Transform ○ Kafka and Kafka Connect @ vexere ○ Pros/Cons of Kafka Connect vs Custom Producer ● Use Case/Demo ○ Customer Journey Map Notification ○ Social Proof ○ Event Sourcing + CQRS ○ Data Warehouse ● Experience / Tools / Troubleshoot ○ Tools: (kafka manager, kafka tool) ○ Troubleshoot connector ○ Monitoring ○ Domain Event vs Event Sourcing Event ● Q & A & Discussion 6
  7. 7. Vietnam’s largest online bus booking systemvexere.com 7
  8. 8. Vietnam’s largest online bus booking systemvexere.com 8 Phase Challenges Launch ● Business Idea Profitability ● Limit resource: ○ Time ○ Technology ○ Money Growth/Expansion ● Increasing Customers ● Increasing Feature ● Adding New Products/BU
  9. 9. Vietnam’s largest online bus booking systemvexere.com ● Event-carried State Transfer ● Event Sourcing 9
  10. 10. Vietnam’s largest online bus booking systemvexere.com 10
  11. 11. Vietnam’s largest online bus booking systemvexere.com 11
  12. 12. Vietnam’s largest online bus booking systemvexere.com 12
  13. 13. Vietnam’s largest online bus booking systemvexere.com ● Two representation of the world: ○ Application State: the current representation of the world, and ○ log of all the events: that changed that world ● The test definition of Event Sourcing: ○ at any time we can blow away the application state and confidently rebuild it from the log. ● Benefit: ○ Audits ○ Debugging 13
  14. 14. Vietnam’s largest online bus booking systemvexere.com ● What is “Event Driven” architecture? ○ Event-carried State Transfer ○ Event Sourcing 14
  15. 15. Vietnam’s largest online bus booking systemvexere.com ● Event Sourcing in real world ○ What is 2 phase write? ○ MSSQL / transaction log ○ Postgresql / WAL 15
  16. 16. Vietnam’s largest online bus booking systemvexere.com 16 Insert without WAL
  17. 17. Vietnam’s largest online bus booking systemvexere.com 17 Insertion operations with WAL
  18. 18. Vietnam’s largest online bus booking systemvexere.com 18
  19. 19. Vietnam’s largest online bus booking systemvexere.com 19
  20. 20. Vietnam’s largest online bus booking systemvexere.com ● Event Sourcing in real world ○ What is 2 phase write? ○ MSSQL / transaction log ○ Postgresql / WAL 20
  21. 21. Vietnam’s largest online bus booking systemvexere.com ● What is Kafka & Kafka Connect? ○ Connector/Task/Worker ○ Transform ○ How we use Kafka and Kafka Connect @ vexere ○ Pros/Cons of Kafka Connect vs Custom Producer 21
  22. 22. Vietnam’s largest online bus booking systemvexere.com ● topic ● producer ● consumer ● broker ● partition ● consumer group 22
  23. 23. Vietnam’s largest online bus booking systemvexere.com 23
  24. 24. Vietnam’s largest online bus booking systemvexere.com 24
  25. 25. Vietnam’s largest online bus booking systemvexere.com 25
  26. 26. Vietnam’s largest online bus booking systemvexere.com 26
  27. 27. Vietnam’s largest online bus booking systemvexere.com 27
  28. 28. Vietnam’s largest online bus booking systemvexere.com 28
  29. 29. Vietnam’s largest online bus booking systemvexere.com Kafka Connect is a framework to stream data into and out of Apache Kafka ● Connectors – the high level abstraction that coordinates data streaming by managing tasks ● Tasks – the implementation of how data is copied to or from Kafka ● Workers – the running processes that execute connectors and tasks ● Converters – the code used to translate data between Connect and the system sending or receiving data ● Transforms – simple logic to alter each message produced by or sent to a connector ● Dead Letter Queue – how Connect handles connector errors 29
  30. 30. Vietnam’s largest online bus booking systemvexere.com No coding required, just json config: 30
  31. 31. Vietnam’s largest online bus booking systemvexere.com 31
  32. 32. Vietnam’s largest online bus booking systemvexere.com 32
  33. 33. Vietnam’s largest online bus booking systemvexere.com 33
  34. 34. Vietnam’s largest online bus booking systemvexere.com 34
  35. 35. Vietnam’s largest online bus booking systemvexere.com 35
  36. 36. Vietnam’s largest online bus booking systemvexere.com 36
  37. 37. Vietnam’s largest online bus booking systemvexere.com Pros Cons ● Many Connectors (source/sink) ● No coding required ● Simple transform only ● Hard to customize or write your own connector 37
  38. 38. Vietnam’s largest online bus booking systemvexere.com ● What is Kafka & Kafka Connect? ○ Connector/Task/Worker ■ MSSQL Source Connector ○ Transform ○ How we use Kafka and Kafka Connect @ vexere ○ Pros/Cons of Kafka Connect vs Custom Producer 38
  39. 39. Vietnam’s largest online bus booking systemvexere.com ● Monitor kafka connect job ● AlwaysOn Cluster Config ● Database schema evolution 39
  40. 40. Vietnam’s largest online bus booking systemvexere.com 40
  41. 41. Vietnam’s largest online bus booking systemvexere.com In Kafka Connect, task is being killed and will not recover until manually restarted Solution: ● Cronjob to monitor task status, then restart task by calling restful to task api ● Dead Letter Queue to handle error in: ○ Convert ○ Transform 41
  42. 42. Vietnam’s largest online bus booking systemvexere.com 42 Reference: https://debezium.io/documentation/reference/connectors/sqlserver.html#sqlserver-schema-evolution
  43. 43. Vietnam’s largest online bus booking systemvexere.com 43
  44. 44. Vietnam’s largest online bus booking systemvexere.com 44 ● NVARCHAR(max) is not supported in CDC table (cannot record before value, only have after update value)
  45. 45. Vietnam’s largest online bus booking systemvexere.com ● Experience / Tools / Troubleshoot ○ Tools: (kafka manager, kafka tool) ○ Troubleshoot connector ○ Monitoring 45
  46. 46. Vietnam’s largest online bus booking systemvexere.com 46 Reward Thay đổi thông tin tài xế, biển số xe Đánh giá sau chuyến điThanh toán
  47. 47. Vietnam’s largest online bus booking systemvexere.com Calculated from Ticket & Booking event 47
  48. 48. Vietnam’s largest online bus booking systemvexere.com Separate read & write model Write model: MSSQL Read model: Elasticsearch MSSQL ⇒ Kafka ⇒ Kafka consumer ⇒ Elasticsearch 48
  49. 49. Vietnam’s largest online bus booking systemvexere.com Example: Real Time sync data from MSSQL ⇒ Stagging Postgres ⇒ Bigquery Note: ● be careful when backfill data ● If we new column, we have to trigger dummy update to trigger all record event => a lot of trash in transaction log ==> need to write your own job 49
  50. 50. Vietnam’s largest online bus booking systemvexere.com 50
  51. 51. Vietnam’s largest online bus booking systemvexere.com 51 Reference: https://www.enterpriseintegrationpatterns.com/patterns/messaging/index.html
  52. 52. Vietnam’s largest online bus booking systemvexere.com ● https://martinfowler.com/articles/201701-event-driven.html ● https://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latenc y-data-pipelines/ ● https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/about-change- data-capture-sql-server?view=sql-server-2017 ● https://docs.confluent.io/current/connect/concepts.html ● https://www.slideshare.net/ConfluentInc/from-zero-to-hero-with-kafka-connect ● https://www.innoq.com/en/blog/domain-events-versus-event-sourcing/#eventsfromeven tsourcing%E2%89%A0domainevents 52
  53. 53. Vietnam’s largest online bus booking systemvexere.com 53
  54. 54. Vietnam’s largest online bus booking systemvexere.com 54

×