Deep Dive into Kafka Connect Protocol with Catalin Pop
Report
HostedbyConfluentFollow
Jun. 2, 2023•0 likes•143 views
Deep Dive into Kafka Connect Protocol with Catalin Pop
Jun. 2, 2023•0 likes•143 views
Download to read offline
Report
Technology
"Kafka Connect is the highway to our central nervous system for streaming data into Kafka. This highway has allowed us to connect hundreds of external services without the need to write code for each individual integration. Kafka Connect is based on the concept of having Connectors and tasks that do the work for us. There’s two fundamental questions that come up when it comes to our highway: • How is this highway made? How are connectors created/started? • How do we make sure there’s no traffic jams? Also known as, • How are connectors evenly distributed across nodes? In this presentation, we will answer the above questions and cover the following topics: • Breaking down how a Connector/task is created • Breaking down each Kafka Connect protocol(eager, compatible, sessioned) • Walk through rebalances for each protocol • Pros/cons of each protocol"
HostedbyConfluentFollow
Recommended
Scaling your apps with Kubernetes and Docker - TheConf 2018Erick Wendel
Continuous Delivery @ Codemotion Miguel Serrano Milano
Continuous Deployment of Architectural ChangeMatt Graham
Distributed Versioning Tools, BeJUG 2010Pursuit Consulting
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...confluent
Trunk-Based Development and TogglingBryan Liu
More Related Content
Similar to Deep Dive into Kafka Connect Protocol with Catalin Pop
Edition Based Redefinition Made Easy - Oren NakdimonOren Nakdimon
flowr streamlining computing workflowssahil seth
GraphQL vs. (the) RESTcoliquio GmbH
The Power of RefactoringStefan Koopmanschap
Deployment With Subversion - Lorna Mitchelldpc
Version Control ThinkVitaminAlex Hillman
More from HostedbyConfluent
Chill, Distill, No Overkill: Best Practices to Stress Test Kafka with Siva Ku...HostedbyConfluent
Manage Consistent Configurations Across Multiple Kafka Environments with Naga...HostedbyConfluent
The Cost of Kafka’s High Availability on Cloud with Geetha AnneHostedbyConfluent
Case-Study: Building Real-Time Applications at Scale-Cyclist Crash Detection ...HostedbyConfluent
Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh...HostedbyConfluent
Eliminating the Double Write Problem in Apache Kafka Using the Outbox Pattern...HostedbyConfluent
Recently uploaded
How to Manage Your Offshore Software Development Team EfficientlyCapital Numbers
Keynote: Two years at the British Library... and counting / Alan Danskin (Bri...CILIP MDG
TaketoFujikawa_KES2023Matsushita Laboratory
inQuba Webinar Series 2023 - Sanlam Hyper Personalization v2.0 FINAL.pptxLizaNolte
Unleashing the Power of Modern Carpooling Apps, Inspired by BlaBlaCarArchie Cadell
Top tips for boosting your digital skills, with BT Group and Age UKAbilityNet
Deep Dive into Kafka Connect Protocol with Catalin Pop
- 1. Deep Dive in Connect Protocol @cdpop_ Catalin Pop
- 2. ❏ High level overview of Connect ❏ Breaking down creation of Connectors ❏ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 3. High level overview of Connect @cdpop_
- 4. High level overview of Connect @cdpop_
- 5. High level overview of Connect @cdpop_
- 6. High level overview of Connect @cdpop_
- 7. High level overview of Connect @cdpop_
- 8. High level overview of Connect @cdpop_
- 9. High level overview of Connect @cdpop_
- 10. High level overview of Connect @cdpop_
- 11. High level overview of Connect @cdpop_
- 12. ✓ High level overview of Connect ❏ Breaking down creation of Connectors ❏ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 13. Connector Creation @cdpop_
- 14. Connector Creation @cdpop_
- 15. Connector Creation @cdpop_
- 16. Connector Creation @cdpop_
- 17. Connector Creation @cdpop_
- 18. Connector Creation @cdpop_
- 19. Connector Creation @cdpop_
- 20. Connector Creation @cdpop_
- 21. Connector Creation @cdpop_
- 22. Connector Creation @cdpop_
- 23. Connector Creation @cdpop_
- 24. Connector Creation @cdpop_
- 25. Connector Creation @cdpop_
- 26. Connector Creation @cdpop_
- 27. Connector Creation @cdpop_
- 28. Connector Creation @cdpop_
- 29. Connector Creation @cdpop_
- 30. Connector Creation @cdpop_
- 31. Connector Creation @cdpop_
- 32. Connector Creation @cdpop_
- 33. Connector Creation @cdpop_
- 34. Connector Creation @cdpop_
- 35. Connector Creation @cdpop_
- 36. Connector Creation @cdpop_
- 37. ✓ High level overview of Connect ✓ Breaking down creation of Connectors ❏ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_ @cdpop_
- 38. Breaking down connect protocols @cdpop_
- 39. Breaking down connect protocols @cdpop_
- 40. Eager Protocol ● Revoke all Connectors ● Perform round-robin assignment ● Send Connector assignment to queue ● Workers pickup assignment ● Start assigned connectors @cdpop_
- 41. Compatible Protocol ● Revoke connectors when the worker left the group or connector/task change ● Generate connector assignment only for the revoked connectors ● Send Connector assignment to queue ● Workers pickup assignment ● Start assigned connectors @cdpop_
- 42. Sessioned Protocol ● Setup Session key for encrypted authentication ● Revoke connectors when the worker left the group or connector/task change ● Generate connector assignment only for the revoked connectors ● Send Connector assignment to queue ● Workers pickup assignment ● Start assigned connectors @cdpop_
- 43. ✓ High level overview of Connect ✓ Breaking down creation of Connectors ✓ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 44. Walk through rebalance @cdpop_
- 45. Walk through rebalance @cdpop_
- 46. Walk through rebalance @cdpop_
- 47. Walk through rebalance @cdpop_
- 48. Walk through rebalance @cdpop_
- 49. Walk through rebalance @cdpop_
- 50. Walk through rebalance @cdpop_
- 51. Walk through rebalance @cdpop_
- 52. Walk through rebalance @cdpop_
- 53. Walk through rebalance @cdpop_
- 54. Walk through rebalance @cdpop_
- 55. Walk through rebalance @cdpop_
- 56. Walk through rebalance @cdpop_
- 57. Walk through rebalance - Eager @cdpop_
- 58. Walk through rebalance - Eager @cdpop_
- 59. Walk through rebalance - Eager @cdpop_
- 60. Walk through rebalance - Eager @cdpop_
- 61. Walk through rebalance - Eager @cdpop_
- 62. Walk through rebalance - Compatible/Sessioned @cdpop_
- 63. Walk through rebalance - Compatible/Sessioned @cdpop_
- 64. Walk through rebalance - Compatible/Sessioned @cdpop_
- 65. Walk through rebalance - Compatible/Sessioned @cdpop_
- 66. ✓ High level overview of Connect ✓ Breaking down creation of Connectors ✓ Breaking down Connect Protocols ✓ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 67. Pros/Cons of protocols Eager Pro Evenly distributed Con Not secured Stops all connectors @cdpop_
- 68. Pros/Cons of protocols Eager Pro Evenly distributed Con Not secured Stops all connectors Compatible Pro Incremental rebalance Con Not secured Uneven distribution @cdpop_
- 69. Pros/Cons of protocols Eager Pro Evenly distributed Con Not secured Stops all connectors Sessioned Pro Secure Incremental rebalance Con Uneven distribution Compatible Pro Incremental rebalance Con Not secured Uneven distribution @cdpop_
- 70. Thank you @cdpop_