Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

1. Mason Chen | Apple
Multi Cluster Kafka Source
THIS IS NOT A CONTRIBUTION

2. Agenda
Motivation
FLIP 27 Kafka Source
Source Design
Example

3. Flink Kafka Pipeline

4. Manual Migration Steps

5. Manual Migration Steps
Bring up new cluster

6. Manual Migration Steps
Swap producer

7. Manual Migration Steps
Wait for consumer to drain

8. Manual Migration Steps
Source uid and cluster change

9. Manual Migration Steps
Upgrade with non restore state

10. Manual Migration Steps
Increase parallelism for lag

11. Manual Migration Steps
Revert to steady state

12. Manual Migration Steps
When can we remove nonactive cluster?

13. User Manual Migration Steps
• Change source uid
• Change bootstrap server
• Upgrade application
• With non restore state
• Change parallelism and resources to catch with lag
• Revert to steady state when caught up

14. Manual Migration Steps
• Application downtime
• Need to increase system resources for catchup
• User manual toil
• User could have 100+ jobs
• Multiple hours of team coordination
Drawbacks

15. Scaling Multiple Kafka Clusters
• Hybrid cloud: on-prem, private cloud and public cloud providers
• Scalability
• Topic sharding
• Operability and Failover
• In place upgrade is complex and error prone

16. Agenda
Motivation
FLIP 27 Kafka Source
Source Design
Example

17. FLIP 27 Source
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/sources/

18. FLIP 27 Source
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/sources/

19. FLIP 27 Source
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/sources/

20. FLIP 27 Source
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/sources/

21. FLIP 27 Source
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/sources/

22. FLIP 27 Kafka Source

23. FLIP 27 Kafka Source

24. FLIP 27 Kafka Source

25. FLIP 27 Kafka Source

26. Agenda
Motivation
FLIP 27 Kafka Source
Source Design
Example

27. Kafka Metadata Service
• KafkaStream
• Logical abstraction to physical
clusters and topics
• describeStreams(Collection<String>
streamIds);
• Pluggable implementation
• File based configmap

28. Multi Cluster Kafka Source
Runtime

29. Multi Cluster Kafka Source
Runtime

30. Multi Cluster Kafka Source
Runtime

31. Multi Cluster Kafka Source
Runtime

32. Multi Cluster Kafka Source
Runtime

33. Multi Cluster Kafka Source
Runtime

34. Multi Cluster Kafka Source
Runtime

35. Multi Cluster Kafka Source
Runtime

36. Multi Cluster Kafka Source
Runtime

37. Multi Cluster Kafka Source
Runtime

38. Multi Cluster Kafka Source
Runtime

39. Multi Cluster Kafka Source
Runtime

40. Multi Cluster Kafka Source
Runtime

41. Extension of FLIP 27 Major Components
• Kafka Source components
• Polling, commit, checkpoint, split assignment,
• Source Event RPC
• Enumerator Context Proxy
• Split assignment and wrapping cluster info
• Context thread pools

42. Agenda
Motivation
FLIP 27 Kafka Source
Source Design
Example

43. Migration with Multi Cluster Kafka Source

44. Migration with Multi Cluster Kafka Source
Initial metadata

45. Migration with Multi Cluster Kafka Source
Bring up new cluster

46. Migration with Multi Cluster Kafka Source
Bring up new cluster

47. Migration with Multi Cluster Kafka Source
Add new cluster metadata

48. Migration with Multi Cluster Kafka Source
Reconcile metadata

49. Migration with Multi Cluster Kafka Source
Reconcile metadata

50. Migration with Multi Cluster Kafka Source
Remove old cluster

51. Migration with Multi Cluster Kafka Source
Reconcile metadata

52. Migration with Multi Cluster Kafka Source
Reconcile metadata

53. Migration with Multi Cluster Kafka Source
Remove old cluster

54. User Cluster Migration Steps

55. Multi Cluster Kafka Source Benefits
• Migrations and failover automated transparently within source
• Simplify operations between compute and storage infra
• Hybrid Source compatible
• Can be leveraged for topic migration

56. Future Work
• Integrate with split level watermark alignment
• Optimizations to remove only affected readers
• FLIP-246 (https://cwiki.apache.org/confluence/display/FLINK/
FLIP-246%3A+Multi+Cluster+Kafka+Source)

57. Q&A