Timely Auto-Scaling of Kafka Streams Pipelines with Remotely Connected APIs with Torben Meyer
•
0 likes•172 views
"Calling APIs over the network from Kafka Streams is often a necessary evil: Although it can incur significant costs by blocking the processing and thus making our pipelines less reliable, we are sometimes forced to integrate business process-related microservices or machine learning models and other technology that does not pair well with the JVM. While we scale the Kafka Streams applications and the services in Kubernetes to avoid blocked pipelines, it is often too late, as the scaling traditionally relies on metrics like the consumer group lag or number of HTTP requests. In this talk, we first give an overview of the caveats when integrating such services in Kafka Streams and basic approaches for mitigating those. Second, we present our solution for the timely scaling of complex Kafka Streams pipelines in conjunction with remotely connected APIs. In addition to the well-known metrics, we take further dimensions into account. Powered by Kafka Streams, we observe our data stream, extract metadata, aggregate statistics, and finally expose them as external metrics. We integrate this with auto-scaling frameworks such as KEDA to reliably scale our pipelines just in time."
Recommended
Recommended
More Related Content
Similar to Timely Auto-Scaling of Kafka Streams Pipelines with Remotely Connected APIs with Torben Meyer
Similar to Timely Auto-Scaling of Kafka Streams Pipelines with Remotely Connected APIs with Torben Meyer (20)
More from HostedbyConfluent
More from HostedbyConfluent (20)
Recently uploaded
Recently uploaded (20)
Timely Auto-Scaling of Kafka Streams Pipelines with Remotely Connected APIs with Torben Meyer
- 1. Timely Autoscaling of Kafka Streams Pipelines with Remotely Connected APIs Torben Meyer
- 2. Pipeline 2 Display Event Producer Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Preprocessing
- 3. Error Handling 3 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Display Event Producer Preprocessing
- 4. Option 1: Retry Strategy 4 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Display Event Producer Preprocessing
- 5. Option 2: Dead-Letter Queue 5 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Display Event Producer Preprocessing DLQ
- 6. Option 3: Back off Topics 6 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Display Event Producer Preprocessing 10 100 10000 Exponential Back off Topics
- 7. Option 4: Let it Crash 7 7 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Display Event Producer Preprocessing
- 8. Error Handling 8 Retry Strategy Back off Topics Dead-Letter Queue Let it Crash
- 9. How fast can we go? 9 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ?
- 10. How fast can we go? 10 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS User Prediction 2 Ad Prediction 1.3 Predictor Kafka Streams
- 11. How fast can we go? 11 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency User Prediction 2 500ms Ad Prediction 1.3 750ms Predictor Kafka Streams
- 12. How fast can we go? 12 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency User Prediction 2 500ms Ad Prediction 1.3 750ms Predictor Kafka Streams
- 13. How fast can we go? 13 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency User Prediction 2 500ms Ad Prediction 1.3 750ms Predictor Kafka Streams
- 14. Predictor Kafka Streams Predictor Kafka Streams How fast can we go? 14 User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS User Prediction 2 Ad Prediction 1.3
- 15. How fast can we go? Predictor Kafka Streams Predictor Kafka Streams 15 User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency User Prediction 2 1000ms Ad Prediction 1.3 1500ms
- 16. How fast can we go? Predictor Kafka Streams Predictor Kafka Streams 16 User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency User Prediction 2 1000ms Ad Prediction 1.3 1500ms
- 17. Our Goal: Robust and Performant Pipelines 17
- 20. Caching 20 Display Event Producer Predictor Kafka Streams User Prediction REST API Ad Prediction REST API Preprocessing Cache
- 21. Caching 21 Cache Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency (L) Cache Miss Ratio (CMR) User Prediction 2 500ms 90% Ad Prediction 1.3 750ms 10%
- 22. Caching 22 Cache Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency (L) Cache Miss Ratio (CMR) User Prediction 2 500ms 90% Ad Prediction 1.3 750ms 10%
- 23. Caching 23 Cache Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Service RPS Latency (L) Cache Miss Ratio (CMR) User Prediction 2 500ms 90% Ad Prediction 1.3 750ms 10%
- 25. Autoscaling 25 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale
- 26. Autoscaling 26 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale
- 27. Autoscaling 27 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale
- 28. Autoscaling 28 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale What is the lag of the consumer group?
- 29. Autoscaling 29
- 30. Autoscaling 30
- 31. Autoscaling 31 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale What is the lag of the consumer group?
- 32. Predictor Kafka Streams Ad Prediction REST AP User Prediction REST API Predictor Kafka Streams User Prediction REST API Autoscaling 32 Predictor Kafka Streams Ad Prediction REST AP User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale What is the lag of the consumer group?
- 33. Results - Lag-based Autoscaling 33
- 34. Results - Lag-based Autoscaling 34 Lag increases ⟹ KEDA triggers scale-up
- 35. Results - Lag-based Autoscaling 35 Rebalance
- 36. Results - Lag-based Autoscaling 36 5 replicas catch up
- 37. Results - Lag-based Autoscaling 37 3 replicas continue
- 39. Timely Autoscaling 39 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale Display Event Producer Preprocessing Metrics Consumer / REST API
- 40. Timely Autoscaling 40 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale Display Event Producer Preprocessing Metrics Consumer / REST API How many unique user and ad IDs are there?
- 41. Metric 41 {} {} Data Structures Cached User Unique Elements t0
- 42. Metric 42 user1 → t0 t0 → 1 Data Structures Cached User Unique Elements t0 user1
- 43. Metric 43 user1 → t0 … t0 → 4 Data Structures Cached User Unique Elements t0 user1 …
- 44. Metric 44 user1 → t0 user2 → t1 … t0 → 4 → 1 Data Structures Cached User Unique Elements t0 t1 user1 … user2
- 45. Metric 45 user1 → t0 user2 → t1 … t0 → 4 t1 → 1 Data Structures Cached User Unique Elements t0 t1 user1 … user2 user1
- 46. Metric 46 user1 → t0 user2 → t1 user3 → t2 … t0 → 4 t1 → 3 t2 → 4 … Data Structures Cached User Unique Elements t0 t1 t2 user1 … user2 user1 … user3 …
- 47. Metric 47 user1 → t0 user2 → t1 user3 → t2 … t0 → 4 t1 → 3 t2 → 4 … t30 → 3 Data Structures Cached User Unique Elements t0 t1 t2 user1 … user2 user1 … user3 … 3.8 3.7
- 48. Timely Autoscaling 48 Predictor Kafka Streams User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale Display Event Producer Preprocessing Metrics Consumer / REST API How many unique user and ad IDs are there? ~ required RPS
- 49. User Prediction REST API Timely Autoscaling 49 Predictor Kafka Streams Predictor Kafka Streams Ad Prediction REST API User Prediction REST API Ad Prediction REST API w/spredictor ? Horizontal Pod Autoscaler Scale Display Event Producer Preprocessing Metrics Consumer / REST API How many unique user and ad IDs are there? ~ required RPS
- 51. Results - Timely Autoscaling 51
- 52. Results - Timely Autoscaling 52 Timely autoscaling triggers before lag grows
- 53. Results - Timely Autoscaling 53 Timely autoscaling triggers before lag grows No need to catch up
- 54. Results - Timely Autoscaling 54
- 55. Timely Autoscaling of Kafka Streams Pipelines with Remotely Connected APIs Torben Meyer Software Engineer https://github.com/bakdata/timely-autoscaling-demo torben.meyer@bakdata.com Kafka Summit London 2023