Download to read offline
Uploaded byFlink Forward
One sink to rule them all: Introducing the new Async Sink
The document introduces the new async sink for Apache Flink, detailing its internal architecture and functionality including buffering, batching, retry mechanisms, and managing throughput. It outlines configuration options and current limitations like ordering guarantees and thread pool management. Future enhancements mentioned include rate limiting features and integration with Amazon DynamoDB.
In this document
Powered by AI
Introduction of the Async Sink by AWS team members Danny Cranmer and Steffen Hausmann.
Overview of the Apache Flink connector ecosystem including Amazon Kinesis, Elasticsearch, RabbitMQ, and Google PubSub.
Key aspects for developing a high-quality connector: buffering, batch sending, request retries, checkpointing, and rate limiting.
Detailed internals of the Async Sink focusing on components like the internal buffer, request entries, and error handling.
Methodologies for managing throughput, including in-flight request limits and adjustments based on batch request success or failure.
Configuration settings for Async Sink, including max record size, in-flight requests, batch size, and buffer timing.
Discussion of limitations in the Async Sink such as at-least-once delivery, ordering, and thread pool management.
Next steps for Async Sink including configurable rate limiting and a new Amazon DynamoDB Sink to boost community adoption.
Concluding remarks and thanks from presenters Danny Cranmer and Steffen Hausmann.
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
One sink to rule them all: Introducing the new Async Sink
- 1. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. One sink to rule them all Introducing the new Async Sink Danny Cranmer Sr. Software Development Engineer, AWS Steffen Hausmann Principal Streaming Architect, AWS
- 2. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Apache Flink’s connector ecosystem 2 Amazon Kinesis Data Streams Elasticsearch RabbitMQ Google PubSub
- 3. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. How to build a high quality connector Buffering Send batch requests Retry failed requests Checkpointing Rate limiting and Backpressure
- 4. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Async Sink internals internal buffer endpoint message request entries batch request response requeue failed requests checkpoint sink to AsyncSinkWriter request entries -> batch request ElementConverter message -> request entry AsyncSinkWriter response -> failed requests AsyncSink WriterStateSerializer 4
- 5. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. How the Async Sink manages throughput 6 endpoint sink subtask3 sink subtask2 sink subtask1 1. Limit the number of in-flight request entries 2. If a batch request was successful, increase the limit by a constant factor 3. If a batch request failed, cut the limit in half Additive Increase/Multiplicative Decrease
- 6. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Let’s go build! 7
- 7. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Async Sink configuration endpoint message request entries batch request sink to maxRecordSizeInBytes 8 maxInFlightRequests maxBatchSize maxBatchSizeInBytes maxBufferedRequests maxTimeInBufferMs internal buffer
- 8. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Current limitations At least once Ordering Thread pool management
- 9. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. What’s Next? FLIP-242 Configurable Rate Limiting FLIP-252 Amazon DynamoDB Sink Wider Community Adoption
- 10. ONE SINK TORULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. Danny Cranmer Steffen Hausmann