One sink to rule them all: Introducing the new Async Sink

•Download as PPTX, PDF•

0 likes•379 views

Flink Forward San Francisco 2022. Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework. by Steffen Hausmann & Danny Cranmer

Technology

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
© 2022, Amazon Web Services, Inc. or its affiliates.
One sink to rule them all
Introducing the new Async Sink
Danny Cranmer
Sr. Software Development Engineer, AWS
Steffen Hausmann
Principal Streaming Architect, AWS

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Apache Flink’s connector ecosystem
2
Amazon Kinesis
Data Streams
Elasticsearch
RabbitMQ
Google PubSub

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
How to build a high quality connector
Buffering Send batch
requests
Retry failed
requests
Checkpointing Rate limiting and
Backpressure

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Async Sink internals
internal buffer
endpoint
message request entries
batch request
response
requeue
failed requests
checkpoint
sink to
AsyncSinkWriter
request entries -> batch request
ElementConverter
message -> request entry
AsyncSinkWriter
response -> failed requests
AsyncSink
WriterStateSerializer
4

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
How the Async Sink manages throughput
6
endpoint
sink subtask3
sink subtask2
sink subtask1
1. Limit the number of in-flight request entries
2. If a batch request was successful, increase
the limit by a constant factor
3. If a batch request failed, cut the limit in half
Additive Increase/Multiplicative Decrease

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Async Sink configuration
endpoint
message request entries
batch request
sink to
maxRecordSizeInBytes
8
maxInFlightRequests
maxBatchSize
maxBatchSizeInBytes
maxBufferedRequests
maxTimeInBufferMs
internal buffer

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Current limitations
At least
once
Ordering Thread pool
management

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
What’s Next?
FLIP-242
Configurable
Rate Limiting
FLIP-252
Amazon DynamoDB
Sink
Wider
Community
Adoption

ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK
© 2022, Amazon Web Services, Inc. or its affiliates.
Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates.
Danny Cranmer Steffen Hausmann

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...

Flink Forward

Flink Forward San Francisco 2022. Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink. by Nico Kruber

Practical learnings from running thousands of Flink jobs

Flink Forward

Flink Forward San Francisco 2022. Task Managers constantly running out of memory? Flink job keeps restarting from cryptic Akka exceptions? Flink job running but doesn’t seem to be processing any records? We share practical learnings from running thousands of Flink Jobs for different use-cases and take a look at common challenges they have experienced such as out-of-memory errors, timeouts and job stability. We will cover memory tuning, S3 and Akka configurations to address common pitfalls and the approaches that we take on automating health monitoring and management of Flink jobs at scale. by Hong Teoh & Usamah Jassat

Where is my bottleneck? Performance troubleshooting in Flink

Flink Forward

Flinkn Forward San Francisco 2022. In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times. by Piotr Nowojski

Building a fully managed stream processing platform on Flink at scale for Lin...

Flink Forward

Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Processing Semantically-Ordered Streams in Financial Services

Flink Forward

Flink Forward San Francisco 2022. What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases. by Patrick Lucas

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Flink Forward San Francisco 2022. This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications. We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs. After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are. by David Moravek

Evening out the uneven: dealing with skew in Flink

Flink Forward

Flink Forward San Francisco 2022. When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment. by Jun Qin & Karl Friedrich

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward

Flink Forward San Francisco 2022. To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy. by Aansh Shah

Flink powered stream processing platform at Pinterest

Flink Forward

Flink Forward San Francisco 2022. Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework. by Rainie Li & Kanchi Masalia

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Near real-time statistical modeling and anomaly detection using Flink!

Flink Forward

Flink Forward San Francisco 2022. At ThousandEyes we receive billions of events every day that allow us to monitor the internet; the most important aspect of our platform is to detect outages and anomalies that have a potential to cause serious impact to customer applications and user experience. Automatic detection of such events at lowest latency and highest accuracy is extremely important for our customers and their business. After launching several resilient and low latency data pipelines in production using Flink we decided to take it up a notch; we leveraged Flink to build statistical models in near real-time and apply them on incoming stream of events to detect anomalies! In this session we will deep dive into the design as well as discuss pitfalls and learnings while developing our real-time platform that leverages Debezium, Kafka, Flink, ElasticCache and DynamoDB to process events at scale! by Kunal Umrigar & Balint Kurnasz

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Databricks

Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flink Forward

Flink Forward San Francisco 2022. Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way. by Jeff Chao

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Virtual Flink Forward 2020: A deep dive into Flink SQL - Jark Wu

Flink Forward

During last two major versions (1.9 & 1.10), Apache Flink community spent lots of effort to improve the architecture for further unified batch & streaming processing. One example for that is Flink SQL added the ability to support multiple SQL planners under the same API. This talk will first discuss the motivation behind these movements, but more importantly will have a deep dive into Flink SQL. The presentation shows the unified architecture to handle streaming and batch queries and explain how Flink translates queries into the relational expressions, leverages Apache Calcite to optimize them, and generates efficient runtime code for execution. Besides, this talk will also describe the lifetime of a query in detail, how optimizer improve the plan based on relational node patterns, how Flink leverages binary data format for its basic data structure, and how does certain operator works. This would give audience better understanding of Flink SQL internals.

Introduction to Kafka Cruise Control

Jiangjie Qin

Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake

Databricks

Change Data Capture CDC is a typical use case in Real-Time Data Warehousing. It tracks the data change log -binlog- of a relational database [OLTP], and replay these change log timely to an external storage to do Real-Time OLAP, such as delta/kudu. To implement a robust CDC streaming pipeline, lots of factors should be concerned, such as how to ensure data accuracy , how to process OLTP source schema changed, whether it is easy to build for variety databases with less code.

Apache Iceberg Presentation for the St. Louis Big Data IDEA

Adam Doyle

Apache Flink Worst Practices

Konstantin Knauf

Distributed stream processing is evolving from a technology in the sidelines of Big Data to a key enabler for businesses to provide more scalable, real-time services to their customers. We at Ververica, the company founded by the original creators of Apache Flink, and other prominent players in the Flink community have witnessed this development from the driver’s seat. Working with our customer and the wider community we have seen great success stories and we have seen things going wrong. In this talk, I would like to share anecdotes and hard-learned lessons of adopting distributed stream processing – Apache Flink specific as well as across frameworks. Afterwards, you will know, how not to model your use cases as a stream processing application, which data structures not to use, how not to deal with failure, how not to approach the topic of monitoring and much more. Video: https://www.youtube.com/watch?v=F7HQd3KX2TQ&list=PLDX4T_cnKjD207Aa8b5CsZjc7Z_KRezGz&index=48&t=6s

Batch Processing at Scale with Flink & Iceberg

Flink Forward

Flink Forward San Francisco 2022. Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features. by Andreas Hailu

The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...

HostedbyConfluent

How does Kafka Streams and ksqlDB reason about time, how does it affect my application, and how do I take advantage of it? In this talk, we explore the "time engine" of Kafka Streams and ksqlDB and answer important questions how you can work with time. What is the difference between sliding, time, and session windows and how do they relate to time? What timestamps are computed for result records? What temporal semantics are offered in joins? And why does the suppress() operator not emit data? Besides answering those questions, we will share tips and tricks how you can "bend" time to your needs and when mixing event-time and processing-time semantics makes sense. Six month ago, the question "What's the time? …and Why?" was asked and partly answered at Kafka Summit in San Francisco, focusing on writing data, data storage and retention, as well as consuming data. In this talk, we continue our journey and delve into data stream processing with Kafka Streams and ksqlDB, that both offer rich time semantics. At the end of the talk, you will be well prepared to process past, present, and future data with Kafka Streams and ksqlDB.

Hive + Tez: A Performance Deep DiveDataWorks Summit

Apache Kafka Fundamentals for Architects, Admins and Developers

confluent

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in You...

Amazon Web Services

CON317_Advanced container management at catsndogs.lol

Amazon Web Services

"Catsndogs.lol is a (fictional) company that needs help deploying and scaling its container-based application. During this workshop, attendees will join the new DevOps team at CatsnDogs.lol, and help the company to manage their applications using Amazon EC2 Container Service (ECS), and help release new features to make our customers happier than ever. Attendees will get hands-on with service and container-instance auto-scaling, spot-fleet integration, container placement strategies, service discovery, secrets management with AWS Systems Manager Parameter Store, time-based and event-based scheduling, and automated deployment pipelines. If you are a developer interested in learning more about how Amazon ECS can accelerate your application development and deployment workflows, or if you are a systems administrator or devops person interested in understanding how Amazon ECS can simplify the operational model associated with running containers at scale, then this workshop is for you. You should have basic familiarity with ECS, EC2, and IAM. The workshop requires: A laptop (Windows, OSX, or Linux) with the AWSCLI or AWS Powershell installed An AWS account with administrative permissions (including the ability to create IAM roles and policies) created at least 24 hours in advance."

What's hot

Apache Flink in the Cloud-Native Era

Flink Forward

Evening out the uneven: dealing with skew in Flink

Flink Forward

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink Forward

Flink powered stream processing platform at Pinterest

Flink Forward

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Near real-time statistical modeling and anomaly detection using Flink!

Flink Forward

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Flink Forward

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Databricks

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Flink Forward

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Virtual Flink Forward 2020: A deep dive into Flink SQL - Jark Wu

Flink Forward

Introduction to Kafka Cruise Control

Jiangjie Qin

Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake

Databricks

Apache Iceberg Presentation for the St. Louis Big Data IDEA

Adam Doyle

Apache Flink Worst Practices

Konstantin Knauf

Batch Processing at Scale with Flink & Iceberg

Flink Forward

The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...

HostedbyConfluent

Hive + Tez: A Performance Deep DiveDataWorks Summit

Apache Kafka Fundamentals for Architects, Admins and Developers

confluent

What's hot (20)

Apache Flink in the Cloud-Native Era

Evening out the uneven: dealing with skew in Flink

“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...

Flink powered stream processing platform at Pinterest

Introducing the Apache Flink Kubernetes Operator

Near real-time statistical modeling and anomaly detection using Flink!

Using the New Apache Flink Kubernetes Operator in a Production Deployment

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Stephan Ewen - Experiences running Flink at Very Large Scale

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Virtual Flink Forward 2020: A deep dive into Flink SQL - Jark Wu

Introduction to Kafka Cruise Control

Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake

Apache Iceberg Presentation for the St. Louis Big Data IDEA

Apache Flink Worst Practices

Batch Processing at Scale with Flink & Iceberg

The Flux Capacitor of Kafka Streams and ksqlDB (Matthias J. Sax, Confluent) K...

Hive + Tez: A Performance Deep Dive

Apache Kafka Fundamentals for Architects, Admins and Developers

Similar to One sink to rule them all: Introducing the new Async Sink

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in You...

Amazon Web Services

CON317_Advanced container management at catsndogs.lol

Amazon Web Services

From One to Many: Diving Deeper into Evolving VPC Design (ARC310-R2) - AWS re...

Amazon Web Services

Most organizations today run their production workloads inside Amazon Virtual Private Cloud (Amazon VPC). This software-defined network structure provides the boundaries that are needed for the security that an organization and its customers require. For most organizations, the natural evolution in their architecture, security, and environment involves migrating from a single VPC to multiple VPCs in the same AWS Region and across many other AWS Regions. The question of how to enforce security policies while simplifying the flow of traffic between multiple VPCs, the data center, and remote offices while adhering to AWS best practices becomes an intricate one to answer. In this chalk talk, we provide solutions to scenarios like these and more. Topics include Amazon security groups, NACLs, static and dynamic VPNs, AWS Direct Connect, IPS and IDS, transit VPC architectures, designing for security, and more.

AWS re:Invent serverless recap day: Controls for-managing-events-in-serverles...

⛷️ Ben Smith

Event-driven architectures can help decouple services and simplify dependencies as your application grows. There are now new options for developers looking to benefit from this approach with the release of AWS Lambda controls for asynchronous and stream processing events, Lambda event destinations, and Amazon EventBridge, a serverless event bus service. In this tech talk, we'll demonstrate strategies for managing events to help you build more resilient event-driven applications. You will learn how to customize responses to Lambda function results with new service features to handle retries, failures, and prevent “poison pill” events. We will also demonstrate how to configure Lambda event destinations to route execution results to other AWS services, and how to set up routing rules in Amazon EventBridge that react to all of your data sources.

Born in the Cloud, Built like a Startup

Amazon Web Services

by Itzik Paz, Solutions Architect & Rich Cowper, Solutions Architect Manager, AWS This presentation compares three modern architecture patterns that startups are building their businesses around. It includes a realistic analysis of cost, team management, and security implications of each approach. It covers AWS Elastic Beanstalk, Amazon ECS, Docker, Amazon API Gateway, AWS Lambda, Amazon DynamoDB, and Amazon CloudFront.

Running a High-Performance Kubernetes Cluster with Amazon EKS (CON318-R1) - A...

Amazon Web Services

How do you ensure that a containerized system can handle the needs of your application? Designing and testing for performance is a critical aspect of operating containerized architectures at scale. In this session, we cover best practices for designing perfomant containerized applications on AWS using Kubernetes. We also show you how State Street deployed a high-performance database at scale using Amazon Elastic Container Service for Kubernetes (Amazon EKS).

AWS DevDay Berlin - Automating building blocks choices you will face with con...

Cobus Bernard

Advanced Patterns in Microservices Implementation with Amazon ECS - CON402 - ...

Amazon Web Services

Scaling a microservice-based infrastructure can be challenging in terms of both technical implementation and developer workflow. In this talk, AWS Solutions Architect Pierre Steckmeyer will be joined by Will McCutchen, Architect at BuzzFeed, to discuss Amazon ECS as a platform for building a robust infrastructure for microservices. We will look at the key attributes of microservice architectures and how Amazon ECS supports these requirements in production, from configuration to sophisticated workload scheduling to networking capabilities to resource optimization. We will also examine what it takes to build an end-to-end platform on top of the wider AWS ecosystem, and what it's like to migrate a large engineering organization from a monolithic approach to microservices.

CON319_Interstella GTC CICD for Containers on AWS

Amazon Web Services

"Interstella 8888 is an intergalactic trading company that deals in rare resources, but their antiquated monolithic logistics systems are causing the business to lose money. Join this workshop to learn how to set up a CI/CD pipeline for containerized microservices. You'll get hands-on experience deploying Docker container images using Amazon ECS AWS CloudFormation, AWS CodeBuild, and AWS CodePipline, automating everything from code check-in to production. AWS credits are provided. Bring your laptop, and have an active AWS account."

Interstella 8888: CICD for Containers on AWS - CON319 - re:Invent 2017

Amazon Web Services

Interstella 8888 is an intergalactic trading company that deals in rare resources, but their antiquated monolithic logistics systems are causing the business to lose money. Join this workshop to learn how to set up a CI/CD pipeline for containerized microservices. You'll get hands-on experience deploying Docker container images using Amazon ECS AWS CloudFormation, AWS CodeBuild, and AWS CodePipline, automating everything from code check-in to production. AWS credits are provided. Bring your laptop, and have an active AWS account.

Introduction to AWS Fargate & Amazon Elastic Container Service for Kubernetes

Amazon Web Services

AWS DevDay Vienna - Automating building blocks choices you will face with con...

Cobus Bernard

CI/CD with AWS Developer Tools and Fargate

Amazon Web Services

CI/CD for a Docker Node.JS application using Code* services. This session will walkthrough what a solution like this would look like, what Code* services are used, how your build will work, and how deploys will work. The purpose of this session is to allow customers to see how to deploy their containerized applications in Amazon Elastic Container Service (ECS) Fargate using our CI/CD solutions. Come with your questions and pain points. We will also talk about how to use Bitbucket as your source control rather than Code Commit for the many customers already using BitBucket and Jenkins.

Aws container services overview

Patricio Vazquez

Containers on AWS: What You Missed at re:Invent 2017 - AWS Online Tech Talks

Amazon Web Services

DEV329_Cisco’s Journey from Monolith to Microservices

Amazon Web Services

Cisco’s video solutions were historically designed for on-premises dedicated hardware deployments. Typically, major releases occurred annually or bi-annually. The release process lacked the ability to absorb frequent changes and adapt to rapid market trends. This session looks into how Cisco’s IVP Solution team evolved a production system from its monolithic design into a microservices platform, leveraging cloud services, automated deployments, and delivery pipelines. Through this transition the team adopted a biweekly deployment cadence. This ultimately enabled a fast-paced migration to an AWS environment, using AWS services such as Amazon EC2, Amazon RDS, and Amazon Elasticsearch Service.

VMware cloud on AWS

Amazon Web Services

Adding the Sec to Your DevOps Pipelines (SEC332-R1) - AWS re:Invent 2018

Amazon Web Services

DevSecOps is the premise that everyone in the software development lifecycle is responsible for security. DevSecOps aims to embed security in every part of the development process. In this workshop, participants explore taking a standard CI/CD pipeline and adding security stages to improve security posture. Learn how to use AWS CodeCommit and AWS CodePipeline to build and publish golden AMI images. Also, learn how to modify pipeline flow to add security test cases. You also have to opportunity to perform CVE analysis and code analysis using Amazon Inspector and perform observational container analysis using Amazon GuardDuty.

Building an Immersive VR Streaming Solution on AWS - AWS Online Tech Talks

Amazon Web Services

Expert Tips for Successful Kubernetes Deployments on AWS

Amazon Web Services

Running and managing a Kubernetes environment at scale can be easier than you think! In this session we will share some expert tips that can help you run production grade Kubernetes deployments on AWS. We will review native AWS services you can use to improve security, deployments, networking, logging and monitoring, and CI/CD. We will also highlight third party services that are popular across a multitude of customers. Speaker: Mitch Beaumont, Solutions Architect, AWS

Similar to One sink to rule them all: Introducing the new Async Sink (20)

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in You...

CON317_Advanced container management at catsndogs.lol

From One to Many: Diving Deeper into Evolving VPC Design (ARC310-R2) - AWS re...

AWS re:Invent serverless recap day: Controls for-managing-events-in-serverles...

Born in the Cloud, Built like a Startup

Running a High-Performance Kubernetes Cluster with Amazon EKS (CON318-R1) - A...

AWS DevDay Berlin - Automating building blocks choices you will face with con...

Advanced Patterns in Microservices Implementation with Amazon ECS - CON402 - ...

CON319_Interstella GTC CICD for Containers on AWS

Interstella 8888: CICD for Containers on AWS - CON319 - re:Invent 2017

Introduction to AWS Fargate & Amazon Elastic Container Service for Kubernetes

AWS DevDay Vienna - Automating building blocks choices you will face with con...

CI/CD with AWS Developer Tools and Fargate

Aws container services overview

Containers on AWS: What You Missed at re:Invent 2017 - AWS Online Tech Talks

DEV329_Cisco’s Journey from Monolith to Microservices

VMware cloud on AWS

Adding the Sec to Your DevOps Pipelines (SEC332-R1) - AWS re:Invent 2018

Building an Immersive VR Streaming Solution on AWS - AWS Online Tech Talks

Expert Tips for Successful Kubernetes Deployments on AWS

More from Flink Forward

The Current State of Table API in 2022

Flink Forward

Flink Forward San Francisco 2022. The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades. by David Andreson

Flink SQL on Pulsar made easy

Flink Forward

Flink Forward San Francisco 2022. Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects: 1. Two different modes of use Pulsar as a metadata store 2. Data format transformation and management 3. SQL semantics support within Pulsar context by Sijie Guo & Neng Lu

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Flink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Welcome to the Flink Community!

Flink Forward

Flink Forward San Francisco 2022. At Flink Forward, we get to hear creative, unique use cases, often on the bleeding edge of some of the most exciting current technologies. This talk will give you a chance to get to open up the hood on our driven and innovative Open Source community. I will cover what our community has been working on this past year, and how this work relates to our (Ververica's) exciting new Flink engineering roadmap! I will also go through some best practices and upcoming opportunities for getting involved in this community! by Caito Scherr

Extending Flink SQL for stream processing use cases

Flink Forward

Flink Forward San Francisco 2022. Apache Flink is a powerful stream processing platform that enables users to build complex real time applications. Flink SQL provides a SQL interface that implements standard SQL. While the standard SQL provides a perfect interface for batch processing, in stream processing context, it can result is ambiguity and complex syntax. As an example, consider these three types of streams: Append-only stream, Retract stream and Upsert stream. Using standard SQL, we would represent all of these streams as Table along with the Table concept in batch processing. Such overloading of concepts can result in ambiguity in SQL statements in streaming context. In this talk, we will present extensions to the Flink SQL that simplify SQL statements in the context of stream processing. We will show how such extensions work in the context of a Flink application using different use cases. These extensions are only sugar syntax and users should be able to use Flink SQL as is if they desire. by Hojjat Jafarpour

The top 3 challenges running multi-tenant Flink at scale

Flink Forward

Apache Flink is the foundation for Decodable's real-time SaaS data platform. Flink runs critical data processing jobs with strong security requirements. In addition, Decodable has to scale to thousands of tenants, power various use cases, provide an intuitive user experience and maintain cost-efficiency. We've learned a lot of lessons while building and maintaining the platform. In this talk, I'll share the top 3 toughest challenges building and operating this platform with Flink, and how we solved them.

Using Queryable State for Fun and Profit

Flink Forward

Flink Forward San Francisco 2022. A particular feature in our system relies on a streaming 90-minute trailing window of 1-minute samples - implemented as a lookaside cache - to speed up a particular query, allowing our customers to rapidly see an overview of their estate. Across our entire customer base, there is a substantial amount of data flowing into this cache - ~1,000,000 entries/second, with the entire cache requiring ~600GB of RAM. The current implementation is simplistic but expensive. In this talk I describe a replacement implementation as a stateful streaming Flink application leveraging Queryable State. This Flink application reduces the net cost by ~90%. In this session, the implementation is described in detail, including windowing considerations, a sliding-window state buffer that avoids the sliding window replication penalty, and a comparison of queryable state and Redis queries. The talk concludes with a frank discussion of when this distinctive approach is, and is not, appropriate. by Ron Crocker

Changelog Stream Processing with Apache Flink

Flink Forward

Flink Forward San Francisco 2022. The world is constantly changing. Data is continuously produced and thus should be consumed in a similar fashion by enterprise systems. Only this enables real-time decisions at scale. Message logs such as Apache Kafka can be found in almost every architecture, while databases and other batch systems still provide the foundation. Change Data Capture (CDC) propagates changes downstream. In this talk, we will highlight what it means to be a general data processor and how Flink can act as an integration hub. We present the current state of Flink and how it can power various use cases on both finite and infinite streams. We demonstrate Flink's SQL engine as a changelog processor that is shipped with an ecosystem tailored to process CDC data and maintain materialized views. We will use Kafka as an upsert log, Debezium for connecting to databases, and enrich streams of various sources. Finally, we will combine Flink's Table API with DataStream API for event-driven applications beyond SQL. by Timo Walther

Large Scale Real Time Fraudulent Web Behavior Detection

Flink Forward

Flink Forward San Francisco 2022. Neuro-ID analyzes web behavior at a large scale to determine visitors' intent on web pages, specifically in the online lending industry. When users interact with an online loan application, our software analyzes their behavior to determine if the applicant may be potentially fraudulent. Lenders can then request various scores describing the applicant's intentions in real-time to use to make decisions during the application flow. Flink gives our product the ability to observe behavior in a stateful manner. As an applicant interacts with an online loan application, a Flink application is used to compare earlier actions to later actions. This processing in Flink can determine the applicant's intent throughout the process of the application. by Jeff Niemann & Randy Hanak

Building Reliable Lakehouses with Apache Flink and Delta Lake

Flink Forward

Flink Forward San Francisco 2022. Apache Flink and Delta Lake together allow you to build the foundation for your data lakehouses by ensuring the reliability of your concurrent streams from processing to the underlying cloud object-store. Together, the Flink/Delta Connector enables you to store data in Delta tables such that you harness Delta’s reliability by providing ACID transactions and scalability while maintaining Flink’s end-to-end exactly-once processing. This ensures that the data from Flink is written to Delta Tables in an idempotent manner such that even if the Flink pipeline is restarted from its checkpoint information, the pipeline will guarantee no data is lost or duplicated thus preserving the exactly-once semantics of Flink. by Scott Sandre & Denny Lee

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Flink Forward

Flink Forward San Francisco 2022. With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi. by Ethan Guo & Kyle Weller

More from Flink Forward (12)

The Current State of Table API in 2022

Flink SQL on Pulsar made easy

Dynamic Rule-based Real-time Market Data Alerts

Exactly-Once Financial Data Processing at Scale with Flink and Pinot

Welcome to the Flink Community!

Extending Flink SQL for stream processing use cases

The top 3 challenges running multi-tenant Flink at scale

Using Queryable State for Fun and Profit

Changelog Stream Processing with Apache Flink

Large Scale Real Time Fraudulent Web Behavior Detection

Building Reliable Lakehouses with Apache Flink and Delta Lake

How to build a streaming Lakehouse with Flink, Kafka, and Hudi

Recently uploaded

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Product School

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Bits & Pixels using AI for Good.........

Alison B. Lowndes

The Future of Platform Engineering

Jemma Hussein Allen

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Recently uploaded (20)

Securing your Kubernetes cluster_ a step-by-step guide to success !

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Monitoring Java Application Security with JDK Tools and JFR Events

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

PCI PIN Basics Webinar from the Controlcase Team

Bits & Pixels using AI for Good.........

The Future of Platform Engineering

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPath Test Automation using UiPath Test Suite series, part 3

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Epistemic Interaction - tuning interfaces to provide information for AI support

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

One sink to rule them all: Introducing the new Async Sink

1. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. One sink to rule them all Introducing the new Async Sink Danny Cranmer Sr. Software Development Engineer, AWS Steffen Hausmann Principal Streaming Architect, AWS

2. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Apache Flink’s connector ecosystem 2 Amazon Kinesis Data Streams Elasticsearch RabbitMQ Google PubSub

3. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. How to build a high quality connector Buffering Send batch requests Retry failed requests Checkpointing Rate limiting and Backpressure

4. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Async Sink internals internal buffer endpoint message request entries batch request response requeue failed requests checkpoint sink to AsyncSinkWriter request entries -> batch request ElementConverter message -> request entry AsyncSinkWriter response -> failed requests AsyncSink WriterStateSerializer 4

5. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. How the Async Sink manages throughput 6 endpoint sink subtask3 sink subtask2 sink subtask1 1. Limit the number of in-flight request entries 2. If a batch request was successful, increase the limit by a constant factor 3. If a batch request failed, cut the limit in half Additive Increase/Multiplicative Decrease

7. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Async Sink configuration endpoint message request entries batch request sink to maxRecordSizeInBytes 8 maxInFlightRequests maxBatchSize maxBatchSizeInBytes maxBufferedRequests maxTimeInBufferMs internal buffer

9. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. What’s Next? FLIP-242 Configurable Rate Limiting FLIP-252 Amazon DynamoDB Sink Wider Community Adoption

10. ONE SINK TO RULE THEM ALL: INTRODUCING THE NEW ASYNC SINK © 2022, Amazon Web Services, Inc. or its affiliates. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. Danny Cranmer Steffen Hausmann

One sink to rule them all: Introducing the new Async Sink

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to One sink to rule them all: Introducing the new Async Sink

Similar to One sink to rule them all: Introducing the new Async Sink (20)

More from Flink Forward

More from Flink Forward (12)

Recently uploaded

Recently uploaded (20)

One sink to rule them all: Introducing the new Async Sink