Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022

•

0 likes•366 views

Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022 Apache Kafka is well known as a low-latency, high-throughput and highly configurable streaming platform. At AWS, we run thousands of Kafka clusters, each cluster with different hardware and software configurations. Managing such a large and diverse Kafka fleet has taught us several operational lessons. We would like to share some of these lessons with you. We’ll talk about several topics including (a) monitoring Kafka health, (b) optimizing Kafka to address compute, storage and networking bottlenecks, (c) automating detection and mitigation of infrastructure failures related to compute, storage and networking and (d) continuous software patching.

Technology

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
© 2022, Amazon Web Services, Inc. or its affiliates.
Kafka on AWS:
Best Practices
Lessons learned from operating
thousands of clusters
Mehari Beyene (he/him)
T U E S D A Y , O C T O B E R 4
Sr. Software Dev Engineer
AWS
Tom Schutte (he/him)
Software Dev Engineer
AWS

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Speakers
2
Tom Schutte
Software Engineer
Amazon MSK
Mehari Beyene
Senior Software Engineer
Amazon MSK

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Data is everything - everything is data
• 2.5 Million Terabytes of data is generated everyday
• Thousands of Terabytes streamed each day
• Latest data insights are critical
• Used by over 75% of Fortune 100 companies
• Hundreds of data streaming use cases
• Data streaming is still early days…
3

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Amazon Managed Streaming for Apache Kafka (MSK)
• Offers open source Apache Kafka as
a service to customers
• Customers Can Create, Scale and
Upgrade Kafka clusters
• The MSK team monitors the health
of clusters and mitigate cluster
health problems
• The MSK team periodically update
software, apply patches and make
sure that clusters are healthy and
secure
5

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Cluster Health Metrics to Monitor
Kafka & Zookeeper Metrics
• JMX metrics emitted by Kafka
& Zookeeper
Host Level Metrics
• CPU
• Memory
• Disk Usage
• Network Connectivity
Metrics from Agents
• Agents heartbeat
• Healthy/Unhealthy
7

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Challenges of monitoring at scale
• Flexibility of alarming
• Aggregate system health
• Prevent large issues from obscuring
• Cluster and Node level monitoring
• … Automate!
8

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Amazon MSK’s Monitoring Architecture
• Stream metrics from each node
• Ingest records into a Flink application
• Filter metrics of interest
• Tune the sensitivity of each alarm
• Record health state information
• Take action!
9

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Failure Modes
Compute
• Degraded Hardware
• High Memory usage
• Overloaded CPU
Storage
• Disk Full
• Slow or Stuck disk
• Corrupted disk
Networking
• Inaccessible Network
interfaces
• Slow Dns Propagation
• Data Center Outages
11

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Challenges of Automated Mitigation
• Heterogeneity of Fleet
• Node types
• Kafka Versions
• Customized configurations and features
• Recovery from large scale events
12

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Automated Mitigations
• Terminate and Replace Nodes
• Restart Nodes
• Detach/Attach Volumes
• Replace Volumes
• Restart/Update Software
13

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Patching
Regularly Patch
• Operating System
• Kafka/Zookeeper Software
• Agents
Challenges
• Cluster availability
• Heterogeneous Fleet
• Zero Day Vulnerabilities
14

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Amazon MSK Patching Tenets
• Update all software running on Clusters
• No impact to Cluster availability
• Should be done regularly
• Fast enough to patch an entire fleet and
slow enough not to disrupt Cluster
availability
15

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Update Dimensions
Compute
• Node Type
• Number of Brokers
Storage
• Increase disk size
• Auto Scaling
• Provisioned throughput
Connectivity
• Authentication and Encryption
• Public end points
17

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Amazon MSK Update Tenets
• Guardrails for stable updates
• Safe and controlled – rolling restart, monitoring, automated
mitigation
• Speed matters
18

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
• Scalable monitoring and alarming system
• Automated detection and mitigation
• Regular and continuous patching
• Controlled mutation of clusters
19

© 2022, Amazon Web Services, Inc. or its affiliates.
LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS
Thank you!
© 2022, Amazon Web Services, Inc. or its affiliates.
Mehari Beyene
mehbey@amazon.com
Tom Schutte
tomschu@amazon.com

Similar to Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022

How can you accelerate the delivery of new, high-quality services? How can you be able to experiment and get feedback quickly from your customers? To get the most out of the agility afforded by serverless and containers, it is essential to build CI/CD pipelines that help teams iterate on code and quickly release features. In this talk, we demonstrate how developers can build effective CI/CD release workflows to manage their serverless or containerized deployments on AWS. We cover infrastructure-as-code (IaC) application models, such as AWS Serverless Application Model (AWS SAM) and new imperative IaC tools. We also demonstrate how to set up CI/CD release pipelines with AWS CodePipeline and AWS CodeBuild, and we show you how to automate safer deployments with AWS CodeDeploy.

CICDforModernApplications_Stockholm.pdf

Amazon Web Services

Apache Kafka is a popular, open-source technology for collecting, processing, and analyzing streaming data in real-time. Amazon MSK is a fully managed service that removes the complexities of managing Kafka clusters so that you can focus on building real-time applications. In this session, we provide an overview of Amazon MSK and then discuss how to get started. We then look at some best practices and top tips, and walk through how to decide whether to choose Amazon MSK, Amazon Kinesis, or a mix of both to address your data streaming use cases.

A deep dive into Amazon MSK - ADB206 - Chicago AWS Summit

Amazon Web Services

AWS DevDay Berlin - Automating building blocks choices you will face with con...

Cobus Bernard

Building Modern Applications on AWS.pptx

Nelson Kimathi

Rapid growth of technology and tooling in the cloud has enabled us to build modern applications that are more secure, scalable, and focused on our business. In this session, we cover the key compute primitives that enable us to accelerate towards building and running modern, cloud-native applications. We highlight what we’ve learned from customers running applications with AWS Lambda and AWS Fargate, two modern compute technologies for running applications in the cloud. In addition, we cover architecture patterns of modern application, key primitives required for building modern systems, steps you can take to start building and monitoring modern applications today, and secrets to fearlessly going faster and farther in the cloud.

SRV205 Architectures and Strategies for Building Modern Applications on AWS

Amazon Web Services

kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...

kreuzwerker GmbH

This workshop will provide practical advice and guidance for designing and building secure Amazon Virtual Private Clouds (VPCs). Using a hands-on approach, we'll take you through Amazon VPC features such as subnets, security groups, network ACLs, routing, flow logs and service endpoints. The AWS team will also provide some guidance around best practices for VPC design and management, based on our experience of supporting customers running large-scale infrastructures.

NET309_Best Practices for Securing an Amazon Virtual Private Cloud

Amazon Web Services

Running kubernetes with amazon eks

yanaisama

Designed for cloud-native applications, GitOps makes developers more productive while improving application stability, security, and compliance. The decentralized nature of cloud infrastructure has significantly increased the threat landscape; thus raising many security issues and concerns many companies grapple with. Trusted delivery allows organizations to build and run containerized artifacts that are guaranteed while image generation pipelines are key to a secure environment. These features simplify the shifting left of security and accelerate the adoption of DevSecOps practices. With automated continuous security and compliance checks, in the form of policy as code, fully automated deployments are now possible. In this Webinar, you will learn: Security challenges of cloud-native infrastructure How GitOps help eliminate risks associated with continuous deployment How to leverage policy as code for environments & application artifacts How to produce trustworthy deployment images

Trusted Application Delivery: Achieving Ultimate Security

Weaveworks

AWS DevDay Cologne - Automating building blocks choices you will face with co...

Cobus Bernard

쿠버네티스에 어플리케이션을 손쉽게 배포하는 방법은 무엇일까요? 복잡하게 배포된 어플리케이션의 파드들은 어떻게 디버깅하고 로깅해야 할까요? 또한 요즘 자주 이야기 되는 클라우드 네이티브 아키텍처로 설계된 어플리케이션은 어떻게 만들고 배포해야하는 걸까요?삼성전자 무선사업부에서 삼성헬스를 EKS 에 배포한 사례를 살펴보며, 이러한 문제를 어떻게 해결했는지 알아봅니다. 또한 복잡하게만 느껴졌던 쿠버네티스의 어플리케이션 배포와 클라우드 네이티브 아키텍처의 베스트 프렉티스를 EKS 에 어플리케이션을 배포하고, 관리하는 예제를 통하여 간편하게 이해할 수 있게 도와드립니다.

[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...

Amazon Web Services Korea

Discover how to architect fully available and scalable Microsoft solutions and environments on AWS. Find out how Microsoft solutions can work alongside various AWS services to boost resiliency, simplify architecture, provide scalability, and introduce DevOps concepts, such as compliance, governance, automation, and repeatability. Also, learn about authentication and authorization, and explore various hybrid scenarios involving on-premises solutions or infrastructure. Examine common architecture patterns for network design, Microsoft Active Directory, and business productivity; as well as common scenarios for custom .NET, .NET Core with Microsoft SQL deployments, and migrations. Artur Rodrigues, Senior Solutions Architect, Amazon Web Services

Simplifying Microsoft Architectures with AWS Services

Amazon Web Services

SRV313 Introduction to Building Web Apps on AWS

Amazon Web Services

Nesta sessão, aprenda como fazer a arquitetura de soluções e ambientes Microsoft totalmente disponíveis e escaláveis na AWS. Descubra como as soluções da Microsoft podem alavancar diversos serviços AWS para conseguir maior resiliência, substituir complexidades desnecessárias, simplificar a arquitetura, fornecer escalabilidade e apresentar conceitos de DevOps como compliance, governança, automação e replicabilidade. Além disso, planeje autenticação e autorização e explore vários cenários híbridos com outros ambientes de nuvem e soluções ou infraestrutura on-premise . Saiba mais sobre padrões comuns de arquitetura para design de rede, Microsoft Active Directory e soluções de produtividade de negócios como Microsoft Dynamics AX, CRM e Microsoft SharePoint, bem como os cenários mais comuns de customização .NET, .NET Core com a implementação e migrações Microsoft SQL.

Simplificando Arquiteturas Microsoft com os Serviços da AWS - ARC204 - Sao P...

Amazon Web Services

Come learn how Elastic Beanstalk can help you go from code to running application in a matter of minutes, without the need to provision or manage any of the underlying Amazon Web Services (AWS) resources. Hear how Qualcomm is able to migrate application to AWS faster than before through Forge, an internally built application platform that leverages Elastic Beanstalk to simplify the development and deployment of applications to AWS with security and organizational best practices out of the box.

PaaS – From Code to Running Application using AWS Elastic Beanstalk (DEV323) ...

Amazon Web Services

Apache Kafka is a popular stream-processing platform, but it’s no secret that it can be tough to set up, manage, and scale. Amazon Managed Streaming for Kafka (Amazon MSK) can help remove some of that toil for you. In this session, you learn about new Amazon MSK features and capabilities. You also get a glimpse under the hood, giving you a better understanding of how Amazon MSK operationalizes Apache Kafka so you don't have to. We compare and contrast Amazon Kinesis Data Streams and Apache Kafka (with/without MSK) and show how to lift-and-shift your workload into Amazon MSK with minimal downtime.

Ditching the overhead - Moving Apache Kafka workloads into Amazon MSK - ADB30...

Amazon Web Services

Frome Code to Cloud: Exploring AWS CDK for Infrastructure Management

Sujay Pillai

Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you! We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance. By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies.

Control Planes on Kubernetes and Policy Validation

Carlos Santana

Presented at All Things Open 2023 Presented by Carlos Santana - AWS Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you! We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance. By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

All Things Open

Amazon EKS makes it easy to run Kubernetes on AWS without managing master nodes or etcd operators. Kubernetes offers a powerful abstraction layer for managing containerized infrastructure, which presents unique challenges to AWS media customers. In this session, we share lessons from Synamedia, and we discuss its reasons for moving to EKS and the security and governance implications for migrating workloads. Learn about the approach and benefits for establishing security and governance with Open Policy Agent (OPA), which uses Kubernetes validating and mutating admission controllers to establish policy guardrails for container registries, input, load balancers, and other objects within EKS.

Architecting security and governance through policy guardrails in Amazon EKS ...

Amazon Web Services

Similar to Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022 (20)

CICDforModernApplications_Stockholm.pdf

A deep dive into Amazon MSK - ADB206 - Chicago AWS Summit

AWS DevDay Berlin - Automating building blocks choices you will face with con...

Building Modern Applications on AWS.pptx

SRV205 Architectures and Strategies for Building Modern Applications on AWS

kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...

NET309_Best Practices for Securing an Amazon Virtual Private Cloud

Running kubernetes with amazon eks

Trusted Application Delivery: Achieving Ultimate Security

AWS DevDay Cologne - Automating building blocks choices you will face with co...

[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...

Simplifying Microsoft Architectures with AWS Services

SRV313 Introduction to Building Web Apps on AWS

Simplificando Arquiteturas Microsoft com os Serviços da AWS - ARC204 - Sao P...

PaaS – From Code to Running Application using AWS Elastic Beanstalk (DEV323) ...

Ditching the overhead - Moving Apache Kafka workloads into Amazon MSK - ADB30...

Frome Code to Cloud: Exploring AWS CDK for Infrastructure Management

Control Planes on Kubernetes and Policy Validation

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Architecting security and governance through policy guardrails in Amazon EKS ...

More from HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

ADP Passwordless Journey Case Study.pptx

FIDO Alliance

Generative AI refers to a class of machine learning algorithms that are designed to generate new data samples that are similar to those in the training data. Unlike traditional AI models that are trained to recognize patterns and make predictions, generative AI models have the ability to create entirely new data based on the patterns they have learned. This is achieved through techniques such as generative adversarial networks (GANs), variational autoencoders (VAEs), and transformer architectures, among others.

Generative AI Use Cases and Applications.pdf

alexjohnson7307

The presentation was made in “Web3 Fusion: Embracing AI and Beyond” is more than a conference; it's a journey into the heart of digital transformation. The conference a provided a platform where the future of technology meets practical application. This three-day hybrid event, set in the heart of innovation, served as a gateway to the latest trends and transformative discussions in AI, Blockchain, IoT, AR/VR, and their collective impact on the information space.

AI in Action: Real World Use Cases by Anitaraj

AnitaRaj43

Overview of Hyperledger Foundation

Hyperleger Tokyo Meetup

In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.

The Zero-ETL Approach: Enhancing Data Agility and Insight

Safe Software

At Skynet Technologies, our team of accessibility experts performs automated, semi-automated, and manual audits of websites and web applications as per WCAG 2.2 level AA, ADA, and section 508. Based on evaluations of the accessibility compliance level of the website’s UI, design, source code, navigation, interactive elements, and overall usability, we will provide a digital accessibility evaluation report with in-depth details of potential accessibility barriers and remediation recommendations. Get a manual website WCAG audit (2.0, 2.1, 2.2 level AA) for a small website: 10 pages: $2,500 within 7 business days 30 pages: $7,500 within 14 business days 50 pages: $12,500 within 28 business days For medium websites: 100 pages: $25,000 within 70 business days For larger websites or audits of all pages, please reach out hello@skynettechnologies.com.

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...

Skynet Technologies

State of the Smart Building Startup Landscape 2024!

Memoori

Portal Kombat : extension du réseau de propagande russe

中央社

Microsoft CSP Briefing Pre-Engagement - Questionnaire

Exakis Nelite

AI mind or machine power point presentation

yogeshlabana357357

Simplifying Mobile A11y Presentation.pptx

MarkSteadman7

Explore the latest trends and insights on JavaScript usage with Pixlogix's informative blog. Discover key statistics and facts about JavaScript's role in web development, its popularity among developers, and its impact on modern websites. Stay updated with the evolving landscape of JavaScript frameworks and libraries, and learn how they're shaping the future of web development. Gain valuable insights to enhance your JavaScript skills and stay ahead in the digital realm.

JavaScript Usage Statistics 2024 - The Ultimate Guide

Pixlogix Infotech

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

FIDO Alliance

CORS (Kitworks Team Study 양다윗 발표자료 240510)

Wonjun Hwang

Design and Development of a Provenance Capture Platform for Data Science

Paolo Missier

Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots

Leah Henrickson

Working together SRE & Platform Engineering

Marcus Vechiato

Vector Search @ sw2con for slideshare.pptx

jbellis

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

ScyllaDB

"The Ultimate Prompt Engineering Guide for Generative AI" provides a comprehensive guide to leveraging the power of AI assistants through effective prompt design. It explores fundamental prompt concepts and details strategies for crafting prompts that maximize output quality. Readers learn about iterative refinement, examples, constraints, and advanced techniques like chaining and decomposition. Case studies demonstrate real-world applications in content creation, coding, analysis, and more. Trends in multimodal, automated, and responsible prompting are also examined. This book is a must-read for anyone seeking to optimize generative AI capabilities.

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

SOFTTECHHUB

Recently uploaded (20)

ADP Passwordless Journey Case Study.pptx

Generative AI Use Cases and Applications.pdf

AI in Action: Real World Use Cases by Anitaraj

Overview of Hyperledger Foundation

The Zero-ETL Approach: Enhancing Data Agility and Insight

Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...

State of the Smart Building Startup Landscape 2024!

Portal Kombat : extension du réseau de propagande russe

Microsoft CSP Briefing Pre-Engagement - Questionnaire

AI mind or machine power point presentation

Simplifying Mobile A11y Presentation.pptx

JavaScript Usage Statistics 2024 - The Ultimate Guide

Hyatt driving innovation and exceptional customer experiences with FIDO passw...

CORS (Kitworks Team Study 양다윗 발표자료 240510)

Design and Development of a Provenance Capture Platform for Data Science

Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots

Working together SRE & Platform Engineering

Vector Search @ sw2con for slideshare.pptx

Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...

The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...

Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022

1. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS © 2022, Amazon Web Services, Inc. or its affiliates. Kafka on AWS: Best Practices Lessons learned from operating thousands of clusters Mehari Beyene (he/him) T U E S D A Y , O C T O B E R 4 Sr. Software Dev Engineer AWS Tom Schutte (he/him) Software Dev Engineer AWS

2. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Speakers 2 Tom Schutte Software Engineer Amazon MSK Mehari Beyene Senior Software Engineer Amazon MSK

3. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Data is everything - everything is data • 2.5 Million Terabytes of data is generated everyday • Thousands of Terabytes streamed each day • Latest data insights are critical • Used by over 75% of Fortune 100 companies • Hundreds of data streaming use cases • Data streaming is still early days… 3

4. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS © 2022, Amazon Web Services, Inc. or its affiliates. Amazon Managed Streaming for Apache Kafka (MSK) 4

5. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Amazon Managed Streaming for Apache Kafka (MSK) • Offers open source Apache Kafka as a service to customers • Customers Can Create, Scale and Upgrade Kafka clusters • The MSK team monitors the health of clusters and mitigate cluster health problems • The MSK team periodically update software, apply patches and make sure that clusters are healthy and secure 5

6. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS © 2022, Amazon Web Services, Inc. or its affiliates. Monitoring Kafka Clusters at scale 6

7. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Cluster Health Metrics to Monitor Kafka & Zookeeper Metrics • JMX metrics emitted by Kafka & Zookeeper Host Level Metrics • CPU • Memory • Disk Usage • Network Connectivity Metrics from Agents • Agents heartbeat • Healthy/Unhealthy 7

8. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Challenges of monitoring at scale • Flexibility of alarming • Aggregate system health • Prevent large issues from obscuring • Cluster and Node level monitoring • … Automate! 8

9. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Amazon MSK’s Monitoring Architecture • Stream metrics from each node • Ingest records into a Flink application • Filter metrics of interest • Tune the sensitivity of each alarm • Record health state information • Take action! 9

10. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS © 2022, Amazon Web Services, Inc. or its affiliates. Automated Mitigation at scale 10

11. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Failure Modes Compute • Degraded Hardware • High Memory usage • Overloaded CPU Storage • Disk Full • Slow or Stuck disk • Corrupted disk Networking • Inaccessible Network interfaces • Slow Dns Propagation • Data Center Outages 11

12. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Challenges of Automated Mitigation • Heterogeneity of Fleet • Node types • Kafka Versions • Customized configurations and features • Recovery from large scale events 12

13. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Automated Mitigations • Terminate and Replace Nodes • Restart Nodes • Detach/Attach Volumes • Replace Volumes • Restart/Update Software 13

14. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Patching Regularly Patch • Operating System • Kafka/Zookeeper Software • Agents Challenges • Cluster availability • Heterogeneous Fleet • Zero Day Vulnerabilities 14

15. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Amazon MSK Patching Tenets • Update all software running on Clusters • No impact to Cluster availability • Should be done regularly • Fast enough to patch an entire fleet and slow enough not to disrupt Cluster availability 15

17. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Update Dimensions Compute • Node Type • Number of Brokers Storage • Increase disk size • Auto Scaling • Provisioned throughput Connectivity • Authentication and Encryption • Public end points 17

18. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Amazon MSK Update Tenets • Guardrails for stable updates • Safe and controlled – rolling restart, monitoring, automated mitigation • Speed matters 18

19. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS • Scalable monitoring and alarming system • Automated detection and mitigation • Regular and continuous patching • Controlled mutation of clusters 19

20. © 2022, Amazon Web Services, Inc. or its affiliates. LESSONS LEARNED FROM RUNNING THOUSANDS OF KAFKA CLUSTERS ON AWS Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. Mehari Beyene mehbey@amazon.com Tom Schutte tomschu@amazon.com

Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022

Recommended

Recommended

More Related Content

Similar to Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022

Similar to Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Running Thousands of Kafka Clusters on AWS With Mehari Beyene and Tom Schutte | Current 2022