Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020

•

0 likes•2,401 views

What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with. In this talk, we’ll be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We’ll need to take care of the consumers as well to ensure that they don’t process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters. We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.

Technology

Presentation title
ANDREY FALKO and CAN CECEN - August 2020
Can Apache Kafka®
Handle a Lyft Ride?

Agenda
Who Are We?
Rides Without Kafka
Rides With PubSub Architecture + Demo
Hardened PubSub Architecture + Demo
Future Work

Who Are We?
Andrey Falko
● Launched Lyft’s internal Kafka platform
● Enjoys hiking, skiing, and kayaking
● Discovered that he is a cat person
Can Cecen
● Launched MSK and now taking Lyft’s
Kafka platform to the next level
● Outside coding, uses fingers to play
bass guitar
● Has a dog, but she’s more like a cat.
Dogcat person?

Persist ride request quickly
Transition states correctly
Get feedback to user quickly
Eﬃciency
Requirements Summary

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
Queue Consumer 1
Consumer 2
Consumer 3

Queueing vs. Publish/Subscribe
1 2 3 4 5 6
Queue Consumer 1 7
Consumer 2
Consumer 3

Queueing vs. Publish/Subscribe
1 2 3 4 5
Queue Consumer 1 7
Consumer 2 6
Consumer 3

Queueing vs. Publish/Subscribe
1 2 3 4
Queue Consumer 1 7
Consumer 2 6
Consumer 3 5

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
PubSub - Immutable Log Consumer Group 1
Consumer Group 2
Consumer Group 3

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
PubSub - Immutable Log Consumer Group 1 7
Consumer Group 2 7
Consumer Group 3 7

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
PubSub - Immutable Log Consumer Group 1 5,6,7
Consumer Group 2 7
Consumer Group 3 7
Oﬀset for Consumer Group 1

Queueing vs. Publish/Subscribe
Queue PubSub
Single Subscriber Multi-subscriber
Non-replayable Replayable

Languages
Python
Golang
Java
Ruby
Onboarding time
How can we simplify?
Non-homogenous Kafka Clients

Simplify conﬁguration
Inherit defaults
Homogenous metrics
Wrapper per programming language
Metrics namespace
Better Client Experience

Security Updates
Need to update Linux Kernel
Network, Host, and Operator Errors
Kafka Version Upgrades
Disaster Recovery
Kafka Operations - Restarts

Make switch from Kafka’s ZNodes
Reuse remediator <https://lft.to/305FyTg>
Fast Cluster Switch

Envoy as Kafka Proxy
KIP-559 - Make it easier to intercept Kafka protocol traﬃc
Envoy gives us:
Observability
Fault injection and a framework to implement health
checks
Service Discovery

Distributed logs are good for state machines
Take higher p50 to gain lower p99 latency
Invest in automation to support multi-cluster Kafka
Takeaways

Thank You
Thank you!
github.com/afalko
Andrey Falko <afalko@lyft.com>
linkedin.com/in/andrey-falko/
github.com/cancecen
Can Cecen <ccecen@lyft.com>
linkedin.com/in/yilmazcancecen
https://bit.ly/32ufowH

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
I’m Svc 1 at 10.1.1.10 I’m Svc 2 at 10.1.1.20

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
localhost:svc-port

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
localhost:svc-port
Svc 2 at 10.1.1.20

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
localhost:svc-port
10.1.1.20:svc-port

What about Kafka with Envoy?
Client-side can handle bootstrap
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy

What about Kafka with Envoy?
Client-side can handle bootstrap
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy
localhost:9092
Kafka at 10.1.1.20,
10.1.1.21, 10.1.123
10.1.1.21:9092

What about Kafka with Envoy?
Client-side can handle bootstrap
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy
Cluster Metadata
Kafka at 10.1.1.20,
10.1.1.21, 10.1.123
Cluster Metadata
Cluster
Metadata

What about Kafka with Envoy?
Model breaks: client-side Envoy bypassed
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy
Write to topic
example-0 w/ leader:
10.1.1.23:9092

What about Kafka with Envoy?
Server-side 2x connections
Kafka
Kubelets
EnvoyWrite to topic
example-0 w/ leader:
10.1.1.23:9092
Client
Envoy

Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale. Most known for its excellent performance, low latency, fault tolerance, and high throughput, it's capable of handling thousands of messages per second. For mission-critical applications, how do you ensure that the performance delivered is the performance required? This is especially important as Kafka is written in Java and Scala and runs on the JVM. The JVM is a fantastic platform that delivers on an internet scale. In this session, we'll explore how making changes to the JVM design can eliminate the problems of garbage collection pauses and raise the throughput of applications. For cloud-based Kafka applications, this can deliver both lower latency and reduced infrastructure costs. All without changing a line of code!

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

HostedbyConfluent

Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 200M messages per second. In this talk, we will share the learning and challenges in our journey moving to Apache Kafka.

Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs ...

HostedbyConfluent

In our payments platform at Goldman Sachs Transaction Banking, Apache Kafka plays a critical role as the messaging bus in our micro-services architecture. Being a part of the financial service industry we need to ensure high-availability of our platform and quick response time during failures. In this talk we will explore how we monitor and alert on the health of our Kafka clusters using our heartbeat application and clients using DataDog dashboards. We will see how we consolidate JMX metrics such as error-rates, connection-rates, latencies and consumer lag from all producers and consumers using JMX agent sidecar to provide a live view of the health of our entire infrastructure. We will also discuss our culture of game days where we regularly test the resiliency of all the clients in our infrastructure by simulating various failure scenarios to improve the overall availability of our infrastructure.

Error Resilient Design: Building Scalable & Fault-Tolerant Microservices with...

HostedbyConfluent

Handling failures is important, but it’s a must have when a product handles sensitive data. It also becomes exponentially harder in the world of microservices, since a failure can happen in any of the services and even in their dependencies. One of the go-to solutions when handling errors is to simply “Retry”. However, in a complex system, A “Retrying” mechanism must be Smart, Customisable, Pluggable, and Persistent. In this talk, we'll discuss what capabilities a ""GOOD"" retrying mechanism should have, how I implemented a Kafka-based retrying solution that answers those capabilities, can handle a large and diverse set of errors and why Kafka is a good match for such a solution.

Everything you ever needed to know about Kafka on Kubernetes but were afraid ...

HostedbyConfluent

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.

Asynchronous Transaction Processing With Kafka as a Single Source of Truth - ...

HostedbyConfluent

This document discusses using Kafka as a central backbone for building an asynchronous mobile ticketing application. It describes moving from a monolithic backend to a microservices architecture with separate databases for each service. It then outlines using Kafka to facilitate asynchronous communication between services, allowing for easy scaling, distribution, and evolution. Transaction management is achieved through message ordering in Kafka and a SAGA pattern. Leveraging Kafka Streams enables materialized views without a separate database. Overall this approach provides scalability, distribution, ease of evolution, and an improved user experience.

Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field

confluent

This document discusses best practices for using Apache Kafka Connect. It begins with an overview of Kafka Connect basics like connectors, converters, transforms, and plugins. It then discusses choosing the right connectors for different data sources and sinks, and how to test connectors using the Confluent CLI. The document concludes with recommendations for planning Kafka Connect deployments, such as understanding schemas, deploying connectors across workers, tuning configurations, and minimizing rebalances.

A Modern C++ Kafka API | Kenneth Jia, Morgan Stanley

HostedbyConfluent

The document discusses a new C++ Kafka API called modern-cpp-kafka that was developed to address requirements for a pub/sub messaging system. It provides examples of how the API simplifies and improves upon an existing C++ Kafka client (librdkafka) for key tasks like producing and consuming messages. The modern-cpp-kafka API matches the Java API naming, uses RAII for lifetime management, and hides polling and queue details. It has led to improved throughput of over 26% for an application. The API is now open source and the team is working to expand it further.

This document introduces KSQL, a streaming SQL engine for Apache Kafka. It provides concise summaries of KSQL's capabilities and how to use it in 3 sentences or less: KSQL allows users to easily query and transform data in Kafka streams using SQL-like queries. It provides simplicity, flexibility, and scalability compared to directly using Kafka Streams APIs. KSQL can be run in standalone, client-server, or application modes and is well-suited for tasks like streaming ETL, anomaly detection, monitoring, and IoT data processing.

Apache Pulsar: A Foundation Backbone for Clever Cloud - Pulsar Virtual Summit...

StreamNative

As a Cloud Service Provider, Clever Cloud makes heavy use of messaging in its core architecture. Initially using other messaging solutions, we faced a number of different challenges. Then we discovered Apache Pulsar, which held the promise to solve these challenges. This session will dive into the adoption of Apache Pulsar at Clever Cloud, including how we worked with StreamNative to develop KoP (Kafka on Pulsar), and how we're using Apache Pulsar as the foundation for our cloud backbone.

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

confluent

Cloud migration: it's practically a rite of passage for anyone who's built infrastructure on bare metal. When we migrated our 5-year-old Kafka deployment from the datacenter to GCP, we were faced with the task of making our highly mutable server infrastructure more cloud-friendly. This led to a surprising decision: we chose to run our Kafka cluster on Kubernetes. I'll share war stories from our Kafka migration journey, explain why we chose Kubernetes over arguably simpler options like GCP VMs, and present the lessons we learned while making our way toward a stable and self-healing Kubernetes deployment. I'll also go through some improvements in the more recent Kafka releases that make upgrades crucial for any Kafka deployment on immutable and ephemeral infrastructure. You'll learn what happens when you try to run one complex distributed system on top of another, and come away with some handy tricks for automating cloud cluster management, plus some migration pitfalls to avoid. And if you're not sure whether running Kafka on Kubernetes is right for you, our experiences should provide some extra data points that you can use as you make that decision.

Introducing Exactly Once Semantics To Apache Kafka

Apurva Mehta

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

confluent

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov,...

confluent

When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer - Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk together with Michael Ng from Confluent

Friends don't let friends do dual writes: Outbox pattern with OpenShift Strea...

Red Hat Developers

Dual writes are a common source of issues in distributed event-driven applications. A dual write occurs when an application has to change data in two different systems - for instance, when an application needs to persist data in the database and send a Kafka message to notify other systems. If one of these two operations fail, you might end up with inconsistent data which can be hard to detect and fix. OpenShift Streams for Apache Kafka is Red Hat's fully hosted and managed Apache Kafka service targeting development teams that want to incorporate streaming data and scalable messaging in their applications, without the burden of setting up and maintaining a Kafka cluster infrastructure. Debezium is an open source distributed platform for change data capture. Built on top of Apache Kafka, it allows applications to react to inserts, updates, and deletes in your databases. In this session you will learn how you can leverage OpenShift Streams for Apache Kafka and Debezium to avoid the dual write issue in an event-driven application using the outbox pattern. More specifically, we will show you how to: Provision a Kafka cluster on OpenShift Streams for Apache Kafka. Deploy and configure Debezium to use OpenShift Streams for Apache Kafka. Refactor an application to leverage Debezium and OpenShift Streams for Apache Kafka to avoid the dual write problem.

Running large scale Kafka upgrades at Yelp (Manpreet Singh,Yelp) Kafka Summit...

confluent

Over the years at Yelp, we have relied on Kafka to build many complex applications and stream processing data-pipelines that solve a multitude of use cases, including powering our product experimentation workflow, search indexing, asynchronous task processing and more. Today, Kafka is at the core of our infrastructure. These applications use different versions of Kafka clients and different programming languages.To fulfill the requirements of these diverse use cases, we run several specialized Kafka clusters for high-availability, consistency, exactly-once and infinite retention. We endeavor to keep our clusters up-to-date with newer Kafka versions that bring with them several critical bug fixes and exciting features like dynamic broker configuration, exactly-once semantics, kafka offset management and improved tooling. Our journey with Kafka started with version 0.8.2.0. Upgrading Kafka while ensuring client compatibility, zero-downtime, negligible performance degradation across our ever-growing multi-regional cluster deployment exposed us to a plethora of unique challenges. This session will focus on the challenges we encountered and how we evolved our infrastructure tooling and upgrade strategy to overcome them. I will be talking about: -- How we rolled out new features such as kafka offset storage, message timestamp, reassignment auto-throttling, etc. -- Core technical issues discovered during upgrades such as failure of log cleaners due to large offsets while upgrading. -- The in-house test-suite that we built in order to: validate new kafka versions against our existing tooling and client-libraries, exercise the upgrade and rollback process and benchmark performance. -- The automation we built for safe and fast rolling upgrades and broker configuration deployment.

Westpac Bank Tech Talk 1: Dive into Apache Kafka

confluent

Kafka aws

Ariel Moskovich

This document discusses building a fault-tolerant Kafka cluster on AWS to handle 2.5 billion requests per day. It covers choosing AWS instance types and broker counts, spreading brokers across availability zones, configuring replication and partitioning, automating fault tolerance, adding metrics and alerts, and testing the cluster's resilience. Key decisions include broker placement, topic partitioning, Zookeeper ensemble sizing, and automation to dynamically reassign partitions and change configurations in response to failures or added capacity.

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Paul Brebner

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

Introduction to Apache Kafka

Shiao-An Yuan

This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.

Apache Kafka - Martin Podval

Martin Podval

Apache Kafka is a distributed messaging system that allows for publishing and subscribing to streams of records, known as topics, in a fault-tolerant and scalable way. It is used for building real-time data pipelines and streaming apps. Producers write data to topics which are committed to disks across partitions and replicated for fault tolerance. Consumers read data from topics in a decoupled manner based on offsets. Kafka can process streaming data in real-time and at large volumes with low latency and high throughput.

Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul

HostedbyConfluent

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

HostedbyConfluent

Confluent Cloud runs a modified version of Apache Kafka - redesigned to be cloud-native and deliver a serverless user experience. In this talk, we will discuss key improvements we've made to Kafka and how they contribute to Confluent Cloud availability, elasticity, and multi-tenancy. You'll learn about innovations that you can use on-prem, and everything you need to make the most of Confluent Cloud.

Netflix Data Pipeline With Kafka

Steven Wu

This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

confluent

This document provides guidance on upgrading Kafka. It emphasizes the importance of upgrading early and often to the latest bugfix release in order to address security vulnerabilities and other bugs. It recommends using automated rolling upgrades to upgrade brokers one by one with zero downtime. It also outlines best practices like backing up configurations, reading release notes, and ensuring protocol compatibility when upgrading.

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

HostedbyConfluent

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

HostedbyConfluent

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including: * How to manage multi-tenant clusters in a hybrid environment * High volume data pipelines with Mirus replicating data to Kafka and blob storage * Kafka Fault Injection Framework built on Trogdor and Kibosh * Automated recovery without data loss * Using Envoy as an SNI-routing Kafka gateway We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...

HostedbyConfluent

- Upgrades should be done often to get bug fixes and improvements, following the upgrade guide carefully. Start with a healthy cluster and upgrade components outward from Zookeeper to Kafka brokers to clients. Don't rush the process or have any unresolved partition reassignments. - Collect JMX metrics to monitor the cluster as outages can be prolonged without visibility. The Kafka defaults are suitable for single node deployments but replication factor, threads, and broker configuration should be tuned for larger clusters. - Quotas like replication throttling and bandwidth/request limits per client or topic should be used to protect the cluster and clients. Log files should separate each component and be retained for a few days. Consider multiple clusters by SLA

Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...

Natan Silnitsky

At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar.

Spark streaming + kafka 0.10

Joan Viladrosa Riera

Spark Streaming has supported Kafka since it's inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.

What's hot

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

confluent

Apache Pulsar: A Foundation Backbone for Clever Cloud - Pulsar Virtual Summit...

StreamNative

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

confluent

Introducing Exactly Once Semantics To Apache Kafka

Apurva Mehta

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

confluent

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov,...

confluent

Friends don't let friends do dual writes: Outbox pattern with OpenShift Strea...

Red Hat Developers

Running large scale Kafka upgrades at Yelp (Manpreet Singh,Yelp) Kafka Summit...

confluent

Westpac Bank Tech Talk 1: Dive into Apache Kafka

confluent

Kafka aws

Ariel Moskovich

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Paul Brebner

Introduction to Apache Kafka

Shiao-An Yuan

Apache Kafka - Martin Podval

Martin Podval

Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul

HostedbyConfluent

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

HostedbyConfluent

Netflix Data Pipeline With Kafka

Steven Wu

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

confluent

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

HostedbyConfluent

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

HostedbyConfluent

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...

HostedbyConfluent

What's hot (20)

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

Apache Pulsar: A Foundation Backbone for Clever Cloud - Pulsar Virtual Summit...

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

Introducing Exactly Once Semantics To Apache Kafka

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov,...

Friends don't let friends do dual writes: Outbox pattern with OpenShift Strea...

Running large scale Kafka upgrades at Yelp (Manpreet Singh,Yelp) Kafka Summit...

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Kafka aws

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Introduction to Apache Kafka

Apache Kafka - Martin Podval

Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

Netflix Data Pipeline With Kafka

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...

Similar to Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020

Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...

Natan Silnitsky

Spark streaming + kafka 0.10

Joan Viladrosa Riera

Kafka Explainaton

NguyenChiHoangMinh

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Joan Viladrosa Riera

This document provides an overview of Apache Kafka and Apache Spark Streaming and their integration. It discusses what Kafka and Spark Streaming are, how they work, their benefits, and semantics when used together. It also provides examples of code for using the new Kafka integration in Spark 2.0+, including getting metadata, storing offsets in Kafka, and achieving at-most-once, at-least-once, and exactly-once processing semantics. Finally, it shares some insights into how Billy Mobile uses Spark Streaming with Kafka to process large volumes of data.

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Trivadis

Apache Kafka - Scalable Message Processing and more!

Guido Schmutz

After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL. Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go. Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Natan Silnitsky

How to build 1000 microservices with Kafka and thrive

Natan Silnitsky

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Natan Silnitsky

Kafka is the bedrock of Wix’s distributed Mega Microservices system. Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices. In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These Include: * How to increase dev velocity of event-driven style code. * How to optimize working with Kafka in polyglot setting * How to migrate from request-reply to event-driven * How to tackle multiple DCs environment.

Scaling big with Apache Kafka

Nikolay Stoitsev

Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.

Apache Kafka - A modern Stream Processing Platform

Guido Schmutz

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Natan Silnitsky

Spark Streaming + Kafka 0.10: an integration story by Joan Viladrosa Riera at...

Big Data Spain

This document provides an overview of Apache Kafka and Spark Streaming and their integration. It discusses: - What Apache Kafka is and how it works as a publish-subscribe messaging system with topics, partitions, producers, and consumers. - What Apache Spark Streaming is and how it provides streaming data processing using micro-batching and leveraging Spark's APIs and engine. - The evolution of the integration between Kafka and Spark Streaming, from using receivers to the direct approach without receivers in Spark 1.3+. - Details on how to use the new direct Kafka integration in Spark 2.0+ including location strategies, consumer strategies, and committing offsets directly to Kafka. - Considerations around at-least

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...

HostedbyConfluent

Event-driven application architectures are becoming increasingly common as a large number of users demand more interactive, real-time, and intelligent responses. Yet it can be challenging to decide how to capture and perform real-time data analysis and deliver differentiating experiences. Join experts from Confluent and AWS to learn how to build Apache Kafka®-based streaming applications backed by machine learning models. Adopting the recommendations will help you establish repeatable patterns for high performing event-based apps.

Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka

Guido Schmutz

[Spark Summit EU 2017] Apache spark streaming + kafka 0.10 an integration story

Joan Viladrosa Riera

Apache Spark Streaming + Kafka 0.10 with Joan Viladrosariera

Spark Summit

Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.

10 Lessons Learned from using Kafka with 1000 microservices - java global summit

Natan Silnitsky

Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices. We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place. In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include: * How to increase dev velocity of event driven style code. * How to optimize working with Kafka in polyglot setting * How to support growing amount of traffic and developers. * How to tackle multiple DCs environment.

Kонструируем микросервисную архитектуру на Symfony и связываем с Kong

Alex Demchenko

This document discusses microservices architecture and using Kong as an API gateway. It begins with an introduction to microservices vs monolithic architecture. Then it covers implementing microservices with RabbitMQ and Symfony. Next, it discusses security considerations for microservices. It introduces Kong and how it provides a centralized gateway for microservices. Finally, it covers setting up Kong with Docker and using its management API and plugins.

Netflix Keystone—Cloud scale event processing pipeline

Monal Daxini

Similar to Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020 (20)