Net flix kafka seattle meetup

•

4 likes•491 views

Nitin Kumar

Learn about how Netflix achieved 5 9s guarentees on KAfka's largest workload.

Engineering

Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Management
HTTP
PROXY

Fronting Kafka Clusters Consumer Kafka Clusters
Number of clusters 24 15
Total number of instances 1700+ 1100+
Instance type d2.2xl i2.2xl
Replication factor 2 2
Retention period 8 to 24 hours 2 to 4 hours

Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 0 1 1 2 2 3
N = Partition N for a topic with 2 replicas
0 ← Off line partition

Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 1 2 0 1 2 3
N = Partition N for a topic with 2 replicas
No offline partition

Event
Producer
Kafka
Buffer exhausted
and message
drop Slow replication
Broker with
networking
problem
Disk read
causes slow
responses
X
X
X

RouterFronting
Kafka
Event
Producer
X
Consumer
Kafka
Copy topic metadata
Consumer

Similar to Net flix kafka seattle meetup

Spotify has built several real-time pipelines using Apache Storm for use cases like ad targeting, music recommendation, data visualization, and notifications. Each of these real-time pipelines have Apache Storm wired to different systems like Apache Kafka, Apache Cassandra, Apache Zookeeper, and other sources and sinks. In this talk the speaker, Kinshuk Mishra, will share his experiences of scaling Apache Storm pipelines at Spotify. The talk will cover the topics such as Spotify's data architecture, best practices, caching, tuning event sources and sinks, monitoring pipeline health and miscellaneous optimizations to make Apache Storm pipelines more robust.

How Spotify scales Apache Storm Pipelines

Kinshuk Mishra

Mirko Damiani - An Embedded soft real time distributed system in Go

linuxlab_conf

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Peter Bakas

Apache Pulsar under the hood

StreamNative

Kafka At Scale in the Cloud

confluent

Many organizations use Apache Kafka® to build data pipelines that span multiple geographically distributed data centers, for use cases ranging from high availability and disaster recovery, to data aggregation and regulatory compliance. The journey from single-cluster deployments to multi-cluster deployments can be daunting, as you need to deal with networking configurations, security models and operational challenges. Geo-replication support for Kafka has come a long way, with both open-source and commercial solutions that support various replication topologies and disaster recovery strategies. So, grab your towel, and join us on this journey as we look at tools, practices, and patterns that can help us build reliable, scalable, secure, global (if not inter-galactic) data pipelines that meet your business needs, and might even save the world from certain destruction.

A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...

HostedbyConfluent

Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a separate ZooKeeper cluster. In 2019, we outlined a plan to break this dependency and bring metadata management into Kafka itself through a dynamic service that runs inside the Kafka Cluster. We call this the Quorum Controller. In this talk, we’ll look at how the Quorum Controller works and how it integrates with other parts of the next-generation Kafka architecture, such as the Raft quorum and snapshotting mechanism. We’ll also explain how the Quorum Controller will simplify operations, improve security, and enhance scalability and performance. Finally, we’ll look at some of the practicalities, such as how to monitor and run the Quorum Controller yourself. We’ll talk about some of the performance gains we’ve seen, and our plans for the future.

Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent

HostedbyConfluent

Kafka Retry and DLQ

George Teo

Open Source Backends for OpenStack Neutron

mestery

My broken container is gone - how to debug containers on container platforms

Aarno Aukia

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Monal Daxini

This talk was presented at the Hotstar Scale Meetup in Bangalore by Jayesh Sidhwani In this talk, the presenter introduces Apache Kafka and the Apache Kafka Streams library. Starting from the need for building streaming applications to thinking the use-cases as a streaming job - this talk covers all the technicalities. It ends with a short description of how Kafka is deployed and used at Hotstar

Build real time stream processing applications using Apache Kafka

Hotstar

Nyc storm meetup_robdoherty

Robert Doherty

Over the last few years, we have been working on removing the dependency on ZooKeeper from Apache Kafka®. Instead of using an external system to store metadata, Kafka can now manage its own metadata. This new mode of operation is called Kafka Raft mode, or ""KRaft"" for short. It has many performance and scalability benefits. This talk will discuss our efforts to get KRaft mode production-ready. We will talk about the old and new architectures, and how we adapted features to work in both worlds. We will also talk about our experiences with testing and deploying the new software. Finally, we'll talk about what's planned for the future.

Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...

HostedbyConfluent

nebulaconf

Pedro Dias

Presentation on reactor design pattern should introduce you a pattern that our server’s core is based on. We will try to give away all pros and cons about it. This pattern is used to simplify development of highly concurrent servers by separating business logic from communication. This is very technical presentation, but it is designed for beginners so it should be understandable to anyone with basic software engineering skills.

Dejan Pekter / Nordeus – Reactor design pattern

ConversionMeetup

Reactive applications are becoming a de-facto industry standard and, if employed correctly, toolkits like Lightbend Reactive Platform make the implementation easier than ever. But design of these systems might be challenging as it requires particular mindset shift to tackle problems we might not be used to. In this talk we’re going to discuss the most common things I’ve seen in the field that prevented applications to work as expected. I’d like to talk about typical pitfalls that might cause troubles, about trade-offs that might not be fully understood or important choices that might be overlooked including persistent actors pitfalls, tackling of network partitions, proper implementations of graceful shutdown or distributed transactions, trade-offs of micro-services or actors and more. This talk should be interesting for anyone who is thinking about, implementing, or have already deployed reactive application. My goal is to provide a comprehensive explanation of common problems to be sure they won’t be repeated by fellow developers. The talk is a little bit more focused on Lightbend platform but understanding of the concepts we are going to talk about should be beneficial for everyone interested in this field.

Reactive mistakes - ScalaDays Chicago 2017

Petr Zapletal

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022

HostedbyConfluent

Transactions were added to Apache Kafka with KIP-98. While much of the protocol remains intact, transactions in Kafka have evolved over time to handle edge cases and errors found over the years. KIP-890 hopes to cover most of the remaining gaps in the protocol. This talk will give a refresher on transactions and idempotency and chronicle the various KIPs that improved the protocol over the years. We will also discuss the problem of hanging transactions and how KIP-890 hopes to solve it as well as strengthen the transactional protocol overall.

Transactions in Action: the Story of Exactly Once in Apache Kafka

HostedbyConfluent

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

confluent

Similar to Net flix kafka seattle meetup (20)

How Spotify scales Apache Storm Pipelines

Mirko Damiani - An Embedded soft real time distributed system in Go

Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec

Apache Pulsar under the hood

Kafka At Scale in the Cloud

A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...

Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent

Kafka Retry and DLQ

Open Source Backends for OpenStack Neutron

My broken container is gone - how to debug containers on container platforms

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Build real time stream processing applications using Apache Kafka

Nyc storm meetup_robdoherty

Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...

nebulaconf

Dejan Pekter / Nordeus – Reactor design pattern

Reactive mistakes - ScalaDays Chicago 2017

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022

Transactions in Action: the Story of Exactly Once in Apache Kafka

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

Recently uploaded

Electrostatic field in a coaxial transmission line

JulioCesarSalazarHer1

Theory for How to calculation capacitor bank

tawat puangthong

Introduction to Artificial Intelligence and History of AI

Sheetal Jain

Intelligent Agents, A discovery on How A Rational Agent Acts

Sheetal Jain

ALCOHOL PRODUCTION- Beer Brewing Process.pdf

Madan Karki

It is a new concept in town planning, and it may have a highly positive impact on cities and communities. It not only promotes sustainability but also helps increase the health of people and boost the economy. Although it has a lot of advantages, critics argue that this concept is not for all cities and describe its effects as well. New cities should follow this pattern to build and develop their cities

15-Minute City: A Completely New Horizon

Morshed Ahmed Rahath

Attraction and Repulsion type Moving Iron Instruments.pptx

karthikeyanS725446

ROAD CONSTRUCTION PRESENTATION.PPTX.pptx

GagandeepKaur617299

This topic aims to describe the project background, problem statement, objectives, scopes, project significance and expected output of the system. • The system is Burger Ordering System. This is online Customer Ordering System of Restaurant, which in most cases; the company has problem with order and disordered order. • This project intends to computerize Burger Ordering System to provide better customer service. Because of that, the restaurant can provide the easier way of travelling to the customer. • Burger Oder system aims to accelerate customer orders. • Burger Ordering System used by servers and kitchen employees to accept customer orders. • An online ordering system is not a new concept to many as this has been running successfully all over the world for quite some time now. • The whole concept behind having a professional website along with a system shows how you present yourself to the online world.

BURGER ORDERING SYSYTEM PROJECT REPORT..pdf

Kamal Acharya

NEWLETTER FRANCE HELICES/ SDS SURFACE DRIVES - MAY 2024

EMMANUELLEFRANCEHELI

The Teachers Record Management System is a comprehensive solution designed to simplify the process of managing teacher records. In today's fast-paced world, it is imperative for schools and individuals to have access to accurate and up-to-date information about teachers. This system addresses the needs of both individuals and schools, offering an efficient and effective way to manage teacher records. This web-based technology was developed with the aim of providing an easy-to-use platform that allows users to search and manage teacher records with ease. The system is designed to be user-friendly, making it accessible to individuals with varying levels of computer literacy. The interface is intuitive and easy to navigate, making it possible for users to quickly find the information they need. For individuals who are searching for good teachers, the Teachers Record Management System offers a comprehensive database of teacher records that can be searched using various parameters such as subject, qualification, experience, and location. This allows users to find the best teachers that match their requirements in a matter of minutes. The system also offers the ability to rate and review teachers, providing valuable feedback to other users who are looking for good teachers. For schools, the Teachers Record Management System offers a robust and secure platform for managing teacher records. The system allows schools to maintain accurate and up-to-date records of their teachers, including their personal information, academic qualifications, work experience, and other relevant details. This makes it easy for schools to keep track of their teachers and ensure that they have the right teachers for the job. The system also offers a range of features that help schools manage their teachers more efficiently. For example, the system allows schools to schedule classes and assign teachers to specific classes based on their qualifications and experience. It also offers a range of reporting and analytics tools that enable schools to track their teacher's performance and identify areas where improvements can be made. In conclusion, the Teachers Record Management System is a powerful tool that offers a range of benefits to both individuals and schools. It provides an easy and efficient way to search and manage teacher records, allowing users to find the best teachers that match their requirements. The system is also designed to be user-friendly and offers a range of features that help schools manage their teachers more efficiently. Overall, the Teachers Record Management System is an essential tool for anyone who wants to find good teachers or manage their teacher records.

Teachers record management system project report..pdf

Kamal Acharya

ANSI(ST)-III_Manufacturing-I_05052020.pdf

BertinKamsipa1

DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf

DrGurudutt

2024 DevOps Pro Europe - Growing at the edge

Paco Orozco

This simple Python software is designed to assist Civil and Geotechnical Engineers in performing site-specific seismic hazard assessments. The program calculates the seismic response spectrum based on user-provided geotechnical and seismic parameters, generating a comprehensive technical report that includes the response spectrum data and figures. The analysis adheres to Eurocode 8 and the Greek Annex, ensuring compliance with international standards for earthquake-resistant design.

Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis

Dr.Costas Sachpazis

Research Methodolgy & Intellectual Property Rights Series 1

T.D. Shashikala

E-Commerce Shopping using MERN Stack where different modules are present

jatinraor66

The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...

Roi Lipman

The project developers created a system entitled Resort Management and Reservation System; it will provide better management and monitoring of the services in every resort business, especially D’ Rock Resort. To accommodate those out-of-town guests who want to remain and utilize the resort's services, the proponents planned to automate the business procedures of the resort and implement the system. As a result, it aims to improve business profitability, lower expenses, and speed up the resort's transaction processing. The resort will now be able to serve those potential guests, especially during the high season. Using websites for faster transactions to reserve on your desired time and date is another step toward technological advancement. Customers don’t need to walk in and hold in line for several hours. There is no problem in converting a paper-based transaction online; it's just the system that will be used that will help the resort expand. Moreover, Gerard (2012) stated that “The flexible online information structure was developed as a tool for the reservation theory's two primary applications. Computer use is more efficient, accurate, and faster than a manual or present lifestyle of operation. Using a computer has a vital role in our daily life and the advantages of the devices we use.

RESORT MANAGEMENT AND RESERVATION SYSTEM PROJECT REPORT.pdf

Kamal Acharya

Instruct Nirmaana 24-Smart and Lean Construction Through Technology.pdf

Er.Sonali Nasikkar

Recently uploaded (20)

Electrostatic field in a coaxial transmission line

Theory for How to calculation capacitor bank

Introduction to Artificial Intelligence and History of AI

Intelligent Agents, A discovery on How A Rational Agent Acts

ALCOHOL PRODUCTION- Beer Brewing Process.pdf

15-Minute City: A Completely New Horizon

Attraction and Repulsion type Moving Iron Instruments.pptx

ROAD CONSTRUCTION PRESENTATION.PPTX.pptx

BURGER ORDERING SYSYTEM PROJECT REPORT..pdf

NEWLETTER FRANCE HELICES/ SDS SURFACE DRIVES - MAY 2024

Teachers record management system project report..pdf

ANSI(ST)-III_Manufacturing-I_05052020.pdf

DR PROF ING GURUDUTT SAHNI WIKIPEDIA.pdf

2024 DevOps Pro Europe - Growing at the edge

Seismic Hazard Assessment Software in Python by Prof. Dr. Costas Sachpazis

Research Methodolgy & Intellectual Property Rights Series 1

E-Commerce Shopping using MERN Stack where different modules are present

The battle for RAG, explore the pros and cons of using KnowledgeGraphs and Ve...

RESORT MANAGEMENT AND RESERVATION SYSTEM PROJECT REPORT.pdf

Instruct Nirmaana 24-Smart and Lean Construction Through Technology.pdf