Submit Search
Upload
Net flix kafka seattle meetup
•
4 likes
•
491 views
Nitin Kumar
Follow
Learn about how Netflix achieved 5 9s guarentees on KAfka's largest workload.
Read less
Read more
Engineering
Report
Share
Report
Share
1 of 38
Download now
Download to read offline
Recommended
Titus AWS VPC networking for containers
Titus AWS VPC networking for containers
Andrew Leung
Blockchain introduction
Blockchain introduction
Thao Huynh Quang
[Lightning] Microsoft q# on vsts mvp lightning
[Lightning] Microsoft q# on vsts mvp lightning
Rolf Huisman
Implementing error budgets
Implementing error budgets
Yaroslav Molochko
Machinekit - The Python Machinetalk Bindings
Machinekit - The Python Machinetalk Bindings
Alexander Rössler
GEO mapbox geo_api_develop2 Intro
GEO mapbox geo_api_develop2 Intro
Max Kleiner
From Three Nines to Five Nines - A Kafka Journey
From Three Nines to Five Nines - A Kafka Journey
Allen (Xiaozhong) Wang
Producer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
Recommended
Titus AWS VPC networking for containers
Titus AWS VPC networking for containers
Andrew Leung
Blockchain introduction
Blockchain introduction
Thao Huynh Quang
[Lightning] Microsoft q# on vsts mvp lightning
[Lightning] Microsoft q# on vsts mvp lightning
Rolf Huisman
Implementing error budgets
Implementing error budgets
Yaroslav Molochko
Machinekit - The Python Machinetalk Bindings
Machinekit - The Python Machinetalk Bindings
Alexander Rössler
GEO mapbox geo_api_develop2 Intro
GEO mapbox geo_api_develop2 Intro
Max Kleiner
From Three Nines to Five Nines - A Kafka Journey
From Three Nines to Five Nines - A Kafka Journey
Allen (Xiaozhong) Wang
Producer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
How Spotify scales Apache Storm Pipelines
How Spotify scales Apache Storm Pipelines
Kinshuk Mishra
Mirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in Go
linuxlab_conf
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Peter Bakas
Apache Pulsar under the hood
Apache Pulsar under the hood
StreamNative
Kafka At Scale in the Cloud
Kafka At Scale in the Cloud
confluent
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
HostedbyConfluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
HostedbyConfluent
Kafka Retry and DLQ
Kafka Retry and DLQ
George Teo
Open Source Backends for OpenStack Neutron
Open Source Backends for OpenStack Neutron
mestery
My broken container is gone - how to debug containers on container platforms
My broken container is gone - how to debug containers on container platforms
Aarno Aukia
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Monal Daxini
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Hotstar
Nyc storm meetup_robdoherty
Nyc storm meetup_robdoherty
Robert Doherty
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
HostedbyConfluent
nebulaconf
nebulaconf
Pedro Dias
Dejan Pekter / Nordeus – Reactor design pattern
Dejan Pekter / Nordeus – Reactor design pattern
ConversionMeetup
Reactive mistakes - ScalaDays Chicago 2017
Reactive mistakes - ScalaDays Chicago 2017
Petr Zapletal
Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022
Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022
HostedbyConfluent
Transactions in Action: the Story of Exactly Once in Apache Kafka
Transactions in Action: the Story of Exactly Once in Apache Kafka
HostedbyConfluent
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
confluent
Deep learning with kafka
Deep learning with kafka
Nitin Kumar
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
Nitin Kumar
More Related Content
Similar to Net flix kafka seattle meetup
How Spotify scales Apache Storm Pipelines
How Spotify scales Apache Storm Pipelines
Kinshuk Mishra
Mirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in Go
linuxlab_conf
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Peter Bakas
Apache Pulsar under the hood
Apache Pulsar under the hood
StreamNative
Kafka At Scale in the Cloud
Kafka At Scale in the Cloud
confluent
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
HostedbyConfluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
HostedbyConfluent
Kafka Retry and DLQ
Kafka Retry and DLQ
George Teo
Open Source Backends for OpenStack Neutron
Open Source Backends for OpenStack Neutron
mestery
My broken container is gone - how to debug containers on container platforms
My broken container is gone - how to debug containers on container platforms
Aarno Aukia
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Monal Daxini
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Hotstar
Nyc storm meetup_robdoherty
Nyc storm meetup_robdoherty
Robert Doherty
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
HostedbyConfluent
nebulaconf
nebulaconf
Pedro Dias
Dejan Pekter / Nordeus – Reactor design pattern
Dejan Pekter / Nordeus – Reactor design pattern
ConversionMeetup
Reactive mistakes - ScalaDays Chicago 2017
Reactive mistakes - ScalaDays Chicago 2017
Petr Zapletal
Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022
Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022
HostedbyConfluent
Transactions in Action: the Story of Exactly Once in Apache Kafka
Transactions in Action: the Story of Exactly Once in Apache Kafka
HostedbyConfluent
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
confluent
Similar to Net flix kafka seattle meetup
(20)
How Spotify scales Apache Storm Pipelines
How Spotify scales Apache Storm Pipelines
Mirko Damiani - An Embedded soft real time distributed system in Go
Mirko Damiani - An Embedded soft real time distributed system in Go
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Netflix Keystone - How Netflix Handles Data Streams up to 11M Events/Sec
Apache Pulsar under the hood
Apache Pulsar under the hood
Kafka At Scale in the Cloud
Kafka At Scale in the Cloud
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka Retry and DLQ
Kafka Retry and DLQ
Open Source Backends for OpenStack Neutron
Open Source Backends for OpenStack Neutron
My broken container is gone - how to debug containers on container platforms
My broken container is gone - how to debug containers on container platforms
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Nyc storm meetup_robdoherty
Nyc storm meetup_robdoherty
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Su...
nebulaconf
nebulaconf
Dejan Pekter / Nordeus – Reactor design pattern
Dejan Pekter / Nordeus – Reactor design pattern
Reactive mistakes - ScalaDays Chicago 2017
Reactive mistakes - ScalaDays Chicago 2017
Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022
Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022
Transactions in Action: the Story of Exactly Once in Apache Kafka
Transactions in Action: the Story of Exactly Once in Apache Kafka
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
Kafka Summit SF 2017 - Running Kafka as a Service at Scale
More from Nitin Kumar
Deep learning with kafka
Deep learning with kafka
Nitin Kumar
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
Nitin Kumar
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Nitin Kumar
Processing trillions of events per day with apache
Processing trillions of events per day with apache
Nitin Kumar
Ren cao kafka connect
Ren cao kafka connect
Nitin Kumar
Insta clustr seattle kafka meetup presentation bb
Insta clustr seattle kafka meetup presentation bb
Nitin Kumar
EventHub for kafka ecosystems kafka meetup
EventHub for kafka ecosystems kafka meetup
Nitin Kumar
Kafka eos
Kafka eos
Nitin Kumar
Microsoft challenges of a multi tenant kafka service
Microsoft challenges of a multi tenant kafka service
Nitin Kumar
Avvo fkafka
Avvo fkafka
Nitin Kumar
Brandon obrien streaming_data
Brandon obrien streaming_data
Nitin Kumar
Confluent kafka meetupseattle jan2017
Confluent kafka meetupseattle jan2017
Nitin Kumar
Microsoft kafka load imbalance
Microsoft kafka load imbalance
Nitin Kumar
Map r seattle streams meetup oct 2016
Map r seattle streams meetup oct 2016
Nitin Kumar
Linked in multi tier, multi-tenant, multi-problem kafka
Linked in multi tier, multi-tenant, multi-problem kafka
Nitin Kumar
Seattle kafka meetup nov 2015 published siphon
Seattle kafka meetup nov 2015 published siphon
Nitin Kumar
More from Nitin Kumar
(16)
Deep learning with kafka
Deep learning with kafka
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
2019 04 seattle_meetup___kafka_machine_learning___kai_waehner
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...
Processing trillions of events per day with apache
Processing trillions of events per day with apache
Ren cao kafka connect
Ren cao kafka connect
Insta clustr seattle kafka meetup presentation bb
Insta clustr seattle kafka meetup presentation bb
EventHub for kafka ecosystems kafka meetup
EventHub for kafka ecosystems kafka meetup
Kafka eos
Kafka eos
Microsoft challenges of a multi tenant kafka service
Microsoft challenges of a multi tenant kafka service
Avvo fkafka
Avvo fkafka
Brandon obrien streaming_data
Brandon obrien streaming_data
Confluent kafka meetupseattle jan2017
Confluent kafka meetupseattle jan2017
Microsoft kafka load imbalance
Microsoft kafka load imbalance
Map r seattle streams meetup oct 2016
Map r seattle streams meetup oct 2016
Linked in multi tier, multi-tenant, multi-problem kafka
Linked in multi tier, multi-tenant, multi-problem kafka
Seattle kafka meetup nov 2015 published siphon
Seattle kafka meetup nov 2015 published siphon
Recently uploaded
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
dharasingh5698
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
Call Girls in Nagpur High Profile
DJARUM4D - SLOT GACOR ONLINE | SLOT DEMO ONLINE
DJARUM4D - SLOT GACOR ONLINE | SLOT DEMO ONLINE
slot gacor bisa pakai pulsa
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Call Girls in Nagpur High Profile
AKTU Computer Networks notes --- Unit 3.pdf
AKTU Computer Networks notes --- Unit 3.pdf
ankushspencer015
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
Dr.Costas Sachpazis
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
rknatarajan
UNIT-III FMM. DIMENSIONAL ANALYSIS
UNIT-III FMM. DIMENSIONAL ANALYSIS
rknatarajan
College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
Call Girls in Nagpur High Profile
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
ranjana rawat
Online banking management system project.pdf
Online banking management system project.pdf
Kamal Acharya
Coefficient of Thermal Expansion and their Importance.pptx
Coefficient of Thermal Expansion and their Importance.pptx
Asutosh Ranjan
BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptx
BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptx
fenichawla
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
Call Girls in Nagpur High Profile
Call Girls Service Nashik Vaishnavi 7001305949 Independent Escort Service Nashik
Call Girls Service Nashik Vaishnavi 7001305949 Independent Escort Service Nashik
Call Girls in Nagpur High Profile
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
Asst.prof M.Gokilavani
Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts
Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts
Call Girls in Nagpur High Profile
Call for Papers - Educational Administration: Theory and Practice, E-ISSN: 21...
Call for Papers - Educational Administration: Theory and Practice, E-ISSN: 21...
Christo Ananth
UNIT-II FMM-Flow Through Circular Conduits
UNIT-II FMM-Flow Through Circular Conduits
rknatarajan
(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...
(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...
ranjana rawat
Recently uploaded
(20)
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Ankleshwar 7001035870 Whatsapp Number, 24/07 Booking
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
DJARUM4D - SLOT GACOR ONLINE | SLOT DEMO ONLINE
DJARUM4D - SLOT GACOR ONLINE | SLOT DEMO ONLINE
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
AKTU Computer Networks notes --- Unit 3.pdf
AKTU Computer Networks notes --- Unit 3.pdf
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-III FMM. DIMENSIONAL ANALYSIS
UNIT-III FMM. DIMENSIONAL ANALYSIS
College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
Online banking management system project.pdf
Online banking management system project.pdf
Coefficient of Thermal Expansion and their Importance.pptx
Coefficient of Thermal Expansion and their Importance.pptx
BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptx
BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptx
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
Call Girls Service Nashik Vaishnavi 7001305949 Independent Escort Service Nashik
Call Girls Service Nashik Vaishnavi 7001305949 Independent Escort Service Nashik
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts
Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts
Call for Papers - Educational Administration: Theory and Practice, E-ISSN: 21...
Call for Papers - Educational Administration: Theory and Practice, E-ISSN: 21...
UNIT-II FMM-Flow Through Circular Conduits
UNIT-II FMM-Flow Through Circular Conduits
(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...
(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...
Net flix kafka seattle meetup
1.
2.
● ○ ○ ● ○ ○ ○
3.
● ● ●
4.
A NETFLIX ORIGINAL
SERVICE
5.
Stream Consumers Router EMR Fronting Kafka Event Producer Consumer Kafka Management HTTP PROXY
6.
Fronting Kafka Clusters
Consumer Kafka Clusters Number of clusters 24 15 Total number of instances 1700+ 1100+ Instance type d2.2xl i2.2xl Replication factor 2 2 Retention period 8 to 24 hours 2 to 4 hours
7.
● ○ ○ ○ ● ○
8.
● ○ ●
9.
● ○ ● ● ● ○ ○
10.
● ● ● 0.1% 0.5% 1%
5% Percent loss
11.
● ● ○ ○ ●
12.
● ● ●
13.
● ○ ○ ● …
14.
15.
● ○ →
16.
● ● ○ ○ ○ → ●
17.
18.
● ● ○ ○ ○
19.
● ○ ○ ○ ● ●
20.
● ○ → ● ○ → ● ○
→
21.
● ● ○ ● ○
22.
Rack 0 Rack
1 0 Broker 0 Broker 1 Broker 2 Broker 3 3 0 1 1 2 2 3 N = Partition N for a topic with 2 replicas 0 ← Off line partition
23.
Rack 0 Rack
1 0 Broker 0 Broker 1 Broker 2 Broker 3 3 1 2 0 1 2 3 N = Partition N for a topic with 2 replicas No offline partition
24.
● ● ○ ● ●
25.
● ○ ○ ○ ● ○ ○
26.
Event Producer Kafka Buffer exhausted and message drop
Slow replication Broker with networking problem Disk read causes slow responses X X X
27.
28.
28
29.
29
30.
● ● ○ ○ ○
31.
● ○ ○ ○
32.
● ○ ○ ● ● ●
33.
34.
● ● ● ●
35.
RouterFronting Kafka Event Producer X Consumer Kafka Copy topic metadata Consumer
36.
● Fully Automated
Download now