Joel Koshy, profile picture
Uploaded byJoel Koshy
PDF, PPTX63,415 views

Consumer offset management in Kafka

The document discusses consumer offset management in Kafka, focusing on the challenges and best practices for storing offsets, particularly in relation to Zookeeper versus Kafka's own storage. It emphasizes the need for durable, high-write-load systems that ensure consistent reads and atomic commits, while also addressing the handling of dead consumers and recommended topic settings. Additionally, it outlines migration strategies between Zookeeper and Kafka, along with key metrics to monitor during these processes.

Embed presentation

Download as PDF, PPTX
Consumer offset management in Kafka Joel Koshy Kafka meetup @ LinkedIn March 24, 2015
Consumers and offsets 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1 3 2 3 3 3 4 PageViewEvent-0 EmailBounceEvent-0 PageViewEvent-0 EmailBounceEvent-0 35 12 Offset map GroupId: audit-consumer
Store offsets in ZooKeeper admin brokers config consumers controller controller_epoch audit-consumer ids owners offsets PageViewEvent EmailBounceEvent 0 0 12 35
(Don’t) store offsets in ZooKeeper •  Heavy write-load on ZooKeeper •  Especially an issue – during 0.7 to 0.8 migration – and before we switched to SSDs •  Non-ideal work-arounds – Increase offset-commit intervals – Filter commits if offsets have not moved – Spread large offset commits over commit interval
Offset management (ideals) •  Durable •  Support high write-load •  Consistent reads •  Atomic offset commits •  Fast commits/fetches
Store offsets in a replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets Next commit Group Offset Partition
Store offsets in a replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets audit-consumer EmailBounceEvent-0 248
Store offsets in a replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323
Store offsets in a replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543
Store offsets in a replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 Partition è abs(GroupId.hashCode()) % NumPartitions
Store offsets in a replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 Offset commits append to the offsets topic partition Offset fetches read from the offsets topic partition
Store offsets in a replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 [audit-consumer, PageViewEvent-0] [audit-consumer, EmailBounceEvent-0] [mirrormaker, ClickEvent-0] [mirrormaker, ClickEvent-1] Offsets cache 323 248 54674 54543 Offset commits append to the offsets topic partition + update the cache Offset fetches read from the offsets topic partition cache
Store offsets in a replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 [audit-consumer, PageViewEvent-0] [audit-consumer, EmailBounceEvent-0] [mirrormaker, ClickEvent-0] [mirrormaker, ClickEvent-1] Offsets cache 323 248 54674 54543 Offset commits append to the offsets topic partition + update the cache Offset fetches read from the offsets topic partition cache How do we GC older offset entries?
log.cleanup.policy = compact 0 1 2 3 4 5 6 7 8 9 10 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 V1 V2 V3 V4 V5 V6 V7 V8 V9 V 10 V 11 3 4 K1 K3 V4 V5 6 K4 V7 8 10 K5 K6 V9 V 11 Compaction Offset Key Value 11 K2 Ø Offset Key Value
Store offsets in a replicated, partitioned, compacted log audit-consumer PageViewEvent-0 126312342 audit-consumer EmailBounceEvent-0 59843 audit-consumer PageViewEvent-0 126319628 audit-consumer EmailBounceEvent-0 86243 audit-consumer PageViewEvent-0 126398102 Key Value audit-consumer EmailBounceEvent-0 86243 audit-consumer PageViewEvent-0 126398102 Compaction Key è [Group, Topic, Partition] Value è Offset
Dealing with dead consumers console-consumer-38587, console-consumer-94777, console-consumer-94774, console-consumer-31199, console-consumer-51555, console-consumer-43182, mobileServiceConsumerDwwewewA13dafddesfasdfdee33, console-consumer-57784, python-kafka-consumer-0959a04da7c241448beb0813f002e34b, console- consumer-70750, console-consumer-94809, console-consumer-87470, touch-me-not, console- consumer-43246, console-consumer-69811, python-kafka-consumer-82c2d653128840d5b6bcbfc5ac7f3abc, console-consumer-33847, console-consumer-18217, console-consumer-87493, console-consumer-26414, console-consumer-67299, voldemort-reader-jjkoshy, console-consumer-80245, kafka_listener_for_comments, test-flow-staging, console-consumer-8441, console-consumer-67258, data- processor-2, console-consumer-94869, console-consumer-55242, pinot-beta-hackday_1_2, console- consumer-6601, cloud-host1, system-metrics-monitor-01, console-consumer-70859, console- consumer-26477, page-view-test-flow-2, page-view-test-flow-1, python-kafka-consumer- bf33d075b22d4ddfb82d4a055303e909, console-consumer-99768, console-consumer-45509, console- consumer-21504, points-test_devel_l1_1686489164, console-consumer-14841, console-consumer-4098, console-consumer-14746, console-consumer-94575, cloud-dcb-host147.company.com, teacup_reporting_alex, console-consumer-4132, console-consumer-48171, ropod-dcb-host794.company.com, console-consumer-63743, console-consumer-36147, console-consumer-48138, console-consumer-33595, console-consumer-6808, console-consumer-31000, console-consumer- 73064, console-consumer-18050, console-consumer-21683, share-message, ropod-dcb-host959.company.com, ropod-dcb-host949.company.com, sensei-test_dcb_host138.company.com_1924844804, console-consumer-38654, console-consumer-92040, console-consumer-67052, console-consumer-82690, console-consumer-92002, console-consumer-69687, console-consumer-31077, console-consumer-94657, console-consumer-36064, console-consumer-45675, console-consumer-45671, console-consumer-70625, MemberSettings-dcx, console-consumer-55513, member- links-dcx, console-consumer-85367, opportunist-company, forum-queue, console-consumer-87912, console-consumer-75909, console-consumer-12320, sensei-test_user2_808173709, ropod-dcb- host937.company.com, console-consumer-8710, console-consumer-48390, python-kafka- consumer-816cebafabb34dd5be6bfce59cbee411, console-consumer-8701, console-consumer-6122, console- consumer-6142, metrics-dcb-monitor19, console-consumer-73329, console-consumer-87942, console- consumer-80552, console-consumer-48368, autometrics-dcb-host13, …!
Dealing with dead consumers •  For offsets older than offset retention period: – Append tombstone – Remove offset entry from cache
Recommended settings for offsets topic Replication factor >= 3 min.insync.replicas >= 2 unclean.leader.election.enable False offsets.commit.required.acks -1 (all)
How to commit/fetch offsets audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 V I P Consumer metadata request Response (manager=2)
How to commit/fetch offsets audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 Offset fetches Offset commits cache replication
When the offset manager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 cache Become Leader load cache
When the offset manager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 cache Become Leader load cache Become follower XXXXXX
When the offset manager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) Offset fetches Offset commits cache __consumer_offsets-34: Leader: 0, ISR: 0, 1, 2 cache X X
When the offset manager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) V I P Consumer metadata request cache __consumer_offsets-34: Leader: 0, ISR: 0, 1, 2 cache Response (manager=0)
When the offset manager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) cache __consumer_offsets-34: Leader: 0, ISR: 0, 1, 2 cache Offset commits Offset fetches replication
Offset{Commit,Fetch} API ConsumerMetadataRequest o Group Id: String ConsumerMetadataResponse o Error code: Short o Offset manager: Kafka broker info
Offset{Commit,Fetch} API OffsetCommitRequest o groupId: String o Offset map §  Key è Topic-partition §  Value è Partition-data •  Offset: Long •  Timestamp: Long •  Metadata: String KAFKA-1634: changes semantics of timestamp to retention
Offset{Commit,Fetch} API OffsetCommitResponse o Response map §  Key è Topic-partition §  Value è Error code
Offset{Commit,Fetch} API OffsetFetchRequest o Group Id: String o Partitions: List<Topic-partition> OffsetFetchResponse o Response map §  Key è Topic-partition §  Value è Partition-data •  Offset: Long •  Metadata: String •  Error code: Short
Offset{Commit,Fetch} API Code samples: http://bit.ly/1LTJBYo
Offset{Commit,Fetch} API KafkaConsumer<K, V> consumer = new KafkaConsumer<K, V>(properties);! …! TopicPartition partition1 = new TopicPartition("topic1", 0);! TopicPartition partition1 = new TopicPartition("topic1", 1);! ! consumer.subscribe(partition1, partition2);! ! Map<TopicPartition, Long> offsets = new LinkedHashMap<TopicPartition, Long>();! offsets.put(partition1, 123L);! offsets.put(partition2, 4320L);! …! // commit offsets! consumer.commit(offsets, CommitType.SYNC);! …! // fetch offsets! long committedOffset = consumer.committed(partition1);! !
How to read the offsets topic To read everything, use the console consumer! ./bin/kafka-console-consumer.sh --topic __consumer_offsets -- zookeeper localhost:2181 --formatter "kafka.server.OffsetManager $OffsetsMessageFormatter" --consumer.config config/ consumer.properties! (Must set exclude.internal.topics = false in consumer.properties) ! To read a single partition, use the simple- consumer-shell ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets -- partition 12 --broker-list localhost:9092 --formatter "kafka.server.OffsetManager$OffsetsMessageFormatter"!
Inside the offsets topic [Group, Topic, Partition]::[Offset, Metadata, Timestamp] [audit-consumer,PageViewEvent,7]::OffsetAndMetadata[53568,NO_METADATA,1416363620711]! [audit-consumer,service-log-event,5]::OffsetAndMetadata[168012,NO_METADATA, 1416363620711]! [audit-consumer,EmailBounceEvent,4]::OffsetAndMetadata[8524676,NO_METADATA, 1416363620711]! [audit-consumer,ClickEvent,0]::OffsetAndMetadata[8132292,NO_METADATA,1416363620711]! [audit-consumer,metrics-event,1]::OffsetAndMetadata[1835900,NO_METADATA,1416363620711]! [audit-consumer,CompanyEvent,0]::OffsetAndMetadata[109337,NO_METADATA,1416363620711]! [audit-consumer,test-topic,1]::OffsetAndMetadata[352989,NO_METADATA,1416363620711]! [audit-consumer,meetup-event,2]::OffsetAndMetadata[39961,NO_METADATA,1416363620711]! [audit-consumer,push-topic,6]::OffsetAndMetadata[4210366,NO_METADATA,1416363620711]!
How to migrate/roll-back Migrate from ZooKeeper to Kafka: •  Config change – offsets.storage=kafka – dual.commit.enabled=true •  Rolling bounce •  Config change – dual.commit.enabled=false •  Rolling bounce
How to migrate/roll-back Migrate from Kafka to ZooKeeper: •  Config change – dual.commit.enabled=true •  Rolling bounce •  Config change – offsets.storage=zookeeper – dual.commit.enabled=false •  Rolling bounce
Key metrics to monitor •  Consumer mbeans –  Kafka commit rate –  ZooKeeper commit rate (during migration) •  Broker mbeans –  Max-dirty ratio and other log cleaner metrics –  Offset cache size –  Group count –  {ConsumerMetadata, OffsetCommit, OffsetFetch} request metrics
0.8.3 •  Support compression in compacted topics (KAFKA-1734) •  Change offset commit “timestamp” to mean retention period: KAFKA-1634 •  Offset client
Monitor it!
Acknowledgments Kafka team @ LinkedIn Jay Kreps, Jun Rao, Neha Narkhede @ Confluent Tejas (2013 intern): http://lnkdin.me/p/tejaspatil1

More Related Content

PDF
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
PDF
Introduction to Kafka Streams
byGuozhang Wang
 
PPTX
Improving Kafka at-least-once performance at Uber
byYing Zheng
 
PDF
A Deep Dive into Kafka Controller
byconfluent
 
PPTX
Apache kafka
byViswanath J
 
PDF
Apache Kafka – (Pattern and) Anti-Pattern
byconfluent
 
PDF
Apache Kafka - Martin Podval
byMartin Podval
 
PDF
Introduction to Redis
byDvir Volk
 
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
Introduction to Kafka Streams
byGuozhang Wang
 
Improving Kafka at-least-once performance at Uber
byYing Zheng
 
A Deep Dive into Kafka Controller
byconfluent
 
Apache kafka
byViswanath J
 
Apache Kafka – (Pattern and) Anti-Pattern
byconfluent
 
Apache Kafka - Martin Podval
byMartin Podval
 
Introduction to Redis
byDvir Volk
 

What's hot

PDF
Fundamentals of Apache Kafka
byChhavi Parasher
 
PPTX
Kafka 101
byClement Demonchy
 
PDF
Kafka Streams: What it is, and how to use it?
byconfluent
 
PDF
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
PDF
Understanding Query Plans and Spark UIs
byDatabricks
 
PPTX
Best practices and lessons learnt from Running Apache NiFi at Renault
byDataWorks Summit
 
PDF
Getting Started with Confluent Schema Registry
byconfluent
 
PDF
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
bymumrah
 
PDF
From Zero to Hero with Kafka Connect
byconfluent
 
PDF
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
byconfluent
 
PPTX
HBase and HDFS: Understanding FileSystem Usage in HBase
byenissoz
 
PPTX
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
byFlink Forward
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
PPTX
Introduction to Apache Kafka
byJeff Holoman
 
PDF
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
byHostedbyConfluent
 
PDF
Apache Kafka Fundamentals for Architects, Admins and Developers
byconfluent
 
PPTX
Tuning Apache Kafka Connectors for Flink.pptx
byFlink Forward
 
ODP
Presto
byKnoldus Inc.
 
PPTX
Kafka presentation
byMohammed Fazuluddin
 
PPTX
Apache Spark Architecture
byAlexey Grishchenko
 
Fundamentals of Apache Kafka
byChhavi Parasher
 
Kafka 101
byClement Demonchy
 
Kafka Streams: What it is, and how to use it?
byconfluent
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
byKai Wähner
 
Understanding Query Plans and Spark UIs
byDatabricks
 
Best practices and lessons learnt from Running Apache NiFi at Renault
byDataWorks Summit
 
Getting Started with Confluent Schema Registry
byconfluent
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
bymumrah
 
From Zero to Hero with Kafka Connect
byconfluent
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
byconfluent
 
HBase and HDFS: Understanding FileSystem Usage in HBase
byenissoz
 
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
byFlink Forward
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
Introduction to Apache Kafka
byJeff Holoman
 
Designing Apache Hudi for Incremental Processing With Vinoth Chandar and Etha...
byHostedbyConfluent
 
Apache Kafka Fundamentals for Architects, Admins and Developers
byconfluent
 
Tuning Apache Kafka Connectors for Flink.pptx
byFlink Forward
 
Presto
byKnoldus Inc.
 
Kafka presentation
byMohammed Fazuluddin
 
Apache Spark Architecture
byAlexey Grishchenko
 

Similar to Consumer offset management in Kafka

PDF
Kafkaesque days at linked in in 2015
byJoel Koshy
 
PDF
Nielsen Presents: Fun with Kafka, Spark and Offset Management
bySimona Meriam
 
PDF
Kafka Needs No Keeper
byC4Media
 
PDF
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
byconfluent
 
PDF
Java zone 2015 How to make life with kafka easier.
byKrzysztof Debski
 
PDF
Kafka indexing service
bySeoeun Park
 
PDF
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
byconfluent
 
PDF
Introduction to Apache Kafka
byShiao-An Yuan
 
PPTX
Kafka RealTime Streaming
byViyaan Jhiingade
 
PPTX
Kafka timestamp offset
byDaeMyung Kang
 
PPTX
Kafka blr-meetup-presentation - Kafka internals
byAyyappadas Ravindran (Appu)
 
PPTX
Deep Dive into Apache Kafka
byconfluent
 
PDF
Introduction to apache kafka
bySamuel Kerrien
 
PDF
JDD2015: Make your world event driven - Krzysztof Dębski
byPROIDEA
 
PPTX
Velocity 2019 - Kafka Operations Deep Dive
byGwen (Chen) Shapira
 
PDF
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
byGuozhang Wang
 
PDF
Kafka internals
byDavid Groozman
 
PDF
Kafka used at scale to deliver real-time notifications
bySérgio Nunes
 
PPTX
Putting Kafka Into Overdrive
byTodd Palino
 
PDF
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
byHostedbyConfluent
 
Kafkaesque days at linked in in 2015
byJoel Koshy
 
Nielsen Presents: Fun with Kafka, Spark and Offset Management
bySimona Meriam
 
Kafka Needs No Keeper
byC4Media
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
byconfluent
 
Java zone 2015 How to make life with kafka easier.
byKrzysztof Debski
 
Kafka indexing service
bySeoeun Park
 
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
byconfluent
 
Introduction to Apache Kafka
byShiao-An Yuan
 
Kafka RealTime Streaming
byViyaan Jhiingade
 
Kafka timestamp offset
byDaeMyung Kang
 
Kafka blr-meetup-presentation - Kafka internals
byAyyappadas Ravindran (Appu)
 
Deep Dive into Apache Kafka
byconfluent
 
Introduction to apache kafka
bySamuel Kerrien
 
JDD2015: Make your world event driven - Krzysztof Dębski
byPROIDEA
 
Velocity 2019 - Kafka Operations Deep Dive
byGwen (Chen) Shapira
 
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
byGuozhang Wang
 
Kafka internals
byDavid Groozman
 
Kafka used at scale to deliver real-time notifications
bySérgio Nunes
 
Putting Kafka Into Overdrive
byTodd Palino
 
Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020
byHostedbyConfluent
 

Recently uploaded

PPTX
Data Science in IOT which is utilized by .pptx
byyanKhan
 
PPTX
Orientation-on-ARAL-Check-in-Assessment.pptx
byJenTeruel1
 
PPTX
essential in computer science structure and other
byhncseven7
 
PPTX
GRADE 8 MATH - Q3 - LESSON 1 -LINEAR EQUATION IN ONE VARIABLE.pptx
byEdlynMalangPanti
 
PPTX
491456461_laudon_mis16_ppt_ch11_KL_CE_Updated_Content_for_2021_Managing.pptx
byFantaneshTegegn1
 
PPT
Perceptron and Activation Function in deep learning
byShrideviS7
 
PPTX
[DSC Europe 25] Dobrica Cosic - Savings by the Second: How Dynamic Pricing an...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...
byDataScienceConferenc1
 
PDF
Comprehensive Worksheet: Information Technology (Grade 9) Topics: Unit 1 (Or...
byShafiEsa1
 
PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PDF
[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Dusan Pavlov - There Is No Spoon: Inferring Vision from Neura...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...
byDataScienceConferenc1
 
PDF
BCS501 Module-05 Software Engineering and Project Management.pdf
byjaadu6918
 
PPTX
[DSC Europe 25] Bogdan Daniel Maruneac - AI - It starts with you.pptx
byDataScienceConferenc1
 
PPTX
Anti_Bastos_ Law_ Presentation.pptx
byairesmijares2003
 
PPTX
IT231_Presenation-Template oftentimes-1.pptx
bygoneweldon026
 
PPTX
[DSC Europe 25] Bassam Maharmeh - Artificial Intelligence: Opportunities and ...
byDataScienceConferenc1
 
PPTX
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
Data Science in IOT which is utilized by .pptx
byyanKhan
 
Orientation-on-ARAL-Check-in-Assessment.pptx
byJenTeruel1
 
essential in computer science structure and other
byhncseven7
 
GRADE 8 MATH - Q3 - LESSON 1 -LINEAR EQUATION IN ONE VARIABLE.pptx
byEdlynMalangPanti
 
491456461_laudon_mis16_ppt_ch11_KL_CE_Updated_Content_for_2021_Managing.pptx
byFantaneshTegegn1
 
Perceptron and Activation Function in deep learning
byShrideviS7
 
[DSC Europe 25] Dobrica Cosic - Savings by the Second: How Dynamic Pricing an...
byDataScienceConferenc1
 
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...
byDataScienceConferenc1
 
Comprehensive Worksheet: Information Technology (Grade 9) Topics: Unit 1 (Or...
byShafiEsa1
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...
byDataScienceConferenc1
 
[DSC Europe 25] Dusan Pavlov - There Is No Spoon: Inferring Vision from Neura...
byDataScienceConferenc1
 
[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...
byDataScienceConferenc1
 
BCS501 Module-05 Software Engineering and Project Management.pdf
byjaadu6918
 
[DSC Europe 25] Bogdan Daniel Maruneac - AI - It starts with you.pptx
byDataScienceConferenc1
 
Anti_Bastos_ Law_ Presentation.pptx
byairesmijares2003
 
IT231_Presenation-Template oftentimes-1.pptx
bygoneweldon026
 
[DSC Europe 25] Bassam Maharmeh - Artificial Intelligence: Opportunities and ...
byDataScienceConferenc1
 
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
In this document
Powered by AI

Overview of consumer offset management in Kafka presented by Joel Koshy during a meetup.

Illustrates offset mapping for consumers, including event types and their corresponding offsets.

Details on storing offsets in ZooKeeper, including key components like admin and broker configurations.

Discusses issues associated with storing offsets in ZooKeeper, especially during migration periods.

Outlines the ideal characteristics for offset management: durability, high write-load support, consistency.

Discusses storing offsets in a replicated log, including detailed examples of offset organization.Details on how partitioning in replicated logs works for efficient offset storage and management.

Explains the compaction process for logs storing offsets, ensuring efficient data storage.

Strategies for managing dead consumers by appending tombstones and removing stale offset entries.

Highlights the recommended settings for the offsets topic to ensure reliability and performance.

Details on the process of committing and fetching offsets through consumer instances and brokers.

Illustrates the behavior and interactions of the offset manager when it changes leadership between brokers.Description and structure of the APIs for committing and fetching offsets in Kafka.

Demonstrates methods to read and analyze the offsets topic utilizing console consumers.

Provides guidance on migrating between ZooKeeper and Kafka for offset storage.

Lists vital metrics to monitor for effective tracking of consumer and broker activities.

Lists key updates in Kafka version 0.8.3 that improve offset management and functionality.

Encourages the implementation of monitoring practices for effective management of offsets.

Acknowledges contributions from the Kafka team and specific individuals at Confluent.

Consumer offset management in Kafka

  • 1.
    Consumer offset management inKafka Joel Koshy Kafka meetup @ LinkedIn March 24, 2015
  • 2.
    Consumers and offsets 56 7 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 2 3 2 4 2 5 2 6 2 7 2 8 2 9 3 0 3 1 3 2 3 3 3 4 PageViewEvent-0 EmailBounceEvent-0 PageViewEvent-0 EmailBounceEvent-0 35 12 Offset map GroupId: audit-consumer
  • 3.
    Store offsets inZooKeeper admin brokers config consumers controller controller_epoch audit-consumer ids owners offsets PageViewEvent EmailBounceEvent 0 0 12 35
  • 4.
    (Don’t) store offsetsin ZooKeeper •  Heavy write-load on ZooKeeper •  Especially an issue – during 0.7 to 0.8 migration – and before we switched to SSDs •  Non-ideal work-arounds – Increase offset-commit intervals – Filter commits if offsets have not moved – Spread large offset commits over commit interval
  • 5.
    Offset management (ideals) • Durable •  Support high write-load •  Consistent reads •  Atomic offset commits •  Fast commits/fetches
  • 6.
    Store offsets ina replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets Next commit Group Offset Partition
  • 7.
    Store offsets ina replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets audit-consumer EmailBounceEvent-0 248
  • 8.
    Store offsets ina replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323
  • 9.
    Store offsets ina replicated log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543
  • 10.
    Store offsets ina replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 Partition è abs(GroupId.hashCode()) % NumPartitions
  • 11.
    Store offsets ina replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 Offset commits append to the offsets topic partition Offset fetches read from the offsets topic partition
  • 12.
    Store offsets ina replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 [audit-consumer, PageViewEvent-0] [audit-consumer, EmailBounceEvent-0] [mirrormaker, ClickEvent-0] [mirrormaker, ClickEvent-1] Offsets cache 323 248 54674 54543 Offset commits append to the offsets topic partition + update the cache Offset fetches read from the offsets topic partition cache
  • 13.
    Store offsets ina replicated, partitioned log audit-consumer PageViewEvent-0 240 audit-consumer EmailBounceEvent-0 232 __consumer_offsets, partition 3 audit-consumer EmailBounceEvent-0 248 audit-consumer PageViewEvent-0 323 mirrormaker ClickEvent-0 54543 mirrormaker ClickEvent-1 54444 mirrormaker ClickEvent-1 54674 __consumer_offsets, partition 8 [audit-consumer, PageViewEvent-0] [audit-consumer, EmailBounceEvent-0] [mirrormaker, ClickEvent-0] [mirrormaker, ClickEvent-1] Offsets cache 323 248 54674 54543 Offset commits append to the offsets topic partition + update the cache Offset fetches read from the offsets topic partition cache How do we GC older offset entries?
  • 14.
    log.cleanup.policy = compact 01 2 3 4 5 6 7 8 9 10 K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 V1 V2 V3 V4 V5 V6 V7 V8 V9 V 10 V 11 3 4 K1 K3 V4 V5 6 K4 V7 8 10 K5 K6 V9 V 11 Compaction Offset Key Value 11 K2 Ø Offset Key Value
  • 15.
    Store offsets ina replicated, partitioned, compacted log audit-consumer PageViewEvent-0 126312342 audit-consumer EmailBounceEvent-0 59843 audit-consumer PageViewEvent-0 126319628 audit-consumer EmailBounceEvent-0 86243 audit-consumer PageViewEvent-0 126398102 Key Value audit-consumer EmailBounceEvent-0 86243 audit-consumer PageViewEvent-0 126398102 Compaction Key è [Group, Topic, Partition] Value è Offset
  • 16.
    Dealing with deadconsumers console-consumer-38587, console-consumer-94777, console-consumer-94774, console-consumer-31199, console-consumer-51555, console-consumer-43182, mobileServiceConsumerDwwewewA13dafddesfasdfdee33, console-consumer-57784, python-kafka-consumer-0959a04da7c241448beb0813f002e34b, console- consumer-70750, console-consumer-94809, console-consumer-87470, touch-me-not, console- consumer-43246, console-consumer-69811, python-kafka-consumer-82c2d653128840d5b6bcbfc5ac7f3abc, console-consumer-33847, console-consumer-18217, console-consumer-87493, console-consumer-26414, console-consumer-67299, voldemort-reader-jjkoshy, console-consumer-80245, kafka_listener_for_comments, test-flow-staging, console-consumer-8441, console-consumer-67258, data- processor-2, console-consumer-94869, console-consumer-55242, pinot-beta-hackday_1_2, console- consumer-6601, cloud-host1, system-metrics-monitor-01, console-consumer-70859, console- consumer-26477, page-view-test-flow-2, page-view-test-flow-1, python-kafka-consumer- bf33d075b22d4ddfb82d4a055303e909, console-consumer-99768, console-consumer-45509, console- consumer-21504, points-test_devel_l1_1686489164, console-consumer-14841, console-consumer-4098, console-consumer-14746, console-consumer-94575, cloud-dcb-host147.company.com, teacup_reporting_alex, console-consumer-4132, console-consumer-48171, ropod-dcb-host794.company.com, console-consumer-63743, console-consumer-36147, console-consumer-48138, console-consumer-33595, console-consumer-6808, console-consumer-31000, console-consumer- 73064, console-consumer-18050, console-consumer-21683, share-message, ropod-dcb-host959.company.com, ropod-dcb-host949.company.com, sensei-test_dcb_host138.company.com_1924844804, console-consumer-38654, console-consumer-92040, console-consumer-67052, console-consumer-82690, console-consumer-92002, console-consumer-69687, console-consumer-31077, console-consumer-94657, console-consumer-36064, console-consumer-45675, console-consumer-45671, console-consumer-70625, MemberSettings-dcx, console-consumer-55513, member- links-dcx, console-consumer-85367, opportunist-company, forum-queue, console-consumer-87912, console-consumer-75909, console-consumer-12320, sensei-test_user2_808173709, ropod-dcb- host937.company.com, console-consumer-8710, console-consumer-48390, python-kafka- consumer-816cebafabb34dd5be6bfce59cbee411, console-consumer-8701, console-consumer-6122, console- consumer-6142, metrics-dcb-monitor19, console-consumer-73329, console-consumer-87942, console- consumer-80552, console-consumer-48368, autometrics-dcb-host13, …!
  • 17.
    Dealing with deadconsumers •  For offsets older than offset retention period: – Append tombstone – Remove offset entry from cache
  • 18.
    Recommended settings for offsetstopic Replication factor >= 3 min.insync.replicas >= 2 unclean.leader.election.enable False offsets.commit.required.acks -1 (all)
  • 19.
    How to commit/fetchoffsets audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 V I P Consumer metadata request Response (manager=2)
  • 20.
    How to commit/fetchoffsets audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 Offset fetches Offset commits cache replication
  • 21.
    When the offsetmanager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 cache Become Leader load cache
  • 22.
    When the offsetmanager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) __consumer_offsets-34: Leader: 2, ISR: 0, 1, 2 cache Become Leader load cache Become follower XXXXXX
  • 23.
    When the offsetmanager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) Offset fetches Offset commits cache __consumer_offsets-34: Leader: 0, ISR: 0, 1, 2 cache X X
  • 24.
    When the offsetmanager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) V I P Consumer metadata request cache __consumer_offsets-34: Leader: 0, ISR: 0, 1, 2 cache Response (manager=0)
  • 25.
    When the offsetmanager moves audit-consumer Consumer instance Broker 0 Broker 1 Broker 2 Broker 3 (controller) cache __consumer_offsets-34: Leader: 0, ISR: 0, 1, 2 cache Offset commits Offset fetches replication
  • 26.
    Offset{Commit,Fetch} API ConsumerMetadataRequest o Group Id:String ConsumerMetadataResponse o Error code: Short o Offset manager: Kafka broker info
  • 27.
    Offset{Commit,Fetch} API OffsetCommitRequest o groupId: String o Offsetmap §  Key è Topic-partition §  Value è Partition-data •  Offset: Long •  Timestamp: Long •  Metadata: String KAFKA-1634: changes semantics of timestamp to retention
  • 28.
    Offset{Commit,Fetch} API OffsetCommitResponse o Response map § Key è Topic-partition §  Value è Error code
  • 29.
    Offset{Commit,Fetch} API OffsetFetchRequest o Group Id:String o Partitions: List<Topic-partition> OffsetFetchResponse o Response map §  Key è Topic-partition §  Value è Partition-data •  Offset: Long •  Metadata: String •  Error code: Short
  • 30.
  • 31.
    Offset{Commit,Fetch} API KafkaConsumer<K, V>consumer = new KafkaConsumer<K, V>(properties);! …! TopicPartition partition1 = new TopicPartition("topic1", 0);! TopicPartition partition1 = new TopicPartition("topic1", 1);! ! consumer.subscribe(partition1, partition2);! ! Map<TopicPartition, Long> offsets = new LinkedHashMap<TopicPartition, Long>();! offsets.put(partition1, 123L);! offsets.put(partition2, 4320L);! …! // commit offsets! consumer.commit(offsets, CommitType.SYNC);! …! // fetch offsets! long committedOffset = consumer.committed(partition1);! !
  • 32.
    How to readthe offsets topic To read everything, use the console consumer! ./bin/kafka-console-consumer.sh --topic __consumer_offsets -- zookeeper localhost:2181 --formatter "kafka.server.OffsetManager $OffsetsMessageFormatter" --consumer.config config/ consumer.properties! (Must set exclude.internal.topics = false in consumer.properties) ! To read a single partition, use the simple- consumer-shell ./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets -- partition 12 --broker-list localhost:9092 --formatter "kafka.server.OffsetManager$OffsetsMessageFormatter"!
  • 33.
    Inside the offsetstopic [Group, Topic, Partition]::[Offset, Metadata, Timestamp] [audit-consumer,PageViewEvent,7]::OffsetAndMetadata[53568,NO_METADATA,1416363620711]! [audit-consumer,service-log-event,5]::OffsetAndMetadata[168012,NO_METADATA, 1416363620711]! [audit-consumer,EmailBounceEvent,4]::OffsetAndMetadata[8524676,NO_METADATA, 1416363620711]! [audit-consumer,ClickEvent,0]::OffsetAndMetadata[8132292,NO_METADATA,1416363620711]! [audit-consumer,metrics-event,1]::OffsetAndMetadata[1835900,NO_METADATA,1416363620711]! [audit-consumer,CompanyEvent,0]::OffsetAndMetadata[109337,NO_METADATA,1416363620711]! [audit-consumer,test-topic,1]::OffsetAndMetadata[352989,NO_METADATA,1416363620711]! [audit-consumer,meetup-event,2]::OffsetAndMetadata[39961,NO_METADATA,1416363620711]! [audit-consumer,push-topic,6]::OffsetAndMetadata[4210366,NO_METADATA,1416363620711]!
  • 34.
    How to migrate/roll-back Migratefrom ZooKeeper to Kafka: •  Config change – offsets.storage=kafka – dual.commit.enabled=true •  Rolling bounce •  Config change – dual.commit.enabled=false •  Rolling bounce
  • 35.
    How to migrate/roll-back Migratefrom Kafka to ZooKeeper: •  Config change – dual.commit.enabled=true •  Rolling bounce •  Config change – offsets.storage=zookeeper – dual.commit.enabled=false •  Rolling bounce
  • 36.
    Key metrics tomonitor •  Consumer mbeans –  Kafka commit rate –  ZooKeeper commit rate (during migration) •  Broker mbeans –  Max-dirty ratio and other log cleaner metrics –  Offset cache size –  Group count –  {ConsumerMetadata, OffsetCommit, OffsetFetch} request metrics
  • 37.
    0.8.3 •  Support compressionin compacted topics (KAFKA-1734) •  Change offset commit “timestamp” to mean retention period: KAFKA-1634 •  Offset client
  • 38.
  • 39.
    Acknowledgments Kafka team @LinkedIn Jay Kreps, Jun Rao, Neha Narkhede @ Confluent Tejas (2013 intern): http://lnkdin.me/p/tejaspatil1