confluent, profile picture
Uploaded byconfluent
1,982 views

Kafka 101 and Developer Best Practices

The document provides an overview of Kafka, highlighting its developer best practices and the event streaming paradigm that enables efficient data integration. It details the architecture of Kafka, including producers, consumers, brokers, and topics, and emphasizes the importance of using schemas and configuring producer properties for data consistency and performance. Confluent is positioned as a key player in the enterprise event streaming market, enhancing Kafka's capabilities for large-scale deployments.

Technology◦
Related topics:
Apache Kafka•

Embed presentation

Kafka 101 & Developer Best Practices
Agenda ● Kafka Overview ● Kafka 101 ● Best Practices for Writing to Kafka: A tour of the Producer ● Best Practices for Reading from Kafka: The Consumer ● General Considerations
3 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
4 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
5 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records
6 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records Both of these are a complete mismatch to how your business works.
7 ETL/Data Integration Messaging Transient Messages Stored records ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm High Throughput Durable Persistent Maintains Order Fast (Low Latency)
8 Fast (Low Latency) Event Streaming Paradigm To rethink data as not stored records or transient messages, but instead as a continually updating stream of events
9 Fast (Low Latency) Event Streaming Paradigm
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. ... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Event-Streaming Applications Universal Event Pipeline Amazon S3 SaaS apps Confluent: Central Nervous System For Enterprise
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Confluent uniquely enables Event Streaming success Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka commits and has over 1M hours technical experience with Kafka Confluent helps enterprises successfully deploy event streaming at scale and accelerate time to market Confluent Platform extends Apache Kafka to be a secure, enterprise-ready platform
Kafka 101
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Scalability of a Filesystem Guarantees of a Database Distributed By Design Rewind and Replay
15 Kafka Broker Kafka Broker Kafka Broker Kafka Broker Kafka Producer Kafka Producer Kafka Producer Kafka Consumer Kafka Consumer (grouped) Kafka Consumer /Producer Kafka Broker Writers Readers Kafka Cluster
KAFKA A MODERN, DISTRIBUTED PLATFORM FOR DATA STREAMS
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Creating a topic $ kafka-topics --zookeeper zk:2181 --create --topic my-topic --replication-factor 3 --partitions 3 Or use the AdminClient API!
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time C C C
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
Producing to Kafka
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Clients - Producer Design Producer Record Topic [Partition] [Key] Value Serializer Partitioner Topic A Partition 0 Batch 0 Batch 1 Batch 2 Topic B Partition 1 Batch 0 Batch 1 Batch 2 Kafka Broker Send() Retry ? Fail ? Yes No Can’t retry, throw exception Success: return metadata Yes
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer Kafka doesn’t care about what you send to it as long as it’s been converted to a byte stream beforehand. JSON CSV Avro Protobufs XML SERIALIZERS 01001010 01010011 01001111 01001110 01000011 01010011 01010110 01001010 01010011 01001111 01001110 01010000 01110010 01101111 01110100 ... 01011000 01001101 01001100 (if you must) Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer private Properties kafkaProps = new Properties(); kafkaProps.put(“bootstrap.servers”, “broker1:9092,broker2:9092”); kafkaProps.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”); kafkaProps.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); kafkaProps.put("schema.registry.url", "https://schema-registry:8083"); producer = new KafkaProducer<String, SpecificRecord>(kafkaProps); Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Record Topic [Partition] [Key] Value Record keys determine the partition with the default kafka partitioner If a key isn’t provided, messages will be produced in a round robin fashion partitioner Record Keys and why they’re important - Ordering
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] AAA Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] BBB Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] CCC Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] DDD Value partitioner Record keys determine the partition with the default kafka partitioner
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Record Keys and why they’re important - Key Cardinality Consumers Key cardinality affects the amount of work done by the individual consumers in a group. Poor key choice can lead to uneven workloads. Keys in Kafka don’t have to be primitives, like strings or ints. Like values, they can be be anything: JSON, Avro, etc… So create a key that will evenly distribute groups of records around the partitions. Car·di·nal·i·ty /ˌkärdəˈnalədē/ Noun the number of elements in a set or other grouping, as a property of that grouping.
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101” } You don’t have to but... use a Schema! Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send JSON “Where’s record.City?” Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic ! Schema Registry Open Source Feature
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101”, “City”: “NA”, “State”: “NA” } Avro allows for evolution of schemas Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send AvroRecord Schema Registry Version 1 Version 2 Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
Developing with Confluent Schema Registry We provide several Maven plugins for developing with the Confluent Schema Registry ● download - download a subject’s schema to your project ● register - register a new schema to the schema registry from your development env ● test-compatibility - test changes made to a schema against compatibility rules set by the schema registry Reference https://docs.confluent.io/current/schema-registry/docs/maven-plugin.html <plugin> <groupId>io.confluent</groupId> <artifactId>kafka-schema-registry-maven-plug <version>5.0.0</version> <configuration> <schemaRegistryUrls> <param>http://192.168.99.100:808 </schemaRegistryUrls> <outputDirectory>src/main/avro</outp <subjectPatterns> <param>^TestSubject000-(key|valu </subjectPatterns> </configuration> </plugin>
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Use Kafka’s Headers Reference https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers Producer Record Topic [Partition] [Timestamp] Value [Headers] [Key] Kafka Headers are simply an interface that requires a key of type String, and a value of type byte[], the headers are stored in an iterator in the ProducerRecord . Example Use Cases ● Data lineage: reference previous topic partition/offsets ● Producing host/application/owner ● Message routing ● Encryption metadata (which key pair was this message payload encrypted with?)
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=0 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 ack Producer Properties acks=1 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234 data: abcd} - offset 3345 Failed ack Successful write Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234, data: abcd} - offset 3345 {key: 1234, data: abcd} - offset 3346 retry ack dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - with exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties enable.idempotence=true max.inflight.requests.per.connection=5 acks = “all” retries > 0 (preferably MAX_INT) (pid, seq) [payload] (100, 1) {key: 1234, data: abcd} - offset 3345 (100, 1) {key: 1234, data: abcd} - rejected, ack re-sent (100, 2) {key: 5678, data: efgh} - offset 3346 retry ack no dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
Transactional Producer Producer T1 T1 T1 T1 T1 KafkaProducer producer = createKafkaProducer( “bootstrap.servers”, “broker:9092”, “transactional.id”, “my-transactional-id”); producer.initTransactions(); -- send some records -- producer.commitTransaction(); Consumer KafkaConsumer consumer = createKafkaConsumer( “bootstrap.servers”, “broker:9092”, “group.id”, “my-group-id”, "isolation.level", "read_committed"); Reference https://www.confluent.io/blog/transactions-apache-kafka/
Consuming from Kafka
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. A basic Java Consumer final Consumer<String, String> consumer = new KafkaConsumer<String, String>(props); consumer.subscribe(Arrays.asList(topic)); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(100); for (ConsumerRecord<String, String> record : records) { -- Do Some Work -- } } } finally { consumer.close(); } }
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Single Consumer C
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C1 C C C2
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C C
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0, 3 1 2 3
Kafka’s Interceptors ProducerInterceptor onSend(ProducerRecord<K, V> record) Returns ProducerRecord<K, V> . Called from send() before key and value get serialized and partition is assigned. This method is allowed to modify the record. onAcknowledgement(RecordMetadata metadata, java.lang.Exception exception) This method is called when the record sent to the server has been acknowledged, or when sending the record fails before it gets sent to the server. Used for observability and reporting. ConsumerInterceptor onConsume(ConsumerRecords<K,V> records) Called just before the records are returned by KafkaConsumer.poll() This method is allowed to modify consumer records, in which case the new records will be returned. onCommit(Map<TopicPartition,OffsetAndMetada ta> offsets) This is called when offsets get committed. Used for observability and reporting Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/producer/ProducerInterceptor.html Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/consumer/ConsumerInterceptor.html
Should I pool connections? NO! Since Kafka connections are long-lived, there is no reason to pool connections. It’s common to keep one connection per thread.
Use a good client! Clients ● Java/Scala - default clients, comes with Kafka ● C/C++ - https://github.com/edenhill/librdkafka ● C#/.Net - https://github.com/confluentinc/confluent-kafka-dotnet ● Python - https://github.com/confluentinc/confluent-kafka-python ● Golang - https://github.com/confluentinc/confluent-kafka-go ● Node/JavaScript - https://github.com/Blizzard/node-rdkafka (not supported by Confluent!) New Kafka features will only be available to modern, updated clients!
Resources Free E-Books from Confluent! I Heart Logs: https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/ Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/ Designing Event Driven Systems: https://www.confluent.io/designing-event-driven-systems/ Confluent Blog: https://www.confluent.io/blog Thank You!
Thank you! pascal@confluent.io

More Related Content

PDF
Benefits of Stream Processing and Apache Kafka Use Cases
byconfluent
 
PDF
ksqlDB: A Stream-Relational Database System
byconfluent
 
PDF
Schema Registry 101 with Bill Bejeck | Kafka Summit London 2022
byHostedbyConfluent
 
PPTX
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
ODP
Stream processing using Kafka
byKnoldus Inc.
 
PDF
Introduction to Kafka Streams
byGuozhang Wang
 
PDF
Apache Kafka Introduction
byAmita Mirajkar
 
PPTX
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
byJean-Paul Azar
 
Benefits of Stream Processing and Apache Kafka Use Cases
byconfluent
 
ksqlDB: A Stream-Relational Database System
byconfluent
 
Schema Registry 101 with Bill Bejeck | Kafka Summit London 2022
byHostedbyConfluent
 
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
Stream processing using Kafka
byKnoldus Inc.
 
Introduction to Kafka Streams
byGuozhang Wang
 
Apache Kafka Introduction
byAmita Mirajkar
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
byJean-Paul Azar
 

What's hot

PPTX
Kafka 101
byClement Demonchy
 
PPTX
Apache kafka
byViswanath J
 
PPTX
Kafka 101
byAparna Pillai
 
PDF
Fundamentals of Apache Kafka
byChhavi Parasher
 
PDF
Apache kafka
byNexThoughts Technologies
 
PPTX
Introduction to Apache Kafka
byAIMDek Technologies
 
PPTX
Deep Dive into Apache Kafka
byconfluent
 
PDF
Kafka Streams: What it is, and how to use it?
byconfluent
 
PDF
An Introduction to Apache Kafka
byAmir Sedighi
 
PPTX
Kafka basics
byJoão Paulo Leonidas Fernandes Dias da Silva
 
PDF
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
PPTX
Apache kafka
byLong Nguyen
 
PPTX
Kafka presentation
byMohammed Fazuluddin
 
PPTX
Introduction to Apache Kafka
byJeff Holoman
 
PPTX
Introduction to Kafka Cruise Control
byJiangjie Qin
 
PPTX
Apache Kafka
byemreakis
 
PDF
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
PDF
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
byFlink Forward
 
PPTX
Apache Kafka 0.8 basic training - Verisign
byMichael Noll
 
PPTX
A visual introduction to Apache Kafka
byPaul Brebner
 
Kafka 101
byClement Demonchy
 
Apache kafka
byViswanath J
 
Kafka 101
byAparna Pillai
 
Fundamentals of Apache Kafka
byChhavi Parasher
 
Apache kafka
byNexThoughts Technologies
 
Introduction to Apache Kafka
byAIMDek Technologies
 
Deep Dive into Apache Kafka
byconfluent
 
Kafka Streams: What it is, and how to use it?
byconfluent
 
An Introduction to Apache Kafka
byAmir Sedighi
 
Kafka basics
byJoão Paulo Leonidas Fernandes Dias da Silva
 
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
Apache kafka
byLong Nguyen
 
Kafka presentation
byMohammed Fazuluddin
 
Introduction to Apache Kafka
byJeff Holoman
 
Introduction to Kafka Cruise Control
byJiangjie Qin
 
Apache Kafka
byemreakis
 
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
byFlink Forward
 
Apache Kafka 0.8 basic training - Verisign
byMichael Noll
 
A visual introduction to Apache Kafka
byPaul Brebner
 

Similar to Kafka 101 and Developer Best Practices

PDF
What is Apache Kafka and What is an Event Streaming Platform?
byconfluent
 
PPTX
Westpac Bank Tech Talk 1: Dive into Apache Kafka
byconfluent
 
PDF
Devoxx university - Kafka de haut en bas
byFlorent Ramiere
 
PDF
Virtual Bash! A Lunchtime Introduction to Kafka
byJason Bell
 
PDF
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
PPTX
Kafka overview v0.1
byMahendran Ponnusamy
 
PDF
Kafka Deep Dive
byKnoldus Inc.
 
PDF
Apache KAfka
byPedro Alcantara
 
PDF
Developing Realtime Data Pipelines With Apache Kafka
byJoe Stein
 
PPTX
Apache kafka
byKumar Shivam
 
PDF
Apache Kafka - From zero to hero
byApache Kafka TLV
 
PDF
Kafka zero to hero
byAvi Levi
 
PDF
Kafka syed academy_v1_introduction
bySyed Hadoop
 
PDF
DevOps Fest 2020. Сергій Калінець. Building Data Streaming Platform with Apac...
byDevOps_Fest
 
PDF
Apache Kafka Scalable Message Processing and more!
byGuido Schmutz
 
PPTX
Fundamentals and Architecture of Apache Kafka
byAngelo Cesaro
 
PDF
Kafka online course | Kafka training
byAccentfuture
 
PPTX
Streaming in Practice - Putting Apache Kafka in Production
byconfluent
 
PDF
Apache Kafka Women Who Code Meetup
bySnehal Nagmote
 
PDF
Introduction to Kafka and Event-Driven
byarconsis
 
What is Apache Kafka and What is an Event Streaming Platform?
byconfluent
 
Westpac Bank Tech Talk 1: Dive into Apache Kafka
byconfluent
 
Devoxx university - Kafka de haut en bas
byFlorent Ramiere
 
Virtual Bash! A Lunchtime Introduction to Kafka
byJason Bell
 
Apache Kafka - Scalable Message Processing and more!
byGuido Schmutz
 
Kafka overview v0.1
byMahendran Ponnusamy
 
Kafka Deep Dive
byKnoldus Inc.
 
Apache KAfka
byPedro Alcantara
 
Developing Realtime Data Pipelines With Apache Kafka
byJoe Stein
 
Apache kafka
byKumar Shivam
 
Apache Kafka - From zero to hero
byApache Kafka TLV
 
Kafka zero to hero
byAvi Levi
 
Kafka syed academy_v1_introduction
bySyed Hadoop
 
DevOps Fest 2020. Сергій Калінець. Building Data Streaming Platform with Apac...
byDevOps_Fest
 
Apache Kafka Scalable Message Processing and more!
byGuido Schmutz
 
Fundamentals and Architecture of Apache Kafka
byAngelo Cesaro
 
Kafka online course | Kafka training
byAccentfuture
 
Streaming in Practice - Putting Apache Kafka in Production
byconfluent
 
Apache Kafka Women Who Code Meetup
bySnehal Nagmote
 
Introduction to Kafka and Event-Driven
byarconsis
 

More from confluent

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
PDF
Migration, backup and restore made easy using Kannika
byconfluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
PDF
Unlocking value with event-driven architecture by Confluent
byconfluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
PDF
Building API data products on top of your real-time data infrastructure
byconfluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
Migration, backup and restore made easy using Kannika
byconfluent
 
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
Unlocking value with event-driven architecture by Confluent
byconfluent
 
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
Building API data products on top of your real-time data infrastructure
byconfluent
 
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 

Recently uploaded

PPTX
IBM_NEXA_DAC2021 NEXA, a cloud-native platform for collaborative hardware log...
byArun Joseph
 
PDF
DataLiva-LivaGRC- Business Software for Governance, Risk, and Compliance
byMustafa Kuğu
 
PDF
Higgsfield AI: The New Way to Create Cinematic Video
bytechnologyupside
 
PPTX
02_Extended Warehouse Management Functions and Features.pptx
byUdayakumar218983
 
PDF
Presentation on Four Stroke Engine of Ic engine
bySonam Sherpa
 
PDF
Digital Transformation Services The Key to Futureproofing Your Business
byQuadrafort Technolgies
 
PDF
Master the FME Connector for ArcGIS: Streamlined Data Management & Robust Met...
bySafe Software
 
PPTX
Unit 1 Introduction of Computer Networks
bySuvarnaSharma5
 
PDF
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
PPTX
Lecture 05. API Security for DevSecOps Eng.pptx
byvinocol222
 
PDF
Best-Travel-Portal-Development-Solutions-for-Online-Growth
bykashishnagarrajput
 
PDF
SEO in Charleston SC | Local SEO Strategies to Grow Your Business.pdf
bycustomdigitalsolutio1
 
PPTX
Design Flaws and Known Attacks in Agentic AI - ITI Business Session
byInformation Technology Institute
 
PDF
Requestly or Postman: What’s the Right API Tool for Your Team?
byhenrycavill30521
 
PDF
JR : Quality Random Data from the Command line
byGiovanni Marigi
 
PDF
Chapter 2 Computer Threat .pdf
byGetnet Tigabie Askale -(GM)
 
PDF
Using Change Data Capture (CDC) to Ingest Data into Apache Kafka
byGiovanni Marigi
 
PDF
SAP WalkMe Overview.pdf SAP WalkMe Overview.pdf
byMurniatiSimbolon2
 
PDF
Josh Chu Boston - An Innovative Technology Executive
byJosh Chu Boston
 
PPTX
On-Device AI with Gemma 3n and PaliGemma 2 Session Slides - GDG VESIT
bygdgcodecellcmpn
 
IBM_NEXA_DAC2021 NEXA, a cloud-native platform for collaborative hardware log...
byArun Joseph
 
DataLiva-LivaGRC- Business Software for Governance, Risk, and Compliance
byMustafa Kuğu
 
Higgsfield AI: The New Way to Create Cinematic Video
bytechnologyupside
 
02_Extended Warehouse Management Functions and Features.pptx
byUdayakumar218983
 
Presentation on Four Stroke Engine of Ic engine
bySonam Sherpa
 
Digital Transformation Services The Key to Futureproofing Your Business
byQuadrafort Technolgies
 
Master the FME Connector for ArcGIS: Streamlined Data Management & Robust Met...
bySafe Software
 
Unit 1 Introduction of Computer Networks
bySuvarnaSharma5
 
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
Lecture 05. API Security for DevSecOps Eng.pptx
byvinocol222
 
Best-Travel-Portal-Development-Solutions-for-Online-Growth
bykashishnagarrajput
 
SEO in Charleston SC | Local SEO Strategies to Grow Your Business.pdf
bycustomdigitalsolutio1
 
Design Flaws and Known Attacks in Agentic AI - ITI Business Session
byInformation Technology Institute
 
Requestly or Postman: What’s the Right API Tool for Your Team?
byhenrycavill30521
 
JR : Quality Random Data from the Command line
byGiovanni Marigi
 
Chapter 2 Computer Threat .pdf
byGetnet Tigabie Askale -(GM)
 
Using Change Data Capture (CDC) to Ingest Data into Apache Kafka
byGiovanni Marigi
 
SAP WalkMe Overview.pdf SAP WalkMe Overview.pdf
byMurniatiSimbolon2
 
Josh Chu Boston - An Innovative Technology Executive
byJosh Chu Boston
 
On-Device AI with Gemma 3n and PaliGemma 2 Session Slides - GDG VESIT
bygdgcodecellcmpn
 

Kafka 101 and Developer Best Practices

  • 1.
    Kafka 101 & DeveloperBest Practices
  • 2.
    Agenda ● Kafka Overview ●Kafka 101 ● Best Practices for Writing to Kafka: A tour of the Producer ● Best Practices for Reading from Kafka: The Consumer ● General Considerations
  • 3.
    3 ETL/Data Integration Messaging Batch Expensive TimeConsuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
  • 4.
    4 ETL/Data Integration Messaging Batch Expensive TimeConsuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency)
  • 5.
    5 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficultto Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records
  • 6.
    6 ETL/Data Integration Batch Expensive Time Consuming Messaging Difficultto Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Transient Messages Stored records Both of these are a complete mismatch to how your business works.
  • 7.
    7 ETL/Data Integration Messaging TransientMessages Stored records ETL/Data Integration Messaging Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay High Throughput Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm High Throughput Durable Persistent Maintains Order Fast (Low Latency)
  • 8.
    8 Fast (Low Latency) EventStreaming Paradigm To rethink data as not stored records or transient messages, but instead as a continually updating stream of events
  • 9.
    9 Fast (Low Latency) EventStreaming Paradigm
  • 10.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. ... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Event-Streaming Applications Universal Event Pipeline Amazon S3 SaaS apps Confluent: Central Nervous System For Enterprise
  • 11.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Confluent uniquely enables Event Streaming success Hall of Innovation CTO Innovation Award Winner 2019 Enterprise Technology Innovation AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka commits and has over 1M hours technical experience with Kafka Confluent helps enterprises successfully deploy event streaming at scale and accelerate time to market Confluent Platform extends Apache Kafka to be a secure, enterprise-ready platform
  • 12.
  • 13.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
  • 14.
    Scalability of a Filesystem Guaranteesof a Database Distributed By Design Rewind and Replay
  • 15.
    15 Kafka Broker Kafka Broker KafkaBroker Kafka Broker Kafka Producer Kafka Producer Kafka Producer Kafka Consumer Kafka Consumer (grouped) Kafka Consumer /Producer Kafka Broker Writers Readers Kafka Cluster
  • 16.
  • 17.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3
  • 18.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Creating a topic $ kafka-topics --zookeeper zk:2181 --create --topic my-topic --replication-factor 3 --partitions 3 Or use the AdminClient API!
  • 19.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time
  • 20.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producing to Kafka Time C C C
  • 21.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
  • 22.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Kafka’s distributed nature Broker 1 Topic-1 partition-1 Broker 2 Broker 3 Broker 4 Topic-1 partition-1 Topic-1 partition-1 Leader Follower Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-2 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-3 Topic-1 partition-3 Topic-1 partition-4 Topic-1 partition-4
  • 23.
  • 24.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Clients - Producer Design Producer Record Topic [Partition] [Key] Value Serializer Partitioner Topic A Partition 0 Batch 0 Batch 1 Batch 2 Topic B Partition 1 Batch 0 Batch 1 Batch 2 Kafka Broker Send() Retry ? Fail ? Yes No Can’t retry, throw exception Success: return metadata Yes
  • 25.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer Kafka doesn’t care about what you send to it as long as it’s been converted to a byte stream beforehand. JSON CSV Avro Protobufs XML SERIALIZERS 01001010 01010011 01001111 01001110 01000011 01010011 01010110 01001010 01010011 01001111 01001110 01010000 01110010 01101111 01110100 ... 01011000 01001101 01001100 (if you must) Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
  • 26.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. The Serializer private Properties kafkaProps = new Properties(); kafkaProps.put(“bootstrap.servers”, “broker1:9092,broker2:9092”); kafkaProps.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”); kafkaProps.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); kafkaProps.put("schema.registry.url", "https://schema-registry:8083"); producer = new KafkaProducer<String, SpecificRecord>(kafkaProps); Reference https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html
  • 27.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Record Topic [Partition] [Key] Value Record keys determine the partition with the default kafka partitioner If a key isn’t provided, messages will be produced in a round robin fashion partitioner Record Keys and why they’re important - Ordering
  • 28.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] AAA Value partitioner Record keys determine the partition with the default kafka partitioner
  • 29.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] BBB Value partitioner Record keys determine the partition with the default kafka partitioner
  • 30.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] CCC Value partitioner Record keys determine the partition with the default kafka partitioner
  • 31.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Keys are used in the default partitioning algorithm: partition = hash(key) % numPartitions Record Keys and why they’re important - Ordering Producer Record Topic [Partition] DDD Value partitioner Record keys determine the partition with the default kafka partitioner
  • 32.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Record Keys and why they’re important - Key Cardinality Consumers Key cardinality affects the amount of work done by the individual consumers in a group. Poor key choice can lead to uneven workloads. Keys in Kafka don’t have to be primitives, like strings or ints. Like values, they can be be anything: JSON, Avro, etc… So create a key that will evenly distribute groups of records around the partitions. Car·di·nal·i·ty /ˌkärdəˈnalədē/ Noun the number of elements in a set or other grouping, as a property of that grouping.
  • 33.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101” } You don’t have to but... use a Schema! Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send JSON “Where’s record.City?” Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
  • 34.
    Schema Registry: MakeData Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic ! Schema Registry Open Source Feature
  • 35.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. { “Name”: “John Smith”, “Address”: “123 Apple St.”, “Zip”: “19101”, “City”: “NA”, “State”: “NA” } Avro allows for evolution of schemas Data Producer Service Data Consumer Service { “Name”: “John Smith”, “Address”: “123 Apple St.”, “City”: “Philadelphia”, “State”: “PA”, “Zip”: “19101” } send AvroRecord Schema Registry Version 1 Version 2 Reference https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you -really-need-one/
  • 36.
    Developing with ConfluentSchema Registry We provide several Maven plugins for developing with the Confluent Schema Registry ● download - download a subject’s schema to your project ● register - register a new schema to the schema registry from your development env ● test-compatibility - test changes made to a schema against compatibility rules set by the schema registry Reference https://docs.confluent.io/current/schema-registry/docs/maven-plugin.html <plugin> <groupId>io.confluent</groupId> <artifactId>kafka-schema-registry-maven-plug <version>5.0.0</version> <configuration> <schemaRegistryUrls> <param>http://192.168.99.100:808 </schemaRegistryUrls> <outputDirectory>src/main/avro</outp <subjectPatterns> <param>^TestSubject000-(key|valu </subjectPatterns> </configuration> </plugin>
  • 37.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Use Kafka’s Headers Reference https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers Producer Record Topic [Partition] [Timestamp] Value [Headers] [Key] Kafka Headers are simply an interface that requires a key of type String, and a value of type byte[], the headers are stored in an iterator in the ProducerRecord . Example Use Cases ● Data lineage: reference previous topic partition/offsets ● Producing host/application/owner ● Message routing ● Encryption metadata (which key pair was this message payload encrypted with?)
  • 38.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=0 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 39.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 ack Producer Properties acks=1 Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 40.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
  • 41.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 ack
  • 42.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234 data: abcd} - offset 3345 Failed ack Successful write Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 43.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - without exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties acks=all min.insync.replica=2 {key: 1234, data: abcd} - offset 3345 {key: 1234, data: abcd} - offset 3346 retry ack dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 44.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Producer Guarantees - with exactly once guarantees P Broker 1 Broker 2 Broker 3 Topic1 partition1 Leader Follower Topic1 partition1 Topic1 partition1 Producer Properties enable.idempotence=true max.inflight.requests.per.connection=5 acks = “all” retries > 0 (preferably MAX_INT) (pid, seq) [payload] (100, 1) {key: 1234, data: abcd} - offset 3345 (100, 1) {key: 1234, data: abcd} - rejected, ack re-sent (100, 2) {key: 5678, data: efgh} - offset 3346 retry ack no dupe! Reference https://www.confluent.io/blog/exactly-once-semantics-are-p ossible-heres-how-apache-kafka-does-it/
  • 45.
    Transactional Producer Producer T1 T1T1 T1 T1 KafkaProducer producer = createKafkaProducer( “bootstrap.servers”, “broker:9092”, “transactional.id”, “my-transactional-id”); producer.initTransactions(); -- send some records -- producer.commitTransaction(); Consumer KafkaConsumer consumer = createKafkaConsumer( “bootstrap.servers”, “broker:9092”, “group.id”, “my-group-id”, "isolation.level", "read_committed"); Reference https://www.confluent.io/blog/transactions-apache-kafka/
  • 46.
  • 47.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. A basic Java Consumer final Consumer<String, String> consumer = new KafkaConsumer<String, String>(props); consumer.subscribe(Arrays.asList(topic)); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(100); for (ConsumerRecord<String, String> record : records) { -- Do Some Work -- } } } finally { consumer.close(); } }
  • 48.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Single Consumer C
  • 49.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C1 C C C2
  • 50.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers C C C C
  • 51.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
  • 52.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0 1 2 3
  • 53.
    Copyright 2021, Confluent,Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. Consuming From Kafka - Grouped Consumers 0, 3 1 2 3
  • 54.
    Kafka’s Interceptors ProducerInterceptor onSend(ProducerRecord<K, V>record) Returns ProducerRecord<K, V> . Called from send() before key and value get serialized and partition is assigned. This method is allowed to modify the record. onAcknowledgement(RecordMetadata metadata, java.lang.Exception exception) This method is called when the record sent to the server has been acknowledged, or when sending the record fails before it gets sent to the server. Used for observability and reporting. ConsumerInterceptor onConsume(ConsumerRecords<K,V> records) Called just before the records are returned by KafkaConsumer.poll() This method is allowed to modify consumer records, in which case the new records will be returned. onCommit(Map<TopicPartition,OffsetAndMetada ta> offsets) This is called when offsets get committed. Used for observability and reporting Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/producer/ProducerInterceptor.html Reference https://kafka.apache.org/20/javadoc/org/apache/kafka/clien ts/consumer/ConsumerInterceptor.html
  • 55.
    Should I poolconnections? NO! Since Kafka connections are long-lived, there is no reason to pool connections. It’s common to keep one connection per thread.
  • 56.
    Use a goodclient! Clients ● Java/Scala - default clients, comes with Kafka ● C/C++ - https://github.com/edenhill/librdkafka ● C#/.Net - https://github.com/confluentinc/confluent-kafka-dotnet ● Python - https://github.com/confluentinc/confluent-kafka-python ● Golang - https://github.com/confluentinc/confluent-kafka-go ● Node/JavaScript - https://github.com/Blizzard/node-rdkafka (not supported by Confluent!) New Kafka features will only be available to modern, updated clients!
  • 57.
    Resources Free E-Books fromConfluent! I Heart Logs: https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/ Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/ Designing Event Driven Systems: https://www.confluent.io/designing-event-driven-systems/ Confluent Blog: https://www.confluent.io/blog Thank You!
  • 58.