SlideShare a Scribd company logo
Paris Apache Kafka
Meetup
Florian HUSSONNOIS
Zenika
@fhussonnois
Async, Sync, Batch, Partitioner et Retries
Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
KafkaProducer<String, String> producer = new KafkaProducer(config);
ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value");
producer.send(record);
producer.close();
L’appel à la méthode send()est asynchrone et retourne immédiatement
Le message est ajouté à un buffer avant d’être envoyé
//...
config.put("batch.size", 16384);
config.put("linger.ms", 1);
//... Latence entre chaque transmission de messages
Taille maximum d’un batch
List<ProducerRecord> batchRecords = new ArrayList<>();
//...
for(ProducerRecord record : batchRecords)
producer.send(record);
producer.flush();
producer.close(); Force l’envoi des messages et bloque jusqu’à leur
complétion
Future<RecordMetadata> future = producer.send(record);
RecordMetadata metadata = future.get(); // BLOCK
LOG.info("message sent to topic {}, partition {}, offset {}",
metadata.topic(),
metadata.partition(),
metadata.offset());
ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value");
Future<RecordMetadata> future = producer.send(record, (metadata, e) -> {
if(e != null)
LOG.info("Message sent to topic {}, partition {}, offset {}",
metadata.topic(),
metadata.partition(),
metadata.offset());
else
LOG.error("Damn it!", e);
});
Configuration
config.put("partitioner.class", DefaultPartitioner.class.getName()
Implémenter un Partitioner
public interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster);
}
Spécifier directement la partition cible
new ProducerRecord("my_topic", 0, "my_key", "my_value");
Acknowledgments
config.put("ack", "all "); // plus lent, messages répliqués par tous les ISR
Le Producer peut rejouer automatiquement les messages en erreurs
config.put("retries", "0 "); // désactivé
/! Peut provoquer des doublons (At-Least Once)
/! Peut changer l’ordre de publication des messages
Event Loop, Polling Model, Offset et Group
Management
Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"));
while(true) {
ConsumerRecords<Object, Object> records = consumer.poll(1000);
records.forEach(record ->
LOG.info("key={}, value={}", record.key(), record.value()));
}
Event Loop, Polling Model
Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
config.put("enable.auto.commit", false); // désactive auto-commit
config.put("auto.commit.interval.ms", 100);
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"));
while(true) {
ConsumerRecords<Object, Object> records = consumer.poll(1000);
records.forEach(record ->
LOG.info("key={}, value={}", record.key(), record.value()));
consumer.commitAsync();
}
}
while(true) {
ConsumerRecords<Object, Object> records = consumer.poll(1000);
consumer.commitSync(); // Commit offsets before processing messages.
records.forEach(record ->
LOG.info("key={}, value={}", record.key(), record.value()));
}
Paris Kafka Meetup - How to develop with Kafka
Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
config.put("group.id", "my_group");
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"));
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"), new
ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
//do some stuff
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
//do some stuff
}
});
Chaque consumer d’un groupe doit notifier le coordinateur
Uniquement possible sur un appel aux méthodes poll, commit, etc.
Déclenché si un consumer rejoint ou quitte un groupe
L’opération de « rebalance » est impactée par les paramètres :
• session.timeout.ms (30 secondes)
• heartbeat.interval.ms
Rebalance intempestif en cas de traitement d’un message trop long
ConsumerRecords<Object, Object> records = consumer.poll(1000);
if( ! records.isEmpty() ) {
consumer.pause(consumer.assignment().toArray(new TopicPartition[0]));
Future<Boolean> future = executorService.submit(() -> {
records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value()));
return true;
});
Boolean isCompleted = false;
while(!isCompleted) {
try {
isCompleted = future.get(5, TimeUnit.SECONDS); // Wait before polling
} catch (TimeoutException e) {
consumer.poll(0); // heart-beat
} catch (CancellationException |ExecutionException | InterruptedException e) {
break;
}
}
consumer.resume(consumer.assignment().toArray(new TopicPartition[0]));
consumer.commitSync();
}
ExecutorService
Se positionner à un offset spécifique
consumer.seek(new TopicPartition("my_topic", 0), 42);
consumer.seekToEnd(new TopicPartition("my_topic", 0));
consumer.seekToBeginning(new TopicPartition("my_topic", 0));
Assignements manuel
consumer.assign(Arrays.asList(new TopicPartition("my_topic", 0)));
Obtenir les métriques
consumer.metrics();
Nous recrutons ! jobs@zenika.com
@ZenikaIT
Prochain Meetup le

More Related Content

What's hot

Programming with Python and PostgreSQL
Programming with Python and PostgreSQLProgramming with Python and PostgreSQL
Programming with Python and PostgreSQL
Peter Eisentraut
 
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Tom Croucher
 
What's new in Ansible 2.0
What's new in Ansible 2.0What's new in Ansible 2.0
What's new in Ansible 2.0
Allan Denot
 
Lambda Jam 2015: Event Processing in Clojure
Lambda Jam 2015: Event Processing in ClojureLambda Jam 2015: Event Processing in Clojure
Lambda Jam 2015: Event Processing in Clojure
Andy Marks
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queue
Brandon Lamb
 
Hopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to anotherHopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to another
Michele Orselli
 
Lightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRLightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMR
Shinji Tanaka
 
Redis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationRedis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your application
rjsmelo
 
Winform
WinformWinform
Winform
quocphu199
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
Tim Bunce
 
Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2
Cong Zhang
 
MySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELK
YoungHeon (Roy) Kim
 
Using Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasetsUsing Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasets
Bartosz Konieczny
 
More than syntax
More than syntaxMore than syntax
More than syntax
Wooga
 
Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018
Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018
Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018
Codemotion
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
Bartosz Konieczny
 
Elk with Openstack
Elk with OpenstackElk with Openstack
Elk with Openstack
Arun prasath
 
glance replicator
glance replicatorglance replicator
glance replicator
irix_jp
 
Unqlite
UnqliteUnqlite
2015 05 27 JSConf - concurrency and parallelism final
2015 05 27   JSConf - concurrency and parallelism final2015 05 27   JSConf - concurrency and parallelism final
2015 05 27 JSConf - concurrency and parallelism final
Naveed Ihsanullah
 

What's hot (20)

Programming with Python and PostgreSQL
Programming with Python and PostgreSQLProgramming with Python and PostgreSQL
Programming with Python and PostgreSQL
 
Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012 Streams are Awesome - (Node.js) TimesOpen Sep 2012
Streams are Awesome - (Node.js) TimesOpen Sep 2012
 
What's new in Ansible 2.0
What's new in Ansible 2.0What's new in Ansible 2.0
What's new in Ansible 2.0
 
Lambda Jam 2015: Event Processing in Clojure
Lambda Jam 2015: Event Processing in ClojureLambda Jam 2015: Event Processing in Clojure
Lambda Jam 2015: Event Processing in Clojure
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queue
 
Hopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to anotherHopping in clouds: a tale of migration from one cloud provider to another
Hopping in clouds: a tale of migration from one cloud provider to another
 
Lightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMRLightweight wrapper for Hive on Amazon EMR
Lightweight wrapper for Hive on Amazon EMR
 
Redis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your applicationRedis & ZeroMQ: How to scale your application
Redis & ZeroMQ: How to scale your application
 
Winform
WinformWinform
Winform
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
 
Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2Using ngx_lua in UPYUN 2
Using ngx_lua in UPYUN 2
 
MySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELKMySQL Audit using Percona audit plugin and ELK
MySQL Audit using Percona audit plugin and ELK
 
Using Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasetsUsing Cerberus and PySpark to validate semi-structured datasets
Using Cerberus and PySpark to validate semi-structured datasets
 
More than syntax
More than syntaxMore than syntax
More than syntax
 
Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018
Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018
Jan Stępień - GraalVM: Fast, Polyglot, Native - Codemotion Berlin 2018
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
 
Elk with Openstack
Elk with OpenstackElk with Openstack
Elk with Openstack
 
glance replicator
glance replicatorglance replicator
glance replicator
 
Unqlite
UnqliteUnqlite
Unqlite
 
2015 05 27 JSConf - concurrency and parallelism final
2015 05 27   JSConf - concurrency and parallelism final2015 05 27   JSConf - concurrency and parallelism final
2015 05 27 JSConf - concurrency and parallelism final
 

Similar to Paris Kafka Meetup - How to develop with Kafka

Store and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraStore and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and Cassandra
Deependra Ariyadewa
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Microsoft Tech Community
 
Hazelcast
HazelcastHazelcast
Hazelcast
oztalip
 
Getting Started with Couchbase Ruby
Getting Started with Couchbase RubyGetting Started with Couchbase Ruby
Getting Started with Couchbase Ruby
Sergey Avseyev
 
I can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and SpringI can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and Spring
Joe Kutner
 
Secure .NET programming
Secure .NET programmingSecure .NET programming
Secure .NET programming
Ante Gulam
 
Kafka Streams: the easiest way to start with stream processing
Kafka Streams: the easiest way to start with stream processingKafka Streams: the easiest way to start with stream processing
Kafka Streams: the easiest way to start with stream processing
Yaroslav Tkachenko
 
The Wonderful World of Apache Kafka®
The Wonderful World of Apache Kafka®The Wonderful World of Apache Kafka®
The Wonderful World of Apache Kafka®
confluent
 
Designing a Scalable Data Platform
Designing a Scalable Data PlatformDesigning a Scalable Data Platform
Designing a Scalable Data Platform
Alex Silva
 
Introduction to Nodejs
Introduction to NodejsIntroduction to Nodejs
Introduction to Nodejs
Gabriele Lana
 
Flux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul DixFlux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul Dix
InfluxData
 
Fun Teaching MongoDB New Tricks
Fun Teaching MongoDB New TricksFun Teaching MongoDB New Tricks
Fun Teaching MongoDB New Tricks
MongoDB
 
Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...
Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...
Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...
confluent
 
Functional streams with Kafka - A comparison between Akka-streams and FS2
Functional streams with Kafka - A comparison between Akka-streams and FS2Functional streams with Kafka - A comparison between Akka-streams and FS2
Functional streams with Kafka - A comparison between Akka-streams and FS2
Luis Miguel Reis
 
Apache Cassandra and Go
Apache Cassandra and GoApache Cassandra and Go
Apache Cassandra and Go
DataStax Academy
 
Hadoop Integration in Cassandra
Hadoop Integration in CassandraHadoop Integration in Cassandra
Hadoop Integration in Cassandra
Jairam Chandar
 
Streaming twitter data using kafka
Streaming twitter data using kafkaStreaming twitter data using kafka
Streaming twitter data using kafka
Kiran Krishna
 
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Mathias Herberts
 
Testing Kafka - The Developer Perspective
Testing Kafka - The Developer PerspectiveTesting Kafka - The Developer Perspective
Testing Kafka - The Developer Perspective
maiktoepfer
 
Setup 3 Node Kafka Cluster on AWS - Hands On
Setup 3 Node Kafka Cluster on AWS - Hands OnSetup 3 Node Kafka Cluster on AWS - Hands On
Setup 3 Node Kafka Cluster on AWS - Hands On
hkbhadraa
 

Similar to Paris Kafka Meetup - How to develop with Kafka (20)

Store and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraStore and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and Cassandra
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
 
Hazelcast
HazelcastHazelcast
Hazelcast
 
Getting Started with Couchbase Ruby
Getting Started with Couchbase RubyGetting Started with Couchbase Ruby
Getting Started with Couchbase Ruby
 
I can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and SpringI can't believe it's not a queue: Kafka and Spring
I can't believe it's not a queue: Kafka and Spring
 
Secure .NET programming
Secure .NET programmingSecure .NET programming
Secure .NET programming
 
Kafka Streams: the easiest way to start with stream processing
Kafka Streams: the easiest way to start with stream processingKafka Streams: the easiest way to start with stream processing
Kafka Streams: the easiest way to start with stream processing
 
The Wonderful World of Apache Kafka®
The Wonderful World of Apache Kafka®The Wonderful World of Apache Kafka®
The Wonderful World of Apache Kafka®
 
Designing a Scalable Data Platform
Designing a Scalable Data PlatformDesigning a Scalable Data Platform
Designing a Scalable Data Platform
 
Introduction to Nodejs
Introduction to NodejsIntroduction to Nodejs
Introduction to Nodejs
 
Flux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul DixFlux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul Dix
 
Fun Teaching MongoDB New Tricks
Fun Teaching MongoDB New TricksFun Teaching MongoDB New Tricks
Fun Teaching MongoDB New Tricks
 
Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...
Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...
Streaming Design Patterns Using Alpakka Kafka Connector (Sean Glover, Lightbe...
 
Functional streams with Kafka - A comparison between Akka-streams and FS2
Functional streams with Kafka - A comparison between Akka-streams and FS2Functional streams with Kafka - A comparison between Akka-streams and FS2
Functional streams with Kafka - A comparison between Akka-streams and FS2
 
Apache Cassandra and Go
Apache Cassandra and GoApache Cassandra and Go
Apache Cassandra and Go
 
Hadoop Integration in Cassandra
Hadoop Integration in CassandraHadoop Integration in Cassandra
Hadoop Integration in Cassandra
 
Streaming twitter data using kafka
Streaming twitter data using kafkaStreaming twitter data using kafka
Streaming twitter data using kafka
 
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108
 
Testing Kafka - The Developer Perspective
Testing Kafka - The Developer PerspectiveTesting Kafka - The Developer Perspective
Testing Kafka - The Developer Perspective
 
Setup 3 Node Kafka Cluster on AWS - Hands On
Setup 3 Node Kafka Cluster on AWS - Hands OnSetup 3 Node Kafka Cluster on AWS - Hands On
Setup 3 Node Kafka Cluster on AWS - Hands On
 

Recently uploaded

Old Tools, New Tricks: Unleashing the Power of Time-Tested Testing Tools
Old Tools, New Tricks: Unleashing the Power of Time-Tested Testing ToolsOld Tools, New Tricks: Unleashing the Power of Time-Tested Testing Tools
Old Tools, New Tricks: Unleashing the Power of Time-Tested Testing Tools
Benjamin Bischoff
 
A Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdf
A Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdfA Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdf
A Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdf
kalichargn70th171
 
09. Ruby Object Oriented Programming - Ruby Core Teaching
09. Ruby Object Oriented Programming - Ruby Core Teaching09. Ruby Object Oriented Programming - Ruby Core Teaching
09. Ruby Object Oriented Programming - Ruby Core Teaching
quanhoangd129
 
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
singhlata50dh
 
Authentication Review-June -2024 AP & TS.pptx
Authentication Review-June -2024 AP & TS.pptxAuthentication Review-June -2024 AP & TS.pptx
Authentication Review-June -2024 AP & TS.pptx
DEMONDUOS
 
04. Ruby Operators Slides - Ruby Core Teaching
04. Ruby Operators Slides - Ruby Core Teaching04. Ruby Operators Slides - Ruby Core Teaching
04. Ruby Operators Slides - Ruby Core Teaching
quanhoangd129
 
How to Secure Your Kubernetes Software Supply Chain at Scale
How to Secure Your Kubernetes Software Supply Chain at ScaleHow to Secure Your Kubernetes Software Supply Chain at Scale
How to Secure Your Kubernetes Software Supply Chain at Scale
Anchore
 
Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...
Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...
Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...
87tomato
 
Empowering Businesses with Intelligent Software Solutions - Grawlix
Empowering Businesses with Intelligent Software Solutions - GrawlixEmpowering Businesses with Intelligent Software Solutions - Grawlix
Empowering Businesses with Intelligent Software Solutions - Grawlix
Aarisha Shaikh
 
Tour and travel website management in odoo,
Tour and travel website management in odoo,Tour and travel website management in odoo,
Tour and travel website management in odoo,
Axis Technolabs
 
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
45unexpected
 
SEO Cheat Sheet with Learning Resources by Balti Bloggers.pdf
SEO Cheat Sheet with Learning Resources by Balti Bloggers.pdfSEO Cheat Sheet with Learning Resources by Balti Bloggers.pdf
SEO Cheat Sheet with Learning Resources by Balti Bloggers.pdf
Balti Bloggers
 
Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...
Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...
Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...
3610stuck
 
AI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdf
AI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdfAI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdf
AI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdf
Daniel Zivkovic
 
GT degree offer diploma Transcript
GT degree offer diploma TranscriptGT degree offer diploma Transcript
GT degree offer diploma Transcript
attueb
 
Test Polarity: Detecting Positive and Negative Tests (FSE 2024)
Test Polarity: Detecting Positive and Negative Tests (FSE 2024)Test Polarity: Detecting Positive and Negative Tests (FSE 2024)
Test Polarity: Detecting Positive and Negative Tests (FSE 2024)
andrehoraa
 
03. Ruby Variables & Regex - Ruby Core Teaching
03. Ruby Variables & Regex - Ruby Core Teaching03. Ruby Variables & Regex - Ruby Core Teaching
03. Ruby Variables & Regex - Ruby Core Teaching
quanhoangd129
 
InflectraCON 360: Risk-Based Testing for Mission Critical Systems
InflectraCON 360: Risk-Based Testing for Mission Critical SystemsInflectraCON 360: Risk-Based Testing for Mission Critical Systems
InflectraCON 360: Risk-Based Testing for Mission Critical Systems
Inflectra
 
Predicting Test Results without Execution (FSE 2024)
Predicting Test Results without Execution (FSE 2024)Predicting Test Results without Execution (FSE 2024)
Predicting Test Results without Execution (FSE 2024)
andrehoraa
 
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Deliverybangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 

Recently uploaded (20)

Old Tools, New Tricks: Unleashing the Power of Time-Tested Testing Tools
Old Tools, New Tricks: Unleashing the Power of Time-Tested Testing ToolsOld Tools, New Tricks: Unleashing the Power of Time-Tested Testing Tools
Old Tools, New Tricks: Unleashing the Power of Time-Tested Testing Tools
 
A Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdf
A Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdfA Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdf
A Step-by-Step Guide to Selecting the Right Automated Software Testing Tools.pdf
 
09. Ruby Object Oriented Programming - Ruby Core Teaching
09. Ruby Object Oriented Programming - Ruby Core Teaching09. Ruby Object Oriented Programming - Ruby Core Teaching
09. Ruby Object Oriented Programming - Ruby Core Teaching
 
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
High Girls Call Chennai 000XX00000 Provide Best And Top Girl Service And No1 ...
 
Authentication Review-June -2024 AP & TS.pptx
Authentication Review-June -2024 AP & TS.pptxAuthentication Review-June -2024 AP & TS.pptx
Authentication Review-June -2024 AP & TS.pptx
 
04. Ruby Operators Slides - Ruby Core Teaching
04. Ruby Operators Slides - Ruby Core Teaching04. Ruby Operators Slides - Ruby Core Teaching
04. Ruby Operators Slides - Ruby Core Teaching
 
How to Secure Your Kubernetes Software Supply Chain at Scale
How to Secure Your Kubernetes Software Supply Chain at ScaleHow to Secure Your Kubernetes Software Supply Chain at Scale
How to Secure Your Kubernetes Software Supply Chain at Scale
 
Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...
Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...
Verified Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeli...
 
Empowering Businesses with Intelligent Software Solutions - Grawlix
Empowering Businesses with Intelligent Software Solutions - GrawlixEmpowering Businesses with Intelligent Software Solutions - Grawlix
Empowering Businesses with Intelligent Software Solutions - Grawlix
 
Tour and travel website management in odoo,
Tour and travel website management in odoo,Tour and travel website management in odoo,
Tour and travel website management in odoo,
 
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
 
SEO Cheat Sheet with Learning Resources by Balti Bloggers.pdf
SEO Cheat Sheet with Learning Resources by Balti Bloggers.pdfSEO Cheat Sheet with Learning Resources by Balti Bloggers.pdf
SEO Cheat Sheet with Learning Resources by Balti Bloggers.pdf
 
Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...
Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...
Mumbai Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service A...
 
AI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdf
AI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdfAI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdf
AI - Your Startup Sidekick (Leveraging AI to Bootstrap a Lean Startup).pdf
 
GT degree offer diploma Transcript
GT degree offer diploma TranscriptGT degree offer diploma Transcript
GT degree offer diploma Transcript
 
Test Polarity: Detecting Positive and Negative Tests (FSE 2024)
Test Polarity: Detecting Positive and Negative Tests (FSE 2024)Test Polarity: Detecting Positive and Negative Tests (FSE 2024)
Test Polarity: Detecting Positive and Negative Tests (FSE 2024)
 
03. Ruby Variables & Regex - Ruby Core Teaching
03. Ruby Variables & Regex - Ruby Core Teaching03. Ruby Variables & Regex - Ruby Core Teaching
03. Ruby Variables & Regex - Ruby Core Teaching
 
InflectraCON 360: Risk-Based Testing for Mission Critical Systems
InflectraCON 360: Risk-Based Testing for Mission Critical SystemsInflectraCON 360: Risk-Based Testing for Mission Critical Systems
InflectraCON 360: Risk-Based Testing for Mission Critical Systems
 
Predicting Test Results without Execution (FSE 2024)
Predicting Test Results without Execution (FSE 2024)Predicting Test Results without Execution (FSE 2024)
Predicting Test Results without Execution (FSE 2024)
 
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Deliverybangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 

Paris Kafka Meetup - How to develop with Kafka

  • 1. Paris Apache Kafka Meetup Florian HUSSONNOIS Zenika @fhussonnois
  • 2. Async, Sync, Batch, Partitioner et Retries
  • 3. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); KafkaProducer<String, String> producer = new KafkaProducer(config); ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value"); producer.send(record); producer.close();
  • 4. L’appel à la méthode send()est asynchrone et retourne immédiatement Le message est ajouté à un buffer avant d’être envoyé //... config.put("batch.size", 16384); config.put("linger.ms", 1); //... Latence entre chaque transmission de messages Taille maximum d’un batch
  • 5. List<ProducerRecord> batchRecords = new ArrayList<>(); //... for(ProducerRecord record : batchRecords) producer.send(record); producer.flush(); producer.close(); Force l’envoi des messages et bloque jusqu’à leur complétion
  • 6. Future<RecordMetadata> future = producer.send(record); RecordMetadata metadata = future.get(); // BLOCK LOG.info("message sent to topic {}, partition {}, offset {}", metadata.topic(), metadata.partition(), metadata.offset());
  • 7. ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value"); Future<RecordMetadata> future = producer.send(record, (metadata, e) -> { if(e != null) LOG.info("Message sent to topic {}, partition {}, offset {}", metadata.topic(), metadata.partition(), metadata.offset()); else LOG.error("Damn it!", e); });
  • 8. Configuration config.put("partitioner.class", DefaultPartitioner.class.getName() Implémenter un Partitioner public interface Partitioner { int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster); } Spécifier directement la partition cible new ProducerRecord("my_topic", 0, "my_key", "my_value");
  • 9. Acknowledgments config.put("ack", "all "); // plus lent, messages répliqués par tous les ISR Le Producer peut rejouer automatiquement les messages en erreurs config.put("retries", "0 "); // désactivé /! Peut provoquer des doublons (At-Least Once) /! Peut changer l’ordre de publication des messages
  • 10. Event Loop, Polling Model, Offset et Group Management
  • 11. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2")); while(true) { ConsumerRecords<Object, Object> records = consumer.poll(1000); records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); } Event Loop, Polling Model
  • 12. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); config.put("enable.auto.commit", false); // désactive auto-commit config.put("auto.commit.interval.ms", 100); KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2")); while(true) { ConsumerRecords<Object, Object> records = consumer.poll(1000); records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); consumer.commitAsync(); } }
  • 13. while(true) { ConsumerRecords<Object, Object> records = consumer.poll(1000); consumer.commitSync(); // Commit offsets before processing messages. records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); }
  • 15. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); config.put("group.id", "my_group"); KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2"));
  • 16. KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) { //do some stuff } @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { //do some stuff } });
  • 17. Chaque consumer d’un groupe doit notifier le coordinateur Uniquement possible sur un appel aux méthodes poll, commit, etc. Déclenché si un consumer rejoint ou quitte un groupe L’opération de « rebalance » est impactée par les paramètres : • session.timeout.ms (30 secondes) • heartbeat.interval.ms Rebalance intempestif en cas de traitement d’un message trop long
  • 18. ConsumerRecords<Object, Object> records = consumer.poll(1000); if( ! records.isEmpty() ) { consumer.pause(consumer.assignment().toArray(new TopicPartition[0])); Future<Boolean> future = executorService.submit(() -> { records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); return true; }); Boolean isCompleted = false; while(!isCompleted) { try { isCompleted = future.get(5, TimeUnit.SECONDS); // Wait before polling } catch (TimeoutException e) { consumer.poll(0); // heart-beat } catch (CancellationException |ExecutionException | InterruptedException e) { break; } } consumer.resume(consumer.assignment().toArray(new TopicPartition[0])); consumer.commitSync(); } ExecutorService
  • 19. Se positionner à un offset spécifique consumer.seek(new TopicPartition("my_topic", 0), 42); consumer.seekToEnd(new TopicPartition("my_topic", 0)); consumer.seekToBeginning(new TopicPartition("my_topic", 0)); Assignements manuel consumer.assign(Arrays.asList(new TopicPartition("my_topic", 0))); Obtenir les métriques consumer.metrics();
  • 20. Nous recrutons ! jobs@zenika.com @ZenikaIT Prochain Meetup le