How to develop with Kafka (meetup May 17th, 2016). Introduction to Producer and Consumer APIs. http://www.meetup.com/fr-FR/Paris-Apache-Kafka-Meetup

1. Paris Apache Kafka
Meetup
Florian HUSSONNOIS
Zenika
@fhussonnois

2. Async, Sync, Batch, Partitioner et Retries

3. Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
KafkaProducer<String, String> producer = new KafkaProducer(config);
ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value");
producer.send(record);
producer.close();

4. L’appel à la méthode send()est asynchrone et retourne immédiatement
Le message est ajouté à un buffer avant d’être envoyé
//...
config.put("batch.size", 16384);
config.put("linger.ms", 1);
//... Latence entre chaque transmission de messages
Taille maximum d’un batch

5. List<ProducerRecord> batchRecords = new ArrayList<>();
//...
for(ProducerRecord record : batchRecords)
producer.send(record);
producer.flush();
producer.close(); Force l’envoi des messages et bloque jusqu’à leur
complétion

6. Future<RecordMetadata> future = producer.send(record);
RecordMetadata metadata = future.get(); // BLOCK
LOG.info("message sent to topic {}, partition {}, offset {}",
metadata.topic(),
metadata.partition(),
metadata.offset());

7. ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value");
Future<RecordMetadata> future = producer.send(record, (metadata, e) -> {
if(e != null)
LOG.info("Message sent to topic {}, partition {}, offset {}",
metadata.topic(),
metadata.partition(),
metadata.offset());
else
LOG.error("Damn it!", e);
});

8. Configuration
config.put("partitioner.class", DefaultPartitioner.class.getName()
Implémenter un Partitioner
public interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster);
}
Spécifier directement la partition cible
new ProducerRecord("my_topic", 0, "my_key", "my_value");

9. Acknowledgments
config.put("ack", "all "); // plus lent, messages répliqués par tous les ISR
Le Producer peut rejouer automatiquement les messages en erreurs
config.put("retries", "0 "); // désactivé
/! Peut provoquer des doublons (At-Least Once)
/! Peut changer l’ordre de publication des messages

10. Event Loop, Polling Model, Offset et Group
Management

11. Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"));
while(true) {
ConsumerRecords<Object, Object> records = consumer.poll(1000);
records.forEach(record ->
LOG.info("key={}, value={}", record.key(), record.value()));
}
Event Loop, Polling Model

12. Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
config.put("enable.auto.commit", false); // désactive auto-commit
config.put("auto.commit.interval.ms", 100);
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"));
while(true) {
ConsumerRecords<Object, Object> records = consumer.poll(1000);
records.forEach(record ->
LOG.info("key={}, value={}", record.key(), record.value()));
consumer.commitAsync();
}
}

13. while(true) {
ConsumerRecords<Object, Object> records = consumer.poll(1000);
consumer.commitSync(); // Commit offsets before processing messages.
records.forEach(record ->
LOG.info("key={}, value={}", record.key(), record.value()));
}

15. Properties config = new Properties();
config.put("bootstrap.servers", "localhost:9092");
config.put("group.id", "my_group");
KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"));

16. KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config);
consumer.subscribe(Arrays.asList("topic1, topic2"), new
ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
//do some stuff
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
//do some stuff
}
});

17. Chaque consumer d’un groupe doit notifier le coordinateur
Uniquement possible sur un appel aux méthodes poll, commit, etc.
Déclenché si un consumer rejoint ou quitte un groupe
L’opération de « rebalance » est impactée par les paramètres :
• session.timeout.ms (30 secondes)
• heartbeat.interval.ms
Rebalance intempestif en cas de traitement d’un message trop long

18. ConsumerRecords<Object, Object> records = consumer.poll(1000);
if( ! records.isEmpty() ) {
consumer.pause(consumer.assignment().toArray(new TopicPartition[0]));
Future<Boolean> future = executorService.submit(() -> {
records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value()));
return true;
});
Boolean isCompleted = false;
while(!isCompleted) {
try {
isCompleted = future.get(5, TimeUnit.SECONDS); // Wait before polling
} catch (TimeoutException e) {
consumer.poll(0); // heart-beat
} catch (CancellationException |ExecutionException | InterruptedException e) {
break;
}
}
consumer.resume(consumer.assignment().toArray(new TopicPartition[0]));
consumer.commitSync();
}
ExecutorService

19. Se positionner à un offset spécifique
consumer.seek(new TopicPartition("my_topic", 0), 42);
consumer.seekToEnd(new TopicPartition("my_topic", 0));
consumer.seekToBeginning(new TopicPartition("my_topic", 0));
Assignements manuel
consumer.assign(Arrays.asList(new TopicPartition("my_topic", 0)));
Obtenir les métriques
consumer.metrics();

20. Nous recrutons ! jobs@zenika.com
@ZenikaIT
Prochain Meetup le