Small presentation about Apache Kafka including Kafka vs. Message Broker, Quarkus Extension for Apache Kafka, and Testing a Kafka application.

1. Edgar Domingues
October 2021
Kafka Client and Emitters

2. 01
02
03
04
Agenda
Apache Kafka
Kafka vs. Message Broker
Quarkus Extension for Apache Kafka
Testing a Kafka application
2

3. Apache Kafka
Open-source distributed event streaming platform
“event streaming is the practice of
• capturing data in real-time from event sources (like databases, sensors, mobile
devices, cloud services, and software applications) in the form of streams of
events;
• storing these event streams durably for later retrieval;
• manipulating, processing, and reacting to the event streams in real-time as well
as retrospectively; and
• routing the event streams to different destination technologies as needed.”
-- https://kafka.apache.org/documentation/
3

4. 4
https://miro.medium.com/max/3720/1*5DMYoWniIyN7YRoJof4K_w.png

5. Topics & Partitions
5
https://linuxhint.com/wp-content/uploads/2018/10/1-31.png

6. Partitioner
Producer will decide target partition to place any message, depending on:
1. Partition id, if it's specified within the message
2. key % num partitions, if no partition id is mentioned
3. Round robin if neither partition id nor message key are available in message,
meaning only value is available
6

7. Consumer Groups
7
https://betterprogramming.pub/rabbitmq-vs-kafka-1ef22a041793

8. Kafka vs. Message Broker
Kafka
8
• Dumb-Broker and Smart-Consumer
• Pull
• Topics/Partitions/Groups
• Connectors
• Stronger guarantees of messages
ordering
• Persisted events
Message Broker (e.g. RabbitMQ
• Smart-Broker and Dumb-Consumer
• Push
• Queues/Exchanges/Routing
• Superior support when it comes to
routing and filtering messages for
consumers to use.
• Message Time-To-Live and
delayed/scheduled messages
• Delivery retries and dead-letter
exchanges/queues

9. 9
Quarkus Extension for Apache Kafka
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
</dependency>

10. 10
Disable kafka health check
# Keep kafka health disabled to avoid leaking the address of the kafka instances
quarkus.kafka.health.enabled=false

11. 11
Receiving messages from kafka
@Incoming("accounts")
@Blocking
public void consumeAccountEvent(final JsonObject json) {
// handle event
}

12. 12
Configuring Receiver
#kafka.bootstrap.servers # from environment
mp.messaging.incoming.accounts.connector=smallrye-kafka
mp.messaging.incoming.accounts.topic=event-splitter.accounts
mp.messaging.incoming.accounts.value.deserializer=io.vertx.kafka.client.serialization.JsonObjectDeserializer
mp.messaging.incoming.accounts.group.id=wfm-mercury-accounts
mp.messaging.incoming.accounts.auto.offset.reset=earliest
mp.messaging.incoming.accounts.security.protocol=SASL_SSL
mp.messaging.incoming.accounts.sasl.mechanism=SCRAM-SHA-512
mp.messaging.incoming.accounts.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
required username=${kafka.username} password="${kafka.password}";
mp.messaging.incoming.accounts.ssl.truststore.location=${kafka.truststore.location}
mp.messaging.incoming.accounts.ssl.truststore.password=${kafka.truststore.password}

13. 13
Sending messages to kafka
@Inject
@Channel("timeseries-data-reporting")
@OnOverflow(value = OnOverflow.Strategy.UNBOUNDED_BUFFER)
Emitter<TimeSeriesReportEvent> emitter;
void publish(final TimeSeriesReportEvent event) {
// emitter.send(event);
String key = event.getAccountId() + "-" + event.getQueueId();
emitter.send(KafkaRecord.of(key, event));
}

14. 14
Configuring Emitter
#kafka.bootstrap.servers # from environment
mp.messaging.outgoing.timeseries-data-reporting.connector=smallrye-kafka
mp.messaging.outgoing.timeseries-data-reporting.topic=wfm-timeseries.timeseries-data-reporting
mp.messaging.outgoing.timeseries-data-reporting.value.serializer=io.quarkus.kafka.client.serialization.JsonbSerializer
mp.messaging.outgoing.timeseries-data-reporting.security.protocol=SASL_SSL
mp.messaging.outgoing.timeseries-data-reporting.sasl.mechanism=SCRAM-SHA-512
mp.messaging.outgoing.timeseries-data-reporting.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
required username=${kafka.username} password="${kafka.password}";
mp.messaging.outgoing.timeseries-data-reporting.ssl.truststore.location=${kafka.truststore.location}
mp.messaging.outgoing.timeseries-data-reporting.ssl.truststore.password=${kafka.truststore.password}
mp.messaging.outgoing.timeseries-data-reporting.compression.type=zstd

15. Overflow management
// Set the max size to 10 and fail if reached
@OnOverflow(value = OnOverflow.Strategy.BUFFER, bufferSize = 10)
@Inject @Channel("channel") Emitter<String> emitterWithBuffer;
// [DANGER ZONE] no limit
@OnOverflow(OnOverflow.Strategy.UNBOUNDED_BUFFER)
@Inject @Channel("channel") Emitter<String> danger;
// Drop the new messages if the size is reached
@OnOverflow(OnOverflow.Strategy.DROP)
@Inject @Channel("channel") Emitter<String> dropping;
// Drop the previously sent messages if the size is reached
@OnOverflow(OnOverflow.Strategy.LATEST)
@Inject @Channel("channel") Emitter<String> dropOldMessages;
https://smallrye.io/smallrye-reactive-messaging/smallrye-reactive-messaging/3.2/emitter/emitter.html#emitter-overflow

16. 16
Testing a Kafka application
<dependency>
<groupId>io.smallrye.reactive</groupId>
<artifactId>smallrye-reactive-messaging-in-memory</artifactId>
<scope>test</scope>
</dependency>

17. 17
Kafka Test Resource Lifecycle Manager
public class KafkaTestResourceLifecycleManager implements QuarkusTestResourceLifecycleManager {
@Override
public Map<String, String> start() {
return InMemoryConnector.switchIncomingChannelsToInMemory("accounts");
}
@Override
public void stop() {
InMemoryConnector.clear();
}
}

18. 18
Test class
@QuarkusTest
@QuarkusTestResource(KafkaTestResourceLifecycleManager.class)
class KafkaListenerIT {
@Inject
@Any
InMemoryConnector connector;

19. 19
Test receiving messages from kafka
@Test
void handleAccountEvent() {
// Arrange
InMemorySource<JsonObject> emitter = connector.source("accounts");
// Act
emitter.send(...);
// Assert
// assert event handled as expected…

20. 20
Test sending messages to Kafka
@Test
public void sendTimeSeriesReportEvent() {
// Arrange
InMemorySink<TimeSeriesReportEvent> events = connector.sink("timeseries-data-reporting");
//...
// Act
service.publish(event);
// Assert
assertThat(events.received().size(), is(1));
// assert event key and payload as expected...
}

21. Questions?
21

22. 22

23. 23
https://betterprogramming.pub/rabbitmq-vs-kafka-1ef22a041793