Walking through the Spring Stack for Apache Kafka with Soby Chacko | Kafka Summit London 2022

Conﬁdential │ ©2021 VMware, Inc.
Walking through the Spring
Stack for Apache Kafka
Kafka Summit London
April 25-26, 2022

Confidential │ ©2022 VMware, Inc. 2
About me...
Soby Chacko
Committer - Spring Cloud Stream/Spring
Cloud Data Flow at VMware
Tech lead for Spring Cloud Stream
Kafka/Kafka Streams binders
Twitter: @sobychacko
Github: github.com/sobychacko

Agenda
● Comprehensive look at the Apache Kafka
ecosystem in Spring
● Demo Applications
● Questions

Spring Stack for Apache Kafka
Spring Boot
Spring for Apache Kafka
Spring Integration Kafka
Spring Cloud Stream Binders for Kafka
and Kafka Streams
Reactor Kafka
Apache Kafka Java Client / Kafka Streams Client

Spring for Apache Kafka - Top Level Ideas
➔ Foundational library for Apache Kafka in Spring
➔ Spring friendly programming abstractions
➔ Many features that allow developers to focus on the business logic
➔ Provides abstractions for low-level infrastructure concerns
➔ Leverages and builds upon Apache Kafka Java Client / Kafka
Streams

Spring for Apache Kafka - Topic Management
@Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
return new KafkaAdmin(configs);
}
@Bean
public NewTopic myTopic() {
return TopicBuilder.name("my-topic")
.partitions(10)
.replicas(3)
.compact()
.build();
}
@Bean
public KafkaAdmin.NewTopics bunchOfTopics() {
return new KafkaAdmin.NewTopics(
TopicBuilder.name("topic1")
.build(),
.replicas(1)
.build(),
.partitions(3)
.build());
}

Publishing Records using KafkaTemplate API
KafkaTemplate provides a wide variety of template methods to publish records
ListenableFuture<SendResult<K, V>> sendDefault(K key, V data);
ListenableFuture<SendResult<K, V>> send(String topic, K key, V data);
ListenableFuture<SendResult<K, V>> send(String topic, Integer partition, K key, V data);

Async Send Example
ListenableFuture<SendResult<Integer, String>> future =
template.send("topic", 1, "data");
future.addCallback(new KafkaSendCallback<Integer, String>() {
@Override
public void onSuccess(SendResult<Integer, String> result) {...}
@Override
public void onFailure(KafkaProducerException ex) {
ProducerRecord<Integer, String> failed = ex.getFailedProducerRecord();
...
}
});

Producer Factory
● KafkaTemplate uses the producer factory for creating the producer
● Spring Boot auto-configures a producer factory bean
● Applications can provide custom producer factories
@Bean
public ProducerFactory<?, ?> kafkaProducerFactory() {
DefaultKafkaProducerFactory<?, ?> factory = new DefaultKafkaProducerFactory<>(
Map.of(...))
…
return factory;
}

Consuming Records using KafkaListener
● @KafkaListener annotation provides a lot of flexible options to consume records
from Kafka topics
● @EnableKafka annotation
@KafkaListener(id = "my-group", topics = "my-topic")
public void listen(String in) {
logger.info("Data Received : " + in);
}
Basic KafkaListener

KafkaListener with More Options
See the reference docs for all the available options of KafkaListener.
@KafkaListener(id = "my-group", topicPartitions =
@TopicPartition(topic = "my-topic", partitionOffsets = {
@PartitionOffset(partition = "0-2", initialOffset = "0"),
@PartitionOffset(partition = "6-9", initialOffset = "0")}),
concurrency = ${config.concurrency}
properties = "key.deserializer:LongDeserializer")
public void listen(String in, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) Long key,
@Header(KafkaHeaders.RECEIVED_PARTITION_ID) int part,
@Header(KafkaHeaders.OFFSET) int offset) {
logger.info( "Data Received : {} with key {} from partition {} and offset {}.",
in, key,part,offset);
}

KafkaListener on a Class
@KafkaListener(id = "multi", topics = "myTopic")
static class MultiListenerBean {
@KafkaHandler
public void listen1(String foo) {
...
}
@KafkaHandler
public void listen2(Integer bar) {
...
}
@KafkaHandler(isDefault = true)
public void listenDefault(Object object) {
...
}
}

MessageListenerContainer
● Behind the scenes, KafkaListener uses a MessageListener
Container for accomplishing various tasks such as polling,
committing, error handling etc.
● KafkaMessageListenerContainer - Single listener thread
● ConcurrentKafkaMessageListenerContainer - Multiple listeners
● Spring Boot auto-configures a factory for creating a message listener
container

MessageListeners
● Spring for Apache Kafka provides various types of message listeners
● Record and Batch based Message Listeners
● onMessage(...) method that gives access to the consumer record
● You can provide your own implementation to the container
● MessageListener
● AcknowledgingMessageListener
● ConsumerAwareMessageListener
● AcknowledgingConsumerAwareMessageListener
● BatchMessageListener
● BatchAcknowledgingMessageListener
● BatchConsumerAwareMessageListener
● BatchAcknowledgingConsumerAwareMessageListener

Committing the Offsets
● By default, enable.auto.commit is disabled in Spring for Apache
Kafka and the framework is responsible for committing the offsets
● A number of options are available for commits which the applications
can configure on the container through the AckMode property of
ContainerProperties
– RECORD, BATCH, TIME, COUNT, MANUAL,
MANUAL_IMMEDIATE
● Default AckMode is the BATCH

KafkaListener Example of Manually Acknowledging
@KafkaListener(id = "cat",
topics = "myTopic",
containerFactory =
"kafkaManualAckListenerContainerFactory")
public void listen(String data, Acknowledgment ack) {
…
ack.acknowledge();
}

Seeking Offsets
● Listener must implement the ConsumerSeekAware interface
● Framework provides a convenient AbstractConsumerSeekAware
● Using the callbacks, the application can do arbitrary seeks such as
seekToBeginning, seekToEnd, seekToRelative, seekToTimestamp
etc.
void registerSeekCallback(ConsumerSeekCallback callback);
void onPartitionsAssigned(Map<TopicPartition, Long> assignments,
ConsumerSeekCallback callback);
void onPartitionsRevoked(Collection<TopicPartition> partitions);
void onIdleContainer(Map<TopicPartition, Long> assignments,
ConsumerSeekCallback callback);

Container Error Handling
● Container Error Handler - CommonErrorHandler interface
● Default implementation - DefaultErrorHandler
● By default, DefaultErrorHandler provides 10 max attempts and no backoff
● You can provide a custom bean that overrides these defaults
@Bean
public DefaultErrorHandler errorHandler(
DeadletterPublishingRecoverer recoverer) {
DefaultErrorHandler handler = new DefaultErrorHandler(recoverer,
new FixedBackOff(2_000, 2));
handler.addNotRetryableExceptions(IllegalArgumentException.class);
return handler;
}

ErrorHandlingDeserializer
● Convenient way to catch deserialization errors
● ErrorHandlingDeserializer has delegates to the actual key/value
deserializers
● When deserialization fails, the proper headers are populated with
the exception and the raw data that failed
● Then normal container error handling rules apply

Extensive Error Handling Options
● Spring for Apache Kafka provides much more features, options and flexibility for
error handling. See the reference documentation for more information.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-error-handling
● Non Blocking Retries - A feature in active development. See the documentation
section for more details.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#retry-topic

Testing using EmbeddedKafka Broker
● spring-kafka-test provides a convenient way to test with an embedded broker
● EmbeddedKafka annotation and EmbeddedKafkaBroker for Spring test context with JUnit 5
● For non-spring test context, use the EmbeddedKafkaCondition with JUnit 5
● For JUnit4, use EmbeddedKafkaRule, which is a JUnit4 ClassRule
● More details at:
https://docs.spring.io/spring-kafka/docs/current/reference/html/#embedded-kafka-annotation
@SpringBootTest
@EmbeddedKafka(topics = "my-topic", bootstrapServersProperty =
"spring.kafka.bootstrap-servers")
class SpringKafkaApp1Tests {
@Test
void test(EmbeddedKafkaBroker broker) {
// ...
}
}

Advanced Spring for Apache Kafka features
● Request/Reply Semantics -
https://docs.spring.io/spring-kafka/docs/current/reference/html/#replyi
ng-template
● Transactions -
https://docs.spring.io/spring-kafka/docs/current/reference/html/#trans
actions
● Kafka Streams Support -
https://docs.spring.io/spring-kafka/docs/current/reference/html/#strea
ms-kafka-streams

Spring Integration Overview
● Messages and Channels
● Integration Flows
- Spring Integration is used to create complex EIP flows
- Kafka flows can tap into larger flows
● Provides Java Config and DSL

Spring Integration Kafka Support
● Outbound Channel Adapter – KafkaProducerMessageHandler
● Kafka Message Driven Channel Adapter
● Inbound Channel Adapter - Provides a KafkaMessageSource which
is a pollable channel adapter implementation.
● Outbound Gateway - for request/reply operations
● Inbound Gateway - for request/reply operations
See more details in the reference docs for Spring Integration at
https://docs.spring.io/spring-integration/docs/current/reference/html/kafka.html#kafka

Spring Cloud Stream Overview
● General purpose framework for writing event driven/stream
processing micro services
● Programming model based on Spring Cloud Function - Java
8 Functions/Function Composition
● Apache Kafka support is built on top of Spring Boot, Spring
for Apache Kafka, Spring Integration and Kafka Streams

Spring Cloud Stream Kafka Application Architecture
Spring Boot Application
Spring Cloud Stream App Core
Kafka/Kafka Streams Binder
Inputs (Message
Channel/Kafka
Streams)
Outputs(Message
Channel/Kafka
Streams)
Kafka Broker
Spring Integration/Kafka Streams

Spring Cloud Stream - Kafka Binder
● Auto provisioning of topics
● Producer/Consumer Binding
● Message Converters/Native Serialization
● DLQ Support
● Health checks for the binder

Spring Cloud Stream Kafka Binder Programming Model
@SpringBootApplication
…
@Bean
public Supplier<Long> currentTime() {
return System::currentTimeMillis;
}
@Bean
public Function<Long, String> convertToUTC() {
return localTime -> formatUTC();
}
@Bean
public Consumer<String> print() {
System.out::println;
}

Spring Cloud Stream - Kafka Streams Binder
● Built on the Kafka Streams foundations from Spring for Apache
Kafka - StreamsBuilderFactoryBean
● Kafka Streams processors written as Java functions
● KStream, KTable and GlobalKTable bindings
● Serde inference on input/output
● Interactive Query Support
● Multiple functions in the same application
● Function composition

Examples of Spring Cloud Stream Kafka Streams Functions
@Bean
public Function<KStream<Object, String>, KStream<String, Long>> wordCount()
{
return input -> input
.flatMapValues(value ->
Arrays.asList(value.toLowerCase().split("W+")))
.map((key, value) -> new KeyValue<>(value, value))
.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofSeconds(WINDOW_SIZE_SECONDS)))
.count()
.toStream()
.map((key, value) -> new KeyValue<>(key.key(), value));
}
@Bean
public Consumer<KStream<Object, Long>> logWordCounts() {
return kstream -> kstream.foreach((key, value) -> {
logger.info(String.format("Word counts for: %s t %d", key, value));
});
}

Resources
Project Site Reference Docs GitHub StackOverflow
Spring Integration Kafka
Spring Cloud Stream
Spring Boot Kafka Docs

Walking through the Spring Stack for Apache Kafka with Soby Chacko | Kafka Summit London 2022

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Walking through the Spring Stack for Apache Kafka with Soby Chacko | Kafka Summit London 2022

Similar to Walking through the Spring Stack for Apache Kafka with Soby Chacko | Kafka Summit London 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Walking through the Spring Stack for Apache Kafka with Soby Chacko | Kafka Summit London 2022