Spring Cloud Stream with Kafka

OVERVIEW
This sample project demonstrates how to build
applications using
, , ,
and .
real-
time streaming event-driven
architecture Spring Boot Spring Cloud Stream Apache
Kafka Lombok
[ GitPitch @ github/davidkiss/spring-cloud-streams-kafka-demo ]

2 / 60

By end of this tutorial you'll have a simple Spring Boot
based Greetings microservice running that
1. takes a message from a REST api
2. writes it to a Kafka topic
3. reads it from the topic
4. outputs it to the console

3 / 60

LET'S GET STARTED!

4 / 60

WHAT IS SPRING CLOUD STREAMING?
Spring Cloud Stream is a framework built upon Spring
Boot for building message-driven microservices.

5 / 60

WHAT IS KAFKA?
Kafka is a popular high performant and horizontally
scalable messaging platform originally developed by
LinkedIn.

6 / 60

INSTALLING KAFKA

7 / 60

Download Kafka from and untar it:here
> tar -xzf kafka_2.11-1.0.0.tgz
> cd kafka_2.11-1.0.0

8 / 60

Start Zookeeper and Kafka
On Windows:
On Linux or Mac:
> binwindowszookeeper-server-start.bat configzookeeper.propert
> binwindowskafka-server-start.bat configserver.properties
> bin/zookeeper-server-start.sh config/zookeeper.properties
> bin/kafka-server-start.sh config/server.properties

9 / 60

If Kafka is not running and fails to start after your
computer wakes up from hibernation, delete the
<TMP_DIR>/kafka-logs folder and then start Kafka
again.

10 / 60

WHAT IS LOMBOK?
Lombok is a java framework that automatically
generates getters, setters, toString(), builders, loggers,
etc. in the code.

11 / 60

MAVEN DEPENDENCIES

12 / 60

Go to to create a maven project:https://start.spring.io

13 / 60

1. Add necessary dependencies: Spring Cloud
Stream, Kafka, Devtools (for hot redeploys
during development, optional), Actuator (for
monitoring application, optional), Lombok (make
sure to also have the Lombok plugin installed in your
IDE)
2. Click the Generate Project button to download the
project as a zip le
3. Extract zip le and import the maven project to your
favourite IDE

14 / 60

Notice the maven dependencies in the pom.xml le:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>

15 / 60

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- hot reload - press Ctrl+F9 in IntelliJ after a code change
<dependency>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>

16 / 60

... also the <dependencyManagement> section:
<dependencyManagement>
<dependencies>
<dependency>
<!-- Import dependency management from Spring Boo
<artifactId>spring-boot-dependencies</artifactId>
<version>$
{spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
...

17 / 60

...
<dependency>
<artifactId>spring-cloud-stream-dependencies</art
<version>$
{spring-cloud-stream.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

18 / 60

... and the <repository> section:
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-milestone</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>

19 / 60

DEFINE THE KAFKA STREAMS
package com.kaviddiss.streamkafka.stream;
import org.springframework.cloud.stream.annotation.Input;
import org.springframework.cloud.stream.annotation.Output;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.SubscribableChannel;

20 / 60

public interface GreetingsStreams {
String INPUT = "greetings-in";
String OUTPUT = "greetings-out";
@Input(INPUT)
SubscribableChannel inboundGreetings();
@Output(OUTPUT)
MessageChannel outboundGreetings();
}

21 / 60

In order for our application to be able to communicate
with Kafka, we'll need to de ne an outbound stream to
write messages to a Kafka topic, and an inbound stream
to read messages from a Kafka topic.

22 / 60

Spring Cloud provides a convenient way to do this by
simply creating an interface that de nes a separate
method for each stream.

23 / 60

The inboundGreetings() method de nes the
inbound stream to read from Kafka and
outboundGreetings() method de nes the
outbound stream to write to Kafka.

24 / 60

During runtime Spring will create a java proxy based
implementation of the GreetingsStreams interface
that can be injected as a Spring Bean anywhere in the
code to access our two streams.

25 / 60

CONFIGURE SPRING CLOUD STREAM
Our next step is to con gure Spring Cloud Stream to
bind to our streams in the GreetingsStreams
interface.

26 / 60

This can be done by creating a @Configuration class
com.kaviddiss.streamkafka.config.StreamsConfig
with below code:

27 / 60

package com.kaviddiss.streamkafka.config;
import com.kaviddiss.streamkafka.stream.GreetingsStreams;
import org.springframework.cloud.stream.annotation.EnableBinding;
@EnableBinding(GreetingsStreams.class)
public class StreamsConfig {
}

28 / 60

Binding the streams is done using the
@EnableBinding annotation where the
GreatingsService interface is passed to.

29 / 60

CONFIGURATION PROPERTIES FOR KAFKA
By default, the con guration properties are stored in
the
src/main/resources/application.properties
le.

30 / 60

However I prefer to use the YAML format as it's less
verbose and allows to keep both common and
environment-speci c properties in the same le.

31 / 60

For now, let's rename application.properties to
application.yaml and paste below con g snippet
into the le:

32 / 60

spring:
cloud:
stream:
kafka:
binder:
brokers: localhost:9092
bindings:
greetings-in:
destination: greetings
contentType: application/json
greetings-out:
destination: greetings
contentType: application/json

33 / 60

The above con guration properties con gure the
address of the Kafka server to connect to, and the
Kafka topic we use for both the inbound and outbound
streams in our code. They both must use the same
Kafka topic!

34 / 60

The contentType properties tell Spring Cloud Stream
to send/receive our message objects as Strings in the
streams.

35 / 60

CREATE THE MESSAGE OBJECT
Create a simple
com.kaviddiss.streamkafka.model.Greetings
class with below code that will represent the message
object we read from and write to the greetings
Kafka topic:

36 / 60

package com.kaviddiss.streamkafka.model;
// lombok autogenerates getters, setters, toString() and a builde
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
@Getter @Setter @ToString @Builder
public class Greetings {
private long timestamp;
private String message;
}

37 / 60

Notice how the class doesn't have any getters and
setters thanks to the Lombok annotations. The
@ToString will generate a toString() method
using the class' elds and the @Builder annotation
will allow us creating Greetings objects using uent
builder (see below).

38 / 60

CREATE SERVICE LAYER TO WRITE TO KAFKA
Let's create the
com.kaviddiss.streamkafka.service.GreetingsSer
class with below code that will write a Greetings
object to the greetings Kafka topic:

39 / 60

package com.kaviddiss.streamkafka.service;
import com.kaviddiss.streamkafka.model.Greetings;
import lombok.extern.slf4j.Slf4j;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.MessageHeaders;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;
import org.springframework.util.MimeTypeUtils;

40 / 60

@Service
@Slf4j
public class GreetingsService {
private final GreetingsStreams greetingsStreams;
public GreetingsService(GreetingsStreams greetingsStreams) {
this.greetingsStreams = greetingsStreams;
}

41 / 60

public void sendGreeting(final Greetings greetings) {
log.info("Sending greetings {}", greetings);
MessageChannel messageChannel = greetingsStreams.outboundG
messageChannel.send(MessageBuilder
.withPayload(greetings)
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeU
.build());
}
}

42 / 60

The @Service annotation will con gure this class as a
Spring Bean and inject the GreetingsService
dependency via the constructor.

43 / 60

The @Slf4j annotation will generate an SLF4J logger
eld that we can use for logging.

44 / 60

In the sendGreeting() method we use the injected
GreetingsStream object to send a message
represented by the Greetings object.

45 / 60

CREATE REST API
Now we'll be creating a REST api endpoint that will
trigger sending a message to Kafka using the
GreetingsService Spring Bean:

46 / 60

package com.kaviddiss.streamkafka.web;
import com.kaviddiss.streamkafka.service.GreetingsService;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;

47 / 60

@RestController
public class GreetingsController {
private final GreetingsService greetingsService;
public GreetingsController(GreetingsService greetingsService)
this.greetingsService = greetingsService;
}

48 / 60

@GetMapping("/greetings")
@ResponseStatus(HttpStatus.ACCEPTED)
public void greetings(@RequestParam("message") String message
Greetings greetings = Greetings.builder()
.message(message)
.timestamp(System.currentTimeMillis())
.build();
greetingsService.sendGreeting(greetings);
}
}

49 / 60

The @RestController annotation tells Spring that
this is a Controller bean (the C from MVC). The
greetings() method de nes an HTTP GET
/greetings endpoint that takes a message request
param and passes it to the sendGreeting() method
in GreetingsService.

50 / 60

LISTENING ON THE GREETINGS KAFKA TOPIC
Let's create a
com.kaviddiss.streamkafka.service.GreetingsLis
class that will listen to messages on the greetings
Kafka topic and log them on the console:

51 / 60

package com.kaviddiss.streamkafka.service;
import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.stream.annotation.StreamListener
import org.springframework.messaging.handler.annotation.Payload;
import org.springframework.stereotype.Component;

52 / 60

@Component
@Slf4j
public class GreetingsListener {
@StreamListener(GreetingsStreams.INPUT)
public void handleGreetings(@Payload Greetings greetings) {
log.info("Received greetings: {}", greetings);
}
}

53 / 60

The @Component annotation similarly to @Service
and @RestController de nes a Spring Bean.

54 / 60

GreetingsListener has a single method,
handleGreetings() that will be invoked by Spring
Cloud Stream with every new Greetings message
object on the greetings Kafka topic. This is thanks to
the @StreamListener annotation con gured for the
handleGreetings() method.

55 / 60

RUNNING THE APPLICATION
The last piece of the puzzle is the
com.kaviddiss.streamkafka.StreamKafkaApplicati
class that was auto-generated by the Spring Initializer:

56 / 60

package com.kaviddiss.streamkafka;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplicati
@SpringBootApplication
public class StreamKafkaApplication {
public static void main(String[] args) {
SpringApplication.run(StreamKafkaApplication.class, args)
}
}

57 / 60

No need to make any changes here. You can either run
this class as a Java application from your IDE, or run the
application from the command line using the Spring
Boot maven plugin:
> mvn spring-boot:run

58 / 60

Once the application is running, go to
in the
browser and check your console.
http://localhost:8080/greetings?message=hello

59 / 60

SUMMARY
I hope you enjoyed this tutorial. Feel free to ask any
questions and leave your feedback.

60 / 60

Spring Cloud Stream with Kafka

More Related Content

What's hot

Similar to Spring Cloud Stream with Kafka

Recently uploaded

Spring Cloud Stream with Kafka