Presentation from the Developer Group meetup October 2020

1. TESTING KAFKA
 @maik_toepfer

2. OUTLINE
Introduction
Testing Endless
Test Approaches
Recommendations

3. ABOUT ME
software crafter by heart
working as a Software Engineer for PRISMA in Leipzig
organizer of the LE Software Craft Meetup

4. KAFKA IS…
a middleware
a distributed log
the hammer for all your nails 😏

5. KAFKA CAN BE…

6. …A CLASSIC MESSAGE QUEUE
App A
(consume)
Topic
write
App X
(produce)
read

7. …A PUBLISH-SUBSCRIBE SYSTEM

8. …WITH MULTIPLE CONSUMERS
consume
App A
App Z
consume
consume
Topic
App X
produce
App Y

9. …AND MULTIPLE PRODUCERS
App A
consume
App B
App Z
consume
App C
consume
Topic
produce
App X
produce
App Y
produce

10. THE DISTRIBUTED LOG
messages don’t vanish after being read
stored until they are purged (depending on retention policy)

11. INDEPENDENT CONSUMPTION OF MESSAGES IN A
TOPIC
Topic
(aka "the Log")
reads
writes
App X App Y
App A
reads
1
2
1
1
1
0
9876543210

12. READING ON

13. THE TEXTBOOK KAFKA CONSUMER
var props = new Properties();
props.put("bootstrap.servers", "kafka-server:9092");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
var consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(List.of("my-topic"));
while (true) {
var records = consumer.poll(
Duration.ofMillis(Long.MAX_VALUE));
for (var record : records)
System.out.printf("offset = %d, key = %s, value = %s%n",
record.offset(), record.key(), record.value());
}
1 1
2
3
4
5
6
7
8
9 2
10
11 3
12
13 4
14
15
16
17

14. THE MESSAGE RECEIVER
too little creativity - sorry

15. OVERVIEW
Kafka Message Receiver Repository
read write

16. CLASS OVERVIEW
MessageReceiver
kafkaConsumer
topic
repository
run()
pass everything needed into the object

17. USAGE
public class Main {
public static void main(String[] args){
var consumer = new KafkaConsumer<String, String>(props);
var topic = "my-topic";
var repository = new Repository("jdbc:...");
var messageReceiver =
new MessageReceiver(consumer, topic, repository);
messageReceiver.run();
}
}
1
2
3

18. A LITTLE DETOUR

19. CONSTRUCTOR INJECTION
Kafka Consumer Topic Name
Message Receiver
Repository
providing everything the object requires via the constructor
enables testability

20. SIMPLE TO REPLACE DEPENDENCIES
Fake
Kafka Consumer Topic Name
Message Receiver
Fake
Repository

21. MORE ON DI (1)

22. MORE ON DI (2)

23. FIRST IMPLEMENTATION
public MessageReceiver(Consumer<String, String> kafkaConsumer,
String topic, Repository repository ) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository;
}
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (true) { // this is an endless loop
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE))
for (var record : records) {
repository.saveMessage(record.value());
}
}
}
1

24. HOW TO TEST "ENDLESS" ?

25. APPROACHES
1. Divide And Conquer
2. My Own De nition Of "Endless"

26. DIVIDE AND CONQUER
test everything - except the endless loop

27. DIVIDE AND CONQUER
public MessageReceiver(KafkaConsumer<String, String> kafkaConsumer,
String topic, Repository repository) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository; }
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (true) {
processRecords();
}}
protected void processRecords() {
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE));
for (var record : records) {
repository.saveMessage(record.value());
}}

28. A POSSIBLE TEST
@Mock Repository mockRepository;
@Mock KafkaConsumer<String,String> mockConsumer;
@Test
public void writesReceivedMessage() {
// arrange
var records = buildConsumerRecords("my-topic", "HELLO WORLD");
when(mockConsumer.poll(any())).thenAnswer(notUsed-> records);
// act (sut = System Under Test)
var sut = new MessageReceiver(mockConsumer, mockRepository, "my-topic")
// we don't call run() directly
sut.processRecords();
// assert
verify(mockRepository, times(1)).saveMessage("HELLO WORLD");
}
1

29. TEST COVERAGE
public MessageReceiver(KafkaConsumer<String, String> kafkaConsumer,
String topic, Repository repository) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository; }
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (true) {
processRecords();
}}
protected void processRecords() {
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE));
for (var record : records) {
repository.saveMessage(record.value());
}}

30. BUT WE CAN DO BETTER

31. THE MESSAGE RECEIVER REVISITED
What is upsetting here?
public MessageReceiver(Consumer<String, String> kafkaConsumer,
String topic, Repository repository ) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository; }
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (true) {
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE))
for (var record : records) {
repository.saveMessage(record.value());
}
}}

32. THE MESSAGE RECEIVER REVISITED
public MessageReceiver(Consumer<String, String> kafkaConsumer,
String topic, Repository repository ) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository; }
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (true) { // <-- THIS !
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE))
for (var record : records) {
repository.saveMessage(record.value());
}
}}

33. LET’S INJECT "ENDLESS"
public MessageReceiver(Consumer<String, String> kafkaConsumer,
String topic, Repository repository,
BooleanSupplier whileFunc ) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository;
this.whileFunc = whileFunc; }
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (whileFunc.getAsBoolean()) {
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE))
for (var record : records) {
repository.saveMessage(record.value()); }}}

34. LET’S INJECT "ENDLESS"

35. USAGE
public class Main {
public static void main(String[] args){
var consumer = new KafkaConsumer<String, String>(props);
var topic = "my-topic";
var repository = new Repository("jdbc:...");
var messageReceiver = new MessageReceiver(
consumer, topic, repository,
() -> true );
messageReceiver.run();
}
}

36. A POSSIBLE TEST
@Mock Repository mockRepository;
@Mock KafkaConsumer<String,String> mockConsumer;
@Test
public void writesReceivedMessage() {
// arrange
var records = buildConsumerRecords("my-topic", "HELLO WORLD");
when(mockConsumer.poll(any())).thenAnswer(notUsed-> records);
// act
var sut = new MessageReceiver(
mockConsumer, mockRepository, "my-topic",
this::runOnce);
sut.run();
// assert
verify(mockRepository, times(1)).saveMessage("HELLO WORLD");
}

37. ONCE IS ENOUGH
private boolean firstTimeRun = true;
private boolean runOnce() {
if (firstTimeRun) {
firstTimeRun = false;
return true;
}
return false;
}

38. LET’S LOOK AT THE TEST COVERAGE

39. TEST COVERAGE
public MessageReceiver(Consumer<String, String> kafkaConsumer,
String topic, Repository repository,
BooleanSupplier whileFunc ) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository;
this.whileFunc = whileFunc;
}
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (whileFunc.getAsBoolean()) {
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE))
for (var record : records) {
repository.saveMessage(record.value());
}
}
}

40. TSCHAKKA

41. TESTING KAFKA AT DIFFERENT INTEGRATION
LEVELS

42. OUR "SYSTEM UNDER TEST" (SUT)
public MessageReceiver(Consumer<String, String> kafkaConsumer,
String topic, Repository repository,
BooleanSupplier whileFunc ) {
this.kafkaConsumer = kafkaConsumer;
this.topic = topic;
this.repository = repository;
this.whileFunc = whileFunc; }
public void run() {
kafkaConsumer.subscribe(List.of(topic));
while (whileFunc.getAsBoolean()) {
var records = kafkaConsumer.poll(Duration.ofMillis(Long.MAX_VALUE))
for (var record : records) {
repository.saveMessage(record.value());
}
}
}

43. MOCKITO
defacto standard for mocking in Java

44. MOCKITO - CONTEXT

45. MOCKITO - EXAMPLE TEST
@Mock Repository mockRepository;
@Mock KafkaConsumer<String,String> mockConsumer;
@Test
public void writesReceivedMessage() {
// arrange
var records = buildConsumerRecords("my-topic", "HELLO WORLD");
when(mockConsumer.poll(any())).thenAnswer(notUsed-> records);
// act
var sut = new MessageReceiver(
mockConsumer, mockRepository, "my-topic",
this::runOnce);
sut.run();
// assert
verify(mockRepository, times(1)).saveMessage("HELLO WORLD"); }

46. BORING

47. MOCKITO - PROS
execute fast
no Kafka needed

48. MOCKITO - CONS
maybe too isolated

49. KAFKA FAKES OBJECTS
fake objects
close to the actual prod implementation (our hope)
Kafka internally used for testing

50. FAKE OBJECT ?
Fake
Object
Test
Stub
Test
Double
Mock
Object
Complexity
Test
Spy

51. KAFKA FAKE OBJECTS - CONTEXT

52. USING THE PLAIN MOCKCONSUMER
import org.apache.kafka.clients.consumer.MockConsumer;
...
@Test
public void plainMockConsumer() {
// arrange
// preparing mock consumer
var mockConsumer =
new MockConsumer<String, String>(OffsetResetStrategy.EARLIEST);
// need to subscribe here, we'd like it in prod code ☹
mockConsumer.subscribe(List.of("my-topic"));
// voodoo part
var topicPartition=new TopicPartition("my-topic", 0);
mockConsumer.rebalance(List.of(topicPartition));
mockConsumer.updateBeginningOffsets(Map.of(topicPartition, 0L));
// this what we understand
mockConsumer.addRecord(new ConsumerRecord<>("my-topic",
0 0 ll "HELLO WORLD"))

53. THAT’S UGLY!

54. IDEA
Encapsulate the ugly part in a BufferedMockConsumer

55. BUFFERED MOCK CONSUMER
MockConsumer
BufferedMockConsumer
buffer() : called in test
subscribe() : called in prod code
poll() : called in prod code
inherits
50 lines of code
comes with the source of this presentation

56. UPDATED TEST
@Test
public void withBufferedMockConsumer() {
// arrange
var mockConsumer =
new BufferedMockConsumer<String, String>("my-topic");
mockConsumer.buffer("HELLO WORLD");
// act
var sut = new MessageReceiver(mockConsumer, "my-topic",
mockRepository, this::runOnce);
sut.run();
// assert
verify(mockRepository, times(1)).saveMessage("HELLO WORLD");
}

57. KAFKA FAKE OBJECTS - PROS
deeper Kafka testing
fake objects are provided by Kafka project team
should be closer to the real thing then Mockito

58. KAFKA FAKE OBJECTS - CONS
usually requires own helper class

59. EMBEDDED KAFKA
use a real Kafka cluster in your test
started inside the test

60. ALTERNATIVES
embedded-kafka (Scala)
Spring-Kafka

61. EMBEDDED KAFKA - CONTEXT

62. EMBEDDED KAFKA - EXAMPLE TEST
@Mock Repository mockRepository;
@RegisterExtension
public static final SharedKafkaTestResource sharedKafkaTestResource =
new SharedKafkaTestResource();
@Test
public void withEmbeddedKafkaCluster() {
// arrange
var kafkaTestUtils = sharedKafkaTestResource.getKafkaTestUtils();
var testRecord = Map.of("".getBytes(), "HELLO WORLD".getBytes());
kafkaTestUtils.produceRecords(testRecord, "my-topic", 0);
var props = new Properties();
props.put("group.id", "mygroup");
props.put("auto.offset.reset", "earliest"); // this one is needed
var consumer = kafkaTestUtils.getKafkaConsumer(
StringDeserializer.class, StringDeserializer.class, props);
// t

63. EMBEDDED KAFKA - PROS
you test with the real thing
no mocks
very close to production

64. EMBEDDED KAFKA - CONS
shared cluster between tests → no independent tests
in case of an error you deal with a real cluster
slow

65. TESTCONTAINERS
…a Java library that supports JUnit tests, providing lightweight,
throwaway instances of … anything else that can run in a Docker
container.

66. TESTCONTAINERS - CONTEXT

67. TESTCONTAINERS - EXAMPLE TEST
@Mock Repository mockRepository;
@Container
private static final KafkaContainer KAFKA = new KafkaContainer();
@Test
public void withTestcontainers() {
// arrange
var producerProps = new Properties();
producerProps.put("bootstrap.servers", KAFKA.getBootstrapServers());
producerProps.put("key.serializer", StringSerializer.class);
producerProps.put("value.serializer", StringSerializer.class);
var consumerProps = new Properties();
consumerProps.put("bootstrap.servers", KAFKA.getBootstrapServers());
consumerProps.put("group.id", "mygroup");
consumerProps.put("auto.offset.reset", "earliest");
consumerProps.put("key.deserializer", StringDeserializer.class);
consumerProps.put("value.deserializer", StringDeserializer.class);
K fk C St i St i ( P )

68. TESTCONTAINERS - PROS
easy to use
generally good for integration tests (e.g. database)
project has currently a lot of traction

69. TESTCONTAINERS - CONS
requires Docker as a separate dependency
your CI needs to support Docker as well
slow

70. RECOMMENDATIONS

71. RECALL THE TEST PYRAMID
many tests on unit level
little test on end-to-end level

72. OUR APPROACH
majority: Kafka Fake objects combined with an own
BufferedMockConsumer
a couple of integration tests based on Testcontainers

73. THANK YOU FOR YOUR ATTENTION!

74. IMAGE REFERENCES
[tired] https://pixabay.com/photos/winks-sleeping-snooze-sleep-2383407/
[maki] https://pixabay.com/photos/maki-curious-halfaap-peekaboo-3295891/
[boring] https://pixabay.com/photos/cute-animal-pet-tired-yawn-sleep-3258931/
[balled_up_paper] https://unsplash.com/photos/e4Davs7-XnE
[detour] https://unsplash.com/photos/dQLgop4tnsc
[endless] https://pixabay.com/de/photos/cassiopeia-supernova-cassiopeia-2515913/
[craft] https://pixabay.com/de/photos/hobel-hammer-s%C3%A4ge-rost-n%C3%A4gel-4607203/