Speakers: Patrick Schuh, Bearing Point + Patrik Kleindl, Bearing Point Abstract: - Managing topic configurations and dependencies in a microservice deployment - Managing Kafka Streams configurations - KafkaEsque: an open source support tool for Apache Kafka® development (https://github.com/patschuh/KafkaEsque)

1. Patrick Schuh
Patrik Kleindl
Prepared by
Kafka Meetup 02.04.19
Microservices, Kafka Streams and KafkaEsque
02 / 04 / 2019

2. 2Client Name here
Content
I n t r o d u c t i o n
M i c r o s e r v i c e D e v e l o p m e n t
Topic & Kafka Streams
Configuration
D e v / T e s t S u p p o r t
KafkaEsque

3. 3Client Name here
Global Standards - Local Services
Graz
BearingPoint
Technology GmbH
Lassallestrasse 7a
1020 Vienna
Austria
BearingPoint
Technology GmbH
Seering 6
A-8141 Premstätten/Graz
Austria
Jörg Steinbauer
Partner
joerg.steinbauer@bearingpoint.com
M +43 664 81 61 804
Björn Grosser
Partner
bjoern.grosser@bearingpoint.com
M +43 664 41 37 867
BearingPoint Agile Software Engineering:
local services through 200 employees in Vienna / Graz
BearingPoint Management and Technology Consultancy
(4000 employees, mostly Europe)
BearingPoint Austria (450 employees)

4. 4Client Name here
Center of Excellence for Agile Software Engineering
Succeed together…
Digital Innovation
Your understanding of the
business and ours of
technology are the key to
finding innovative ways of
providing services.
Either through design thinking
workshops or “hands-on,
garage-style” approach, our
expertise in existing, and
research on emerging
technologies will make a
difference.
Agile Advisory
We assist your development team through
coaching & training the Agile Delivery Model
that we are using ourselves.
Custom Solutions/SW-Factory
You need a reliable partner for your
SW-solution?
We create reliable solutions running
on any cloud platform using
our proven BearingPoint
Digital Software Factory 4.0.
Infrastructure & Security
You need an infrastructure for
your solution, either custom or
standardized out-of-the box?
We also design or provide
network and security services.
Managed Services
We deliver 7x24 IT
Service Management and
Application Support.
Designed to your needs,
with the highest service
quality.

5. 5Client Name here
Who we are
BearingPoint Technology GmbH
Lasallestraße 7a
1020 Wien
Industries
Professional Background
► Java
► Kafka
► Banking
► Logistics
BearingPoint Technology GmbH
Lasallestraße 7a
1020 Wien
Industries
Professional Background
► Kafka
► Databases (Oracle, SQL Server)
► Performance Tuning
► Telco
► Banking
► Logistics
Patrick Schuh
System Engineer
Patrik Kleindl
Technology Architect

6. 6Client Name here
Microservice Architecture
New way of building systems – new challenges

7. 7Client Name here
Development and Operations challenges
How small problems become big problems fast
1 5
20
40
60
70
10
30
60
120
150
200
0
50
100
150
200
250
Q1/2018 Q2/2018 Q3/2018 Q4/2018 Q1/2019 Q2/2019
Streams Topics
• Fast growth in # of applications and topics
• Multiple deployments per day
• Multiple vendors
• Multiple stages
Common problems:
• Constant changes to code, configuration
and deployment pipeline
• Configuration missing/mismatch
• Resource problems (memory, disk, CPU,
etc.)

8. Topic & Kafka Streams
PA R T 1

9. 9Client Name here
Topics
Phase 2
First steps
Phase 3
Improvements
• Automated provisioning for
AMQP queues in place
• Configuration management in
GIT repo per stage
• Support in delivery pipeline to
map configuration to
deployment
• Deployment/Application
checks on topics
• No cross-vendor
dependencies
• Support for Kafka topics
• Lots of manual configuration
• Lots of code changes to map
properties through
deployment – more
deployments
• More services – more
dependencies – more
problems
• Local environment setup
more complicated
• Unified configuration with
overrides
• No single property mappings
in deployment pipeline
• Use dependency mechanism
• Support provisioning per
application startup and
centrally for operations
Phase 1
Initial situation

10. 10Client Name here
DependenciesConfiguration
componentName: demo-cmp
topics:
- name: demo.topic1
config:
cleanup.policy: compact
- name: demo.topic2
config:
cleanup.policy: compact,delete
delete.retention.ms: 604800000
retention.ms: 604800000
- name: demo.topic3
config:
cleanup.policy: delete,compact
delete.retention.ms: 7776000000
retention.ms: 7776000000
Producer Consumer
Local Dev Test Prod
Overrides

11. 11
Kafka Streams
New challenges
• Kafka Streams apps are full-blown applications
► Persistence technology
► Multiple Consumers
► Topologies can get complex fast
• Testing
• Processing performance
• Resource usage
• Deployment
• Monitorig (State)
New requirements
• Simplify configuration, reduce load on devs and
ops
• Allow quick reaction to problems across stages
• Allow state monitoring
• Allow automatic recovery
• Expose state stores via REST services
• Expose streams topologies
New solutions
• Flexible configuration
• Publish state to topic
• Wrap streams applications
► Restart/Recovery/Start/Stop
► REST services
• Visualization of topologies
https://github.com/zz85/kafka-streams-viz

12. 12Client Name here
• Throughput
• Availability
• JVM Memory
• Native Memory
• Local storage
Configuration
name: stream1
commit.interval.ms: 1000
consumer.max.poll.records: 100
consumer.session.timeout.ms: 30000
cache.max.bytes.buffering: 104857600
num.stream.threads: 4
rocksdb.config.setter: …SimpleCustomRocksDbConfigSetter
statestores.compression: LZ4_COMPRESSION
statestores.write_buffer_size: 8388608
statestores.block_size: 8192
statestores.block_cache_size: 26214400
statestores.enable_bloom_filter: true

13. KafkaEsque
PA R T 2

14. 14Client Name here
Motivation
01 02 03 04
GUI for common
commands
Browse/search/trace messages
Write one or more messages
Display topic settings
Experimenting with the
kafka-clients lib
Consuming messages in mixed
setups
Offset/Partition handling
Playing around with
JavaFX
Setting up a GUI
Provide quick access to
common features
Have fun and learn
something new J

15. 15
Features
Reading
• Consume Messages
► from beginning
► latest X messages (by setting the offset back)
► from a specific offset (will be used for all
partitions)
► continuously until stopped
• Full-Text search results
• Export of consumed displayed message list as CSV
Tracing
• Trace key
► Will consume topic "from beginning" showing
all messages with the given key
► Can be set to "fast-trace" using the default
partitioner to only consume the resulting
partition (does not work for AVRO keys)
• Trace in value
► Will consume topic "from beginning" showing
all messages where a subsequence of the value
matches the regex (Matcher.find() method)
Writing
• Produce Messages
► to specific partition
► letting the partitioner decide (partition: -1)

16. 16
Features
Topic Management
• Creating, removing and describing of topics
• Topic Templates
for creating a set of topics with the same
configuration on different clusters
• https://github.com/patschuh/KafkaEsque/wiki/T
opic-Templates
Message Books
• Insert multiple messages for some simple test
scenarios
• https://github.com/patschuh/KafkaEsque/wiki/
Message-Books
Basic Schema-Registry Browser
• Lists all subjects
• Shows selected subjects schema versions, JSON
formatted and with tree view
Cross Cluster Operations (”Poor man’s connect" ;P)
• Copy messages from one cluster to another
• per default only messages published after the start of the
operation (will copy until stopped)
• can be changed by setting a start timestamp (epoch-ms)
• number of messages to copy can be limited
• can be filtered by a specific key or a regex for the value

17. Demo time

18. https://github.com/patschuh/KafkaEsque
https://kafka.esque.at/downloads/

19. Thank you.From BearingPoint Austria
www.bearingpoint.com
+(43) 664 21 42 430
patrik.kleindl@bearingpoint.com
BearingPoint Technology GmbH
Lasallestraße 7a
1020 Wien