2. Introduction to Kafka with Spring
Integration
• Kafka (Mihail Yordanov)
• Spring integration (Borislav Markov)
• Students Example (Mihail & Borislav)
• Conclusion
3. Motivation
• Real time data being continuously generated
• Producers and consumers relationships
• Streaming systems
• Messaging systems
4. Apache Kafka
• Developed by LinkedIn
• “We built Apache Kafka at LinkedIn with a specific
purpose in mind: to serve as a central repository of
data streams.” - Jay Kreps
• Kafka improvements
– Transporting data between systems
– Richer analytical processing
• “A publish-subscribe messaging system rethought as a
distributed commit log”
5. Kafka Vocabulary
• Brokers
• Producers
• Consumers
• “A publish-subscribe messaging system
rethought as a distributed commit log”
6. Kafka Vocabulary
• Topics
• Partitions
• “A publish-subscribe messaging system rethought
as a distributed commit log”
10. Consumer groups
• Definition - “A publish-subscribe messaging
system rethought as a distributed commit log”
• Publish - Subscribe
• Queueing
11. Spring Integration
• Spring Integration enables lightweight
messaging
• Supports integration with external systems via
declarative adapters
• Primary goal is to provide a simple model for
building enterprise integration solutions
• Separation of concerns
• Maintainable, testable code
12. Features
• Implementation of most of the Enterprise Integration Patterns
• Endpoint
• Channel (Point-to-point and Publish/Subscribe)
• Aggregator
• Filter
• Transformer
• Control Bus
• …
• Integration with External Systems
• ReST/HTTP
• ...
13. The theory...
The patterns can be found at
http://www.enterpriseintegrationpatterns.com/patterns/messag
ing/index.html
14. Message
• Messages are objects that carry information between two applications
• They are constructed by the producer and they are consumed/deconstructed at
the consumer/subscriber
• Message consists of:
– payload
– header
public interface Message<T> {
T getPayload();
MessageHeaders getHeaders();
}
15. Message endpoints
Common endpoints:
• Service Activator - invokes a method on a
bean
• Message Bridge - couples different
messaging modes/adapters. Example: (P2P
with Publish/Subscribe)
• Message Enricher - enrich the incoming
message with additional information.
– Header Enricher - add header
attributes
– Payload Enricher - enrich payload
16. Message endpoints (continued)
• Gateway - Used when we don’t have knowledge for the
messaging system.
– Synchronous Gateway - void or returns T
– Asynchronous Gateway - returns Future<T>
• Delayer - introduce delay between sender and receiver
• Consumers
– Polling Consumers - poll messages in a timely fashion
– Polling Using Triggers - poll messages with PeriodicTrigger
and with CronTrigger
– Event-Driven Consumers - they wait for someone to
deliver the message (framework’s responsibility)
17. Message endpoints (continued)
• Channel Adapter - endpoint that connects a Message Channel
to some other system or transport.
–inbound
–outbound
21. Flow components
• Filters
– Custom filters are tied to the framework, can
be bean method returning boolean
– Framework MessageSelector - use of method
boolean accept(Message m)
– Using annotation - @Filter
• Routers
– PayloadTypeRouter
– HeaderValueRouter
– Custom Routers - any bean method can be
router, the result will be the channel name.
– Recipient List Router - the message goes to all
statically defined channels
22. Flow components (continued...)
• Splitters - splits a message into pieces
– Custom splitters - any bean method;
– Framework AbstractMessageSplitter - use of method Object splitMessage(Message
m);
– Using annotation - @Splitter ;
• Aggregator
– assemble multiple messages to create a single parent message
– Complex task - all messages of a set have to arrive before aggregators can start work
– CorrelationStrategy - defines the key for grouping of the messages, the default
grouping is based on CORRELATION_ID
– ReleaseStrategy - dictates at which point the group of messages should be sent or
released for aggregation. Default is SequenceSizeReleazeStrategy
– Message Store - aggregators hold messages until all of them arrived. Options are:
• in-memory
• external database
• Resequencer - fix order of the messages(work on SEQUENCE_NUMBER)
23. Transformers
• Built-in transformers
– String Transformers - invokes toString() method of the payload
– Map Transformers - transformes POJO to a name-value pair of
Map
– Serializing and Deserializing Transformers - converts payload
to byte array
– JSON Transformers - object-to-json converts POJO to readable
JSON format; json-to-object transformer needs additional
property “type”
– XML Transformers - requires Spring OXM (Object-to-XML);
org.springframework.oxm.Marshaller/Unmarshaller
24. Transformers(continued...)
• Built-in transformers
– XPath Transformers - decodes
XML using XPath expressions, ex.:
xpath-
expression=”/mytag/@prop”
• Custom Transformers - can be any
spring bean method
• Using @Transformer
25. Students Example
• Example application will demo usage
of Kafka and Spring Integration
• App is built with maven
• Ideal candidate for Microservice
• Idea: takes students from outside and
calculates their average score
