Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introduction to Kafka with Spring Integration

732 views

Published on

A brief example that demonstrates asic usage of Kafka and Spring Integration when you build a microservice.

Published in: Software
  • Be the first to comment

Introduction to Kafka with Spring Integration

  1. 1. Introduction to Kafka with Spring Integration
  2. 2. Introduction to Kafka with Spring Integration • Kafka (Mihail Yordanov) • Spring integration (Borislav Markov) • Students Example (Mihail & Borislav) • Conclusion
  3. 3. Motivation • Real time data being continuously generated • Producers and consumers relationships • Streaming systems • Messaging systems
  4. 4. Apache Kafka • Developed by LinkedIn • “We built Apache Kafka at LinkedIn with a specific purpose in mind: to serve as a central repository of data streams.” - Jay Kreps • Kafka improvements – Transporting data between systems – Richer analytical processing • “A publish-subscribe messaging system rethought as a distributed commit log”
  5. 5. Kafka Vocabulary • Brokers • Producers • Consumers • “A publish-subscribe messaging system rethought as a distributed commit log”
  6. 6. Kafka Vocabulary • Topics • Partitions • “A publish-subscribe messaging system rethought as a distributed commit log”
  7. 7. Kafka Speed
  8. 8. Kafka Speed
  9. 9. Kafka Speed
  10. 10. Consumer groups • Definition - “A publish-subscribe messaging system rethought as a distributed commit log” • Publish - Subscribe • Queueing
  11. 11. Spring Integration • Spring Integration enables lightweight messaging • Supports integration with external systems via declarative adapters • Primary goal is to provide a simple model for building enterprise integration solutions • Separation of concerns • Maintainable, testable code
  12. 12. Features • Implementation of most of the Enterprise Integration Patterns • Endpoint • Channel (Point-to-point and Publish/Subscribe) • Aggregator • Filter • Transformer • Control Bus • … • Integration with External Systems • ReST/HTTP • ...
  13. 13. The theory... The patterns can be found at http://www.enterpriseintegrationpatterns.com/patterns/messag ing/index.html
  14. 14. Message • Messages are objects that carry information between two applications • They are constructed by the producer and they are consumed/deconstructed at the consumer/subscriber • Message consists of: – payload – header public interface Message<T> { T getPayload(); MessageHeaders getHeaders(); }
  15. 15. Message endpoints Common endpoints: • Service Activator - invokes a method on a bean • Message Bridge - couples different messaging modes/adapters. Example: (P2P with Publish/Subscribe) • Message Enricher - enrich the incoming message with additional information. – Header Enricher - add header attributes – Payload Enricher - enrich payload
  16. 16. Message endpoints (continued) • Gateway - Used when we don’t have knowledge for the messaging system. – Synchronous Gateway - void or returns T – Asynchronous Gateway - returns Future<T> • Delayer - introduce delay between sender and receiver • Consumers – Polling Consumers - poll messages in a timely fashion – Polling Using Triggers - poll messages with PeriodicTrigger and with CronTrigger – Event-Driven Consumers - they wait for someone to deliver the message (framework’s responsibility)
  17. 17. Message endpoints (continued) • Channel Adapter - endpoint that connects a Message Channel to some other system or transport. –inbound –outbound
  18. 18. Message Channels • interface MessageChannel – boolean send(Message<?> message); – boolean send(Message<?> message, long timeout); • Point-to-Point Mode -> PollableChannel – Message<?> receive(); – Message<?> receive(long timeout); • Publish/Subscribe Mode -> SubscribableChannel – boolean subscribe(MessageHandler handler); – boolean unsubscribe(MessageHandler handler);
  19. 19. Message Channels(continued) •Queue Channel - has capacity •Priority Channel - queue channel with prioritization •PublishSubscribe Channel •DirectChannel - mixed type P2P and Pub/Sub
  20. 20. Message Channels(continued)
  21. 21. Flow components • Filters – Custom filters are tied to the framework, can be bean method returning boolean – Framework MessageSelector - use of method boolean accept(Message m) – Using annotation - @Filter • Routers – PayloadTypeRouter – HeaderValueRouter – Custom Routers - any bean method can be router, the result will be the channel name. – Recipient List Router - the message goes to all statically defined channels
  22. 22. Flow components (continued...) • Splitters - splits a message into pieces – Custom splitters - any bean method; – Framework AbstractMessageSplitter - use of method Object splitMessage(Message m); – Using annotation - @Splitter ; • Aggregator – assemble multiple messages to create a single parent message – Complex task - all messages of a set have to arrive before aggregators can start work – CorrelationStrategy - defines the key for grouping of the messages, the default grouping is based on CORRELATION_ID – ReleaseStrategy - dictates at which point the group of messages should be sent or released for aggregation. Default is SequenceSizeReleazeStrategy – Message Store - aggregators hold messages until all of them arrived. Options are: • in-memory • external database • Resequencer - fix order of the messages(work on SEQUENCE_NUMBER)
  23. 23. Transformers • Built-in transformers – String Transformers - invokes toString() method of the payload – Map Transformers - transformes POJO to a name-value pair of Map – Serializing and Deserializing Transformers - converts payload to byte array – JSON Transformers - object-to-json converts POJO to readable JSON format; json-to-object transformer needs additional property “type” – XML Transformers - requires Spring OXM (Object-to-XML); org.springframework.oxm.Marshaller/Unmarshaller
  24. 24. Transformers(continued...) • Built-in transformers – XPath Transformers - decodes XML using XPath expressions, ex.: xpath- expression=”/mytag/@prop” • Custom Transformers - can be any spring bean method • Using @Transformer
  25. 25. Students Example • Example application will demo usage of Kafka and Spring Integration • App is built with maven • Ideal candidate for Microservice • Idea: takes students from outside and calculates their average score
  26. 26. Maven dependencies
  27. 27. Project Structure • Package “api” – POJO models • Package “config” – beans with factory methods • Package “service” – the business logic • Folder “resources” – application.properties
  28. 28. Spring Integration DSL
  29. 29. Students Example
  30. 30. Service activator
  31. 31. Running the app and giving some inputs
  32. 32. Check the logs
  33. 33. Conclusion Why Kafka and Spring ? • Easy for microservices • Clean and testable code • Widely adopted technologies • Easy to be dockerized

×