Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a
Creative Commons Attr...
Upcoming SlideShare
Loading in …5
×

Spring For Apache Kafka

3,981 views

Published on

SpringOne Platform 2016
Speaker: Gary Russell; Principal Engineer, Pivotal

With the advent of the Kafka 0.9+ pure Java client, the Spring Team has created the new spring-kafka project with familiar Spring abstractions such as the KafkaTemplate, message listener container and POJO listener annotation @KafkaListener.

In this talk we'll take a look at the features of the project as well as the new version (2.0) of spring-integration-kafka which is now based on the Spring for Apache Kafka project.

Published in: Technology
  • Be the first to comment

Spring For Apache Kafka

  1. 1. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring for Apache Kafka Gary P. Russell @gprussell
  2. 2. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring for Apache Kafka • Brings the familiar Spring programming model to Apache Kafka • KafkaTemplate • MessageListenerContainer • @KafkaListener Annotation http://projects.spring.io/spring-kafka/ 2
  3. 3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring for Apache Kafka spring-kafka: • 1.0.2.RELEASE (0.9.x.x client) • 1.1.0.BUILD-SNAPSHOT (0.10.x.x client) - milestone soon • https://github.com/spring-projects/spring-kafka spring-integration-kafka: • inbound/outbound channel adapters • 2.0.1.RELEASE (based on spring-kafka) • 1.3.1.RELEASE (uses 0.8.x.x client directly) • https://github.com/spring-projects/spring-integration-kafka 3
  4. 4. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ ProducerFactory 4 @Bean public ProducerFactory<String, String> producerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, this.brokerAddress); // ... props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); return new DefaultKafkaProducerFactory<>(props); }
  5. 5. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ KafkaTemplate 5 @Bean public KafkaTemplate<String, String> kafkaTemplate() { return new KafkaTemplate<>(producerFactory()); } ListenableFuture<SendResult<K, V>> send(String topic, V data); ListenableFuture<SendResult<K, V>> send(String topic, int partition, V data); etc.
  6. 6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ ConsumerFactory 6 @Bean public ConsumerFactory<String, String> consumerFactory() { Map<String, Object> props = new HashMap<>(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, this.brokerAddress); props.put(ConsumerConfig.GROUP_ID_CONFIG, "s1pGroup"); // ... props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class); return new DefaultKafkaConsumerFactory<>(props); }
  7. 7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @KafkaListener 7 @KafkaListener(id = "foo", topics = “${my.topic}") public void listen1(String foo) { // ... } @KafkaListener(id = "bar", topicPattern = “${topic.root}.*”) public void listen2(@Payload String foo, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) Integer key, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) { // ... }
  8. 8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @KafkaListener - Explicit Partition Assignment 8 @KafkaListener(topicPartitions = @TopicPartition(topic = "${kafka.topic}", partitionOffsets = { @PartitionOffset(partition = "0", initialOffset = "0"), @PartitionOffset(partition = "1", initialOffset = "0"), })) public void listen(String foo) { // ... }
  9. 9. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @KafkaListener - Container Factory 9 @Bean public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() { ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(consumerFactory()); factory.setMessageConverter( new StringJsonMessageConverter()); return factory; }
  10. 10. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Listener Containers 10 @Bean public KafkaMessageListenerContainer<String, String> container( ConsumerFactory<String, String> consumerFactory, ConfigProperties config) { ContainerProperties containerProperties = new ContainerProperties(config.getTopic()); containerProperties.setMessageListener(listener()); return new KafkaMessageListenerContainer<>(consumerFactory, containerProperties); }
  11. 11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ MessageListener 11 public class Listener implements MessageListener<String, String> { @Override public void onMessage(ConsumerRecord<String, String> record) { // ... } }
  12. 12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ AcknowledgingMessageListener 12 public static class AckListener implements AcknowledgingMessageListener<String, String> { @Override public void onMessage(ConsumerRecord<String, String> record, Acknowledgment acknowledgment) { // ... acknowledgment.acknowledge(); } }
  13. 13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Listener Adapters • Add behavior to listeners • Retry adapter • Filtering adapter • Also configurable when using @KafkaListener 13
  14. 14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring for Apache Kafka Test Support • Embedded Kafka Server - JUnit @Rule • Creates topics with configurable partitions • Utility methods to wait for partition assignment • Hamcrest Matchers • AssertJ Conditions 14
  15. 15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Some Caveats • Polling, Pause, Resume, auto commit problems • Let the container manage the commits (or manual) 15
  16. 16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring for Apache Kafka DEMO 16
  17. 17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Learn More. Stay Connected. Contribute!! https://github.com/spring-projects/spring-kafka/CONTRIBUTING.adoc @springcentral spring.io/blog @pivotal pivotal.io/blog @pivotalcf http://engineering.pivotal.io

×