Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Walking Up the Spring for Apache Kafka Stack

152 views

Published on

SpringOne Platform 2018
Walking Up the Spring for Apache Kafka Stack
Gary Russell, Pivotal; Viktor Gamov, Confluent

Published in: Software
  • Be the first to comment

Walking Up the Spring for Apache Kafka Stack

  1. 1. Walking up the Spring for Apache Kafka Stack Gary Russell Viktor Gamov @gprussell @gamussa Pivotal Confluent
  2. 2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 2
  3. 3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 2
  4. 4. @gprussell @gamussa @s1p Gary Russell Viktor Gamov @gprussell @gamussa Pivotal Confluent
  5. 5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Origins in Stream Processing 4
  6. 6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Origins in Stream Processing 4
  7. 7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Origins in Stream Processing 4
  8. 8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p High Throughput Streaming platform Origins in Stream Processing 4
  9. 9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring Boot / Kafka Streams Continuous Computation High Throughput Streaming platform API based clustering Origins in Stream Processing 4
  10. 10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Serving Layer (Cassandra, Elastic, etc.) Spring Boot / Kafka Streams Continuous Computation High Throughput Streaming platform API based clustering Origins in Stream Processing 4
  11. 11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p What is a Streaming Platform? 5 The Log ConnectorsConnectors Producer Consumer Streaming Engine
  12. 12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Kafka’s Distributed Log 6 The Log ConnectorsConnectors Producer Consumer Streaming Engine
  13. 13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p The log is a simple idea 7 Messages are added at the end of the log Old New
  14. 14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Consumers have a position all of their own 8 Sally is here George is here Fred is here Old New Scan Scan Scan
  15. 15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Only Sequential Access 9 Old New Read to offset & scan
  16. 16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Shard data to get scalability Messages are sent to different partitions 10
  17. 17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Shard data to get scalability Messages are sent to different partitions 10 Cluster of machines
  18. 18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Shard data to get scalability Messages are sent to different partitions 10 Producer (1) Producer (2) Producer (3) Cluster of machines Partitions live on different machines Messages are sent to different partitions
  19. 19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Linearly Scalable Architecture 11 Consumers Producers
  20. 20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Linearly Scalable Architecture 11 Consumers Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines
  21. 21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Linearly Scalable Architecture 11 Consumers Producers Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!!
  22. 22. Consumer Group Coordinator
  23. 23. Consumer Group Coordinator Consumers
  24. 24. Consumer Group Coordinator Consumers Consumer Group
  25. 25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Talk is cheap! Show me code! https://cnfl.io/streams-movie
  26. 26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Talk is cheap! Show me code! https://cnfl.io/streams-movie
  27. 27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p The Connect API 14 The Log ConnectorsConnectors Producer Consumer Streaming Engine
  28. 28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Ingest / Output to practically any data source 15 Kafka Connect Kafka Connect Kafka
  29. 29. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Ingest/Output from/to many data sources 16 Amazon S3 Elasticsearch HDFS JDBC Couchbase Cassandra Oracle SAP Vertica Blockchain DynamoDB FTP Github BigQuery Google Pub Sub RethinkDB Salesforce Solr Splunk JMX Kenesis MongoDB MQTT NATS Postgres Rabbit Redis Twitter
  30. 30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Stream Processing in Kafka 17 The Log ConnectorsConnectors Producer Consumer Streaming Engine
  31. 31. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 18 App Streams API
  32. 32. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 18 App Streams API Not running inside brokers!
  33. 33. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 19 Brokers? Nope! App Streams API App Streams API App Streams API Same app, many instances
  34. 34. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Before 20 DashboardProcessing Cluster Your Job Shared Database
  35. 35. @gprussell @gamussa @s1p As developers, we want to build APPS not INFRASTRUCTURE
  36. 36. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p After 22 Dashboard APP Streams API
  37. 37. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Things Kafka Streams Does 23 Runs everywhere Clustering done for you Exactly-once processing Event-time processing Integrated database Joins, windowing, aggregation S/M/L/XL/XXL/XXXL sizes
  38. 38. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Table-Stream Duality 25
  39. 39. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Table-Stream Duality 25
  40. 40. Do you think that’s a table you are querying ?
  41. 41. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 TABLE STREAM TABLE
  42. 42. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 TABLE STREAM TABLE
  43. 43. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 TABLE STREAM TABLE (“Gary”, 1)
  44. 44. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 TABLE STREAM TABLE (“Gary”, 1) Gary 1
  45. 45. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 Gary 1 Viktor 1 TABLE STREAM TABLE (“Gary”, 1) Gary 1
  46. 46. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 Gary 1 Viktor 1 TABLE STREAM TABLE (“Gary”, 1) (“Viktor”, 1) Gary 1 Gary 1 Viktor 1
  47. 47. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1 TABLE STREAM TABLE (“Gary”, 1) (“Viktor”, 1) (“Gary”, 2) Gary 1 Gary 1 Viktor 1
  48. 48. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1 TABLE STREAM TABLE (“Gary”, 1) (“Viktor”, 1) (“Gary”, 2) Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1
  49. 49. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1 Gary 2 Viktor 1 Soby 1 TABLE STREAM TABLE (“Gary”, 1) (“Viktor”, 1) (“Gary”, 2) (“Soby”, 1) Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1
  50. 50. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p 27 Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1 Gary 2 Viktor 1 Soby 1 TABLE STREAM TABLE (“Gary”, 1) (“Viktor”, 1) (“Gary”, 2) (“Soby”, 1) Gary 1 Gary 1 Viktor 1 Gary 2 Viktor 1 Gary 2 Viktor 1 Soby 1
  51. 51. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Compacted Topic Join Stream Table Kafka Kafka Streams Topic Join Streams and Tables 28
  52. 52. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Talk is cheap! Show me code!
  53. 53. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Talk is cheap! Show me code!
  54. 54. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p What’s next?
  55. 55. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31
  56. 56. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework
  57. 57. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka
  58. 58. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients
  59. 59. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients Spring Integration (Kafka Extension)
  60. 60. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients Spring Integration (Kafka Extension) Spring Cloud Stream (Kafka Binder, Streams Binder)
  61. 61. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients Spring Integration (Kafka Extension) Spring Cloud Stream (Kafka Binder, Streams Binder) Kafka Streams
  62. 62. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients Spring Integration (Kafka Extension) Spring Cloud Stream (Kafka Binder, Streams Binder) Spring Boot Kafka Streams
  63. 63. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients Spring Integration (Kafka Extension) Spring Cloud Stream (Kafka Binder, Streams Binder) Spring Boot Kafka Streams Reactor Kafka
  64. 64. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p Spring for Apache Kafka - The Full Stack 31 Spring Framework Spring For Apache Kafka kafka-clients Spring Integration (Kafka Extension) Spring Cloud Stream (Kafka Binder, Streams Binder) Spring Boot Kafka Streams Reactor Kafka Application
  65. 65. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ @gprussell @gamussa @s1p What’s New in Spring for Apache Kafka 2.2 ? • Container Factory - no longer just for @KafkaListener • Create any container - e.g. for Spring Integration Adapter/Gateway • @KafkaListener - autoStartup and concurrency property overrides • Recoverable SeekToCurrentErrorHandler/AfterRollbackProcessor • Optional Publishing to a Dead-Letter Topic • ErrorHandlingDeserializer • Embedded Kafka (test) Broker - added JUnit5 support • 2.2.0.RC1 release candidate available - last chance for requests!! 32
  66. 66. @ @gamussa @S1P @confluentinc www.kafka-summit.org promo: Gamov20
  67. 67. @gprussell @gamussa @s1p > Stay Connected. https://cnfl.io/streams-movie https://github.com/garyrussell/spring-kafka-demos #springone@s1p

×