Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kafka Streams - From the Ground Up to the Cloud


Published on

SpringOne Platform 2017
Marius Bogoevici, Redhat

In this session we will introduce the Kafka Streams API and the Kafka Streams processing engine, followed by the Kafka Streams support in the Spring portfolio - showing how to easily write and deploy Kafka Streams applications using Spring Cloud Stream and deploy them on various cloud platforms using Spring Cloud Data Flow.

Published in: Technology

Kafka Streams - From the Ground Up to the Cloud

  1. 1. Kafka Streams: From the grounds up to the Cloud Marius Bogoevici, Chief Architect, Red Hat Spring One Platform, Dec 4, 2017 @mariusbogoevici
  2. 2. Marius Bogoevici ● Chief Architect, Data Streaming at Red Hat ● Spring ecosystem contributor since 2008 ○ Spring Integration ● Spring team member between 2014 and 2017 ○ Spring XD, Spring Integration Kafka ○ Spring Cloud Stream project lead ● Co-author “Spring Integration in Action”, Manning, 2012
  3. 3. Kafka: from messaging system to streaming platform (based on Distributed log Replication, Fault tolerance Connect and Streams Transactions, Exactly once
  4. 4. How about applications that are both producers and consumers and perform complex computations? Kafka as a distributed messaging system
  5. 5. Kafka Streams ● Client library for stream processing ○ Embed stream processing features into regular Java applications (microservice model) ○ Create sophisticated topologies of independent applications ● Functional transformations via DSL: ○ Mapping, filtering, flatMap ○ Aggregation, joins (multiple topics) ○ Windowing ● Kafka-to-Kafka semantics ● One-record-at-a-time processing (no microbatching) ● Stateful processing support ● Transactions/exactly once Kafka Cluster Application Kafka Streams
  6. 6. Kafka Streams - important concepts ● KStream ○ Record stream abstraction ○ Read from/written to external topic or produced from other KStream via operators such as map/filter ● KTable/GlobalKTable ○ Changelog stream abstraction (key is meaningful) ○ Read from external topic as a sequence of updates ○ Produced from other tables or stream joins, aggregations etc ● State Store ○ Key-value store for intermediate aggregation data, KTable materialized views, arbitrary key-value data produced during ○ Replicated externally ● Time windowing
  7. 7. Kafka Streams - high level DSL KStream words =“words”) KTable countsTable = words.flatMapValues(value -> Arrays.asList(value.toLowerCase().split("W+"))) .map((key, value) -> new KeyValue<>(value, value)) .groupByKey(Serdes.String(), Serdes.String()) .count(timeWindows, "WordCounts"); KStream counts = counts.toStream()“counts”)
  8. 8. Kafka Streams stateful processing (default stores) Kafka Cluster Application Kafka Streams In-memory state store Local disk ● Pluggable state store model ● Key-value data store ● Default strategy: ○ In-memory (fast access) ○ Local disk (for fast recovery) ○ Replicated to Kafka (for resilience) ● Tightly integrated with Kafka: state updates are correlated with offset commits changelog
  9. 9. Spring Cloud Stream ● Event-driven microservice framework ● Developer focus on writing business code ● Middleware-agnostic programming model ● Binders: ○ Kafka ○ RabbitMQ ○ AWS Kinesis ○ Google Pub Sub ○ Apache Artemis (community) ● Easy to deploy with Spring Cloud Data Flow
  10. 10. Spring Cloud Stream KStream Processor (since 1.3) countswords Spring Cloud Stream KStream API outputinput Spring Boot Programming model (developer focus) Application model (configuration options, StreamConfig based on Spring Boot properties, KStreamBuilder, KStream binder) Externalized configuration, uberjar construction, health monitoring endpoints
  11. 11. Kafka Streams in the Cloud Application Kafka Streams Application Kafka Streams Docker uberjar Spring Cloud Data Flow
  12. 12. Kafka Streams stateful and stateless deployments Kafka Cluster Application Kafka Streams In-memory state store Local disk ● Changes propagated to changelog topic ● Stored locally for recovery/restart ● Fully stateless deployments require to replay the topic on restart/failover ● State store recovery can be optimized by providing access to stateful deployments changelog
  13. 13. Kafka Streams with Kubernetes StatefulSets Application Kafka Streams Pod Application Kafka Streams Pod Application Kafka Streams Pod volume-word-count-0 word-count-1 word-count-2 volume-word-count-1 volume-word-count-2 word-count-0
  14. 14. Demo time! twitter:@mariusbogoevici Email: