Apache Kafka offers a highly scalable, mission critical infrastructure for distributed messaging and integration. Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or exposing the data via HTTP). Kafka Streams API allows to embed stream processing into any application or microservice to add lightweight but scalable and mission-critical stream processing logic.

1. High Scalable
Streaming
Microservices with
Kafka Streams
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 1

2. Who am I?
Roberto Perez Alcolea
• Mexican
• Streaming Platform @ Target
• Groovy Enthusiast
• roberto@perezalcolea.info
• @rpalcolea
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 2

3. Stream Processing
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 3

4. Paradigms getting input and
producing outputs
• Request/Response
• Batch
• Stream processing
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 4

5. Request/Response
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 5

6. Batch
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 6

7. Stream Processing
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 7

8. Stream Processing
• Applications react to events instantly
• Can handle data volumes that are much larger than other
data processing systems
• Naturally and easily models the continuous and timely
nature of most data
• Decentralizes and decouples the infrastructure
• Asynchronous
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 8

9. Stateful Stream Processing
• Computation maintains contextual state
• State is used to store information derived from the
previously-seen events
• Requires a stream processor that supports state
management
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 9

10. Stream Processing - Hard Parts
• Accurate results
• Partitioning & Scalability
• Fault tolerance
• Time
• Re-processing
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 10

11. Stream Processing - Use cases
• Network monitoring
• Intelligence and surveillance
• Risk management
• E-commerce
• Fraud detection
• Smart order routing
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 11

12. Stream Processing - Semantics
Systems fail! Depending on the action the producer takes to
handle such a failure, you can get different semantics:
• At least once
• At most once
• Exactly once
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 12

13. Apache Kafka
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 13

14. What is Apache Kafka?
• Distributed streaming platform
• Based on an abstraction of a distributed commit log
• Created and open sourced by LinkedIn
• Provides low-latency, high-throughput, fault-tolerant publish
and subscribe pipelines and is able to process streams of
events
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 14

15. Apache Kafka - Concepts
• Run as a cluster on one or more servers that can span
multiple datacenters
• The Kafka cluster stores streams of records in categories
called topics
• Each record consists of a key, a value, and a timestamp
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 15

16. Apache Kafka - Core APIs
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 16

17. Apache Kafka - Topics and logs
• Category or feed name to which records are published
• Multi-subscriber
• Kafka cluster maintains a partitioned log for each topics
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 17

18. Apache Kafka - Topics and logs
• Partition is an ordered, immutable sequence of records
• Records have offsets
• Records are persisted with a retention period
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 18

19. Apache Kafka - Topics and logs
• Distributed over the servers in the Kafka cluster
• Each partition is replicated across a configurable number of
servers for fault tolerance
• Each partition has one server which acts as the "leader" and
zero or more servers which act as "followers"
• MirrorMaker provides geo-replication support for your
clusters
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 19

20. Apache Kafka - Producers
• Publish data to the topics of their choice
• Responsible for choosing which record to assign to which
partition within the topic
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 20

21. Apache Kafka - Consumers
• Has a consumer group
• Records are load balanced over the consumer instances of
the same group
• Partitions are divided across consumer instances
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 21

22. Kafka Streams
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 22

23. What is Kafka Streams?
• Built upon important concepts for stream processing
• Java Library
• Highly scalable, elastic and fault tolerant
• Exactly Once capabilities
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 23

24. What is Kafka Streams?
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 24

25. Concepts
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 25

26. Streams
• Stream: represents unbounded, continuously updating data
set
• Stream Partition: ordered, replayable, and fault-tolerant
sequence of immutable data records
• Stream Application: program that makes use of the Kafka
Streams library.
• Multiple instances
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 26

27. Processor topology
• Defines the computational logic of the data processing that
needs to be performed by a stream processing application
• low-level Processor API or Kafka Streams DSL
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 27

28. KStream
• Record stream abstraction
• Read from/written to external topic or product from other
KStream
• append-only
("alice", 1) --> ("alice", 3)
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 28

29. KTable
• Changelog stream abstraction (snapshot of the latest value
for each key in a stream)
• Each data record represents an update
• Produced from other tables or stream join/aggregation
• Read from external topic
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 29

30. KTable and KStream
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 30

31. State Store
• Fault tolerant Key-value store for stateful operations
• RocksDB or in-memory hash map
• Interactive Queries
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 31

32. Time Windows
• Control how to group records that have the same key for
stateful operations
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 32

33. Important Considerations
• Internal topics
• Need of disk when using RocksDB
• Proper partitioning
• Parallelism
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 33

34. Example
static void main(final String[] args) throws Exception {
Properties streamsConfiguration = KafkaStreamsConfig.getConfig("order-filter-example")
final StreamsBuilder builder = new StreamsBuilder()
KStream<String, String> ordersStream = builder.stream("orders")
KStream<String, String> ordersPerBook = ordersStream.filter({
key, value -> objectMapper.readValue(value, Order).quantity > 5
})
ordersPerBook.to("filtered-orders")
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration)
streams.cleanUp()
streams.start()
}
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 34

35. Demo
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 35

36. Useful links
https://docs.confluent.io/current/streams/index.html
https://kafka.apache.org/documentation/
https://github.com/rpalcolea/gr8confus-2018-presentations
https://github.com/rpalcolea/gr8confus-2018-kafka-streams-
demos
Roberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 36

37. Thank
YouRoberto Perez Alcolea - High Scalable Streaming Microservices with Kafka Streams - GR8ConfUS 2018 37