This document discusses Kafka Streams and stream processing concepts. It defines stream processing as computation over an unbounded, ever-growing data stream. Some key concepts discussed include event time vs processing time, local vs external state, and different types of time windows. Common stream processing patterns like filtering, joins, and maintaining local state are also covered. The document contrasts Kafka and Kafka Streams, and discusses the stream-table duality and how systems that allow transitioning between these views are more powerful.

1. Kafka Streams
Quando streams encontram tabelas

2. Cristiano Altmann
Arquiteto Software
https://www.linkedin.com/in/crisaltmann/

3. Matheus Alagia
Eng. de Computação
https://www.linkedin.com/in/matheusalagia/
https://ubots.com.br

4. “Data models are perhaps the most important
part of developing software, because they have
such a profound effect: not only on how the
software is written, but also on how we think
about the problem that we are solving.”
- Martin Kleppmann
Designing data-intensive applications

5. Microservices
Independent deploy
Low coupling
Horizontal Scalability
Technological choices
Business time
Independent teams
Technological
evolution
Resilient

6. Increasingly we build ecosystem

7. Microservices is really just distributed
systems!

8. The Hardest Part About Microservices:
Your Data

9. Common Patterns
● Shared database
● Database per service
● Event driven approaches

10. The beginning... Shared Database
serviceXserviceY

11. Database per service
serviceXserviceY

12. Event-driven architecture
S2
S3
S1S4
Message Broker

13. What is Streaming Processing?
“Is some kind of computation over a Data Stream. First and foremost, a data
stream is an abstraction representing an unbounded dataset. Unbounded means
infinite and ever growing.”
Kafka: The Definitive Guide

14. Stream processing is a programming paradigm...
Request-Response Batch ProcessingStreaming
Processing
Throughput
Latency

15. Stream-Processing Concepts
Time
Event time
Local state
Log append time
State
Processing time
External state
Time Windows
Slide window
Tumbling window
Hopping Window

16. Stream-Processing Concepts
Time
Event time
Local state
Log append time
State
Processing time
External state
Time Windows
Slide window
Tumbling window
Hopping Window

17. Stream-Processing Concepts
Time
Event time
Local state
Log append time
State
Processing time
External state
Time Windows
Slide window
Tumbling window
Hopping Window

18. Calculation of the time window
Size
Advance interval
How long remains updatable

20. The world always changes, and sometimes we are interested
in the events that caused those changes, whereas other
times we are interested in the current state of the world….
Stream-Table Duality

21. Systems that allow you to transition back and forth between
the two ways of looking at data are more powerful than
systems that support just one.
- Neha Narkhede (Kafka: The definitive guide)

22. Kafka vs Kafka Stream
● Distributed log
● High available
● ⅓ Fortune 500
● APIS:
○ Producer
○ Consumer
○ Connect
○ Streams
● Part of Kafka ecosystem
● Just a lib
● Simple API
● DSL

23. Stream-Processing Design Patterns

24. Single-Event Processing
FILTER

25. Code

26. External Lookup: Stream-Table Join

27. Processing with Local State

28. But sometimes we need state….
Moving the state to a database just push the
problem to another layer.

29. Mantra: Stateless
Service
Client
STATELESS
STATE STORAGE
??
?

31. Stateless is good!!
● Services start instantly
● Thread Safe
● Scaled out linearly
● Not shared state

33. Stream

34. Table

35. Stream

37. Join
● Stream - Stream
● Stream - Table
● Table - Table

38. Stream - Stream (Windowed-join)

39. Stream - Table

43. Stream-Processing Landscape

45. Obrigado!

46. Perguntas?