Understanding Kafka architecture and event driven workflow

1. Francesco Tisiot - @ftisiot - Developer Advocate
Get to Know
Apache Kafka
with
Jupyter Notebooks

2. @ftisiot | @aiven_io
Why
Jupyter
Notebooks

3. @ftisiot | @aiven_io
Title Text
Markdown Text
Images
Executable Code

4. @ftisiot | @aiven_io
https:/
/aiven.io/blog/teach-yourself-apache-kafka-and-python-with-a-jupyter-notebook

5. @ftisiot | @aiven_io
Why
Apache
Kafka

6. @ftisiot | @aiven_io

7. @ftisiot | @aiven_io

8. @ftisiot | @aiven_io
Event-Driven Applications

9. @ftisiot | @aiven_io
What is an Event

10. @ftisiot | @aiven_io
Why Event Driven?

11. @ftisiot | @aiven_io
What is Kafka
Topic A
Topic B
0
1
2
3
4
0
1
2
3

12. @ftisiot | @aiven_io
Distributed
Brokers
Replication Factor
3 2

13. @ftisiot | @aiven_io
What’s a Kafka Event?
<K, V> <"temp_max", 35.3>
<{
shop: "Mario's Pizza",
phone_line: 3
}
{
orderId: 123456
name: "Andrea Pirlo",
pizzas: [
{
pizza: "Margherita",
additionalToppings: ["Tuna","Onions","Ham"]
},
...
]
}
, >

14. @ftisiot | @aiven_io
Write to Kafka
Producer
Topic A
Hostname + Port
Authentication
Encoding

15. @ftisiot | @aiven_io
Read from Kafka
Hostname + Port
Authentication
Decoding
Topic Name(s)
Consumer
Offset
0
1
2
3

16. @ftisiot | @aiven_io
Pizza
Demo!

17. @ftisiot | @aiven_io
Log Size

18. @ftisiot | @aiven_io
Topic Partitions

19. @ftisiot | @aiven_io

20. @ftisiot | @aiven_io
Selecting the Partition
<K, V>
Same Key Same Partition

21. @ftisiot | @aiven_io
Topic Partitions
2 1
3 2 1
3
Partition 0
Partition 1
Guaranteed per Partition

22. @ftisiot | @aiven_io
Scaling Out
P1
P2
P3
C1
C2

23. @ftisiot | @aiven_io
Partitions
Demo!

24. @ftisiot | @aiven_io
Multiple Applications
C1
C2
Consumer
Group 1
CX
Consumer
Group 2

25. @ftisiot | @aiven_io
Kafka Connect
Kafka Connect Kafka Connect

26. @ftisiot | @aiven_io
References
https:/
/github.com/aiven/kafka-python-fake-data-producer
https:/
/aiven.io
https:/
/github.com/aiven/python-notebooks-for-apache-kafka