Event-Driven Microservices with Python and Apache Kafka®, Dave Klein, Senior Developer Advocate, Confluent Meetup link: https://www.meetup.com/dublin-apache-kafka-meetup-by-confluent/events/287396354/

1. with Python and Apache Kafka
Event-driven Microservices
Dave Klein
dklein@confluent.io twitter.com/daveklein

2. What do we mean by Event-driven?
https:/
/martinfowler.com/articles/201701-event-driven.html
https:/
/www.youtube.com/watch?v=STKCRSUsyP0
dklein@confluent.io twitter.com/daveklein

3. What do we mean by Event-driven?
1. Event Noti
fi
cation
2. Event-carried State Transfer
3. Event Sourcing
4. CQRS
https:/
/martinfowler.com/articles/201701-event-driven.html
https:/
/www.youtube.com/watch?v=STKCRSUsyP0
dklein@confluent.io twitter.com/daveklein

4. Kafka
The Diminutive Guide
dklein@confluent.io twitter.com/daveklein

5. Noti
fi
cation
Order Placed
Temperature Read
State
{item:123, price: 29.95,
qty: 2}
{temp:21.5, unit: C,
time: 1606949836369}
Event
dklein@confluent.io twitter.com/daveklein

6. "key": "a1",
"value": {
"eventType": “added-to-cart",
"title": "Kafka Streams in Action”,
"author": "Bill Bejeck",
"price": 44.99
}
dklein@confluent.io twitter.com/daveklein

7. 1 2 3 4 5 6 7 8 9 10 11
Topic (log)
dklein@confluent.io twitter.com/daveklein

8. 1 2 3 4 5 6 7 8 9 10 11
Topic Partitions
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
P1
P2
P3

9. Brokers and Replication
Broker 1 Broker 2 Broker 3

10. 1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
{100}
{200}
{300}
Topic
{101}
{103}
{201} {400}
{301} {401}

11. 1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11
a1:
{100}
b1:
{200}
c1:
{300}
Topic
a1:
{101}
a1:
{103}
b1:
{201}
d1:
{400}
c1:
{301}
d1:
{401}

12. 1 2 3 4 5 6 7 8 9 10 11
Producer
dklein@confluent.io twitter.com/daveklein

13. 1 2 3 4 5 6 7 8 9 10 11
Producer
dklein@confluent.io twitter.com/daveklein

14. 1 2 3 4 5 6 7
8 9 10 11
Producer
Consumer
dklein@confluent.io twitter.com/daveklein

15. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer
dklein@confluent.io twitter.com/daveklein

16. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer
dklein@confluent.io twitter.com/daveklein

17. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer
Committed Offset: 3
dklein@confluent.io twitter.com/daveklein

18. Consumer
1 2 3 4 5 6 7 8 9 10 11
Producer
Committed Offset: 3
dklein@confluent.io twitter.com/daveklein

19. Consumer
1 2 3 4 5 6 7 8 9 10 11
Producer
Committed Offset: 3
dklein@confluent.io twitter.com/daveklein

20. 1 2 3 4 5 6 7 8 9 10 11
Producer
Consumer
Committed Offset: 3
dklein@confluent.io twitter.com/daveklein

21. Schema Registry
Producer
Consumer
1 2 3 4 5 6 7 8 9 10
dklein@confluent.io twitter.com/daveklein

22. Consumer Group A
AppInstance01
AppInstance02

23. AppInstance01
AppInstance02
Consumer Group A
AppInstance03
AppInstance04

24. Consumer Group A
AppInstance01
AppInstance02
Consumer Group B
OtherApp01
OthereApp02

25. Using Apache Kafka with Python
kafka-python
KafkaProducer
KafkaConsumer
https:/
/pypi.org/project/kafka-python
dklein@confluent.io twitter.com/daveklein
KafkaAdminClient

26. Using Apache Kafka with Python
con
fl
uent-kafka-python
Producer
Consumer
AvroProducer
AvroConsumer
AdminClient
https:/
/pypi.org/project/con
fl
uent-kafka
dklein@confluent.io twitter.com/daveklein

27. Back to Event-driven Microservices
dklein@confluent.io twitter.com/daveklein

28. Potential Current State
dklein@confluent.io twitter.com/daveklein

29. dklein@confluent.io twitter.com/daveklein

30. dklein@confluent.io twitter.com/daveklein

31. dklein@confluent.io twitter.com/daveklein

32. How do we get there?
dklein@confluent.io twitter.com/daveklein

33. Find Events
dklein@confluent.io twitter.com/daveklein

34. https:/
/videos.con
fl
uent.io/watch/uw9bRczCiYMoHTcGJYtjau
Event Modeling
dklein@confluent.io twitter.com/daveklein

35. Find Events in Existing Systems
What causes a service to start working?
What info does a service need to do its work?
What action does a service perform?
What information results from that action?
dklein@confluent.io twitter.com/daveklein

36. Next De
fi
ne Your Topics
One topic per event type?
One topic per domain?
Somewhere in-between?
dklein@confluent.io twitter.com/daveklein

37. Don’t forget the schema!
A schema for every event type
Schemas will change.
Think about compatibility
dklein@confluent.io twitter.com/daveklein

38. Demo
Build a random pizza generator
With Python and Kafka
dklein@confluent.io twitter.com/daveklein

39. Pizza Sauce Cheese Meats Veggies
https:/
/github.com/con
fl
uentinc/demo-scene/tree/master/python-microservices
dklein@confluent.io twitter.com/daveklein

40. https:/
/cn
fl
.io/book-bundle
https:/
/developer.con
fl
uent.io/learn-kafka
https:/
/developer.con
fl
uent.io/get-started/python
https:/
/github.com/con
fl
uentinc/demo-scene/tree/master/python-microservices
dklein@confluent.io twitter.com/daveklein