.

1. Microservices interaction at scale
using Apache Kafka

2. • Software engineer at Upwork
• 3500 logged hours
• ivanursul.com
2

3. Agenda
1.Monolith to Microservices
2.Communication patterns
3.Apache Kafka
4.Kafka in details
3

4. Anti - Agenda
1.Pros/cons of microservices
architecture
2.How to build microservices right
4

5. Monolith to Microservices
Module 1 Module 2 Module 3 Module 4 Module 5
Database layer
Network layer DTO
Business
Model
ER
Model
5

6. Reality
6

7. 7

8. Microservices goals
8

9. Scaling in terms of people
4 engineers 2 engineers 1 engineer
9

10. Scaling in terms of data
10

11. Service independence
11

12. Microservices
communication
12

13. Shared database
13

14. 14

15. Database per service pattern
15

16. Service interfaces
16

17. Services equality
Microservice 1 Microservice 2
2000 rps
KO
2000 rps
200
rps
17

18. External request
Critical path
18

19. Consistency
19

20. Asynchronous messaging
20

21. Two types of data
Data-on-the-
outside
for your
storage
Data-on-the-
inside
for other services
21

22. “Pluggable” architecture
22

23. 23

24. What is Apache Kafka ?
1.Distributed publish-subscribe
messaging system
2.Fast, Scalable, Durable
3.Created at LinkedIn, is now an Apache
project
4.confluent.io 24

25. time
Commit Log
msg
1
msg
4
msg
7
msg
10
msg
13P0
P1
P2
2x
2x
2x
Distributed Commit Log
Distributed, Replicated
Commit Log
msg
2
msg
5
msg
8
msg
11
msg
14
msg
3
msg
6
msg
9
msg
12
msg
15
25

26. Apache
Zookeper
Broker
Broke
r
Broke
r
Produc
er
Consum
er
Kafka Streams
Connec
t
Conne
ct
Send messagesRead messages
Zookeper for configuration
Adapter for
third-party
sources
At least one Broker/Node
Consume, process
and produce messages
Apache Kafka architecture
Kafka Cluster
26

27. Kafka usage patterns
1.Event Sourcing
2.Change Data Capture
3.Kafka Connect
27

28. Event Sourcing
28

29. Microservice 1 Microservice 2
Apache
Kafka
29
Write Request Read Request
Kafka Producer
Microservice 1

30. Command Query Responsibility
Segregation
UI
Write Model Read Model
Write DB Read DB
Event
s

31. Service structure
Microservice group
Write
Service
Event
Handler
Read
Service
2 instances 2 instances 3 instances

32. Benefits
32
1.Separate read and writes
2.Data can be replayed
3.Debugging
32

33. Drawbac
ks
1.Higher learning curve
2.Backward compatibility support
3333

34. Change Data Capture
3434

35. 3535
Database
Cache
Search engine
Request
update/invalidat
e

36. Database
Request
insert
updat
e
delete insert
updat
e
Cache Search Index
WAL
3636

37. 37
Bottled water

38. Apache
Kafka
Microservice infrastructureLegacy monolith
extract changes
38

39. Apache
Kafka
extract changes
39

40. Benefits
40
1.Useful for legacy databases
2.Eliminate dual writes

41. Drawbac
ks
1.Not every DB has support for CDC
2.Database upgrade
41

42. Kafka Connect
42

43. Apache
KafkaKafka
Connect
43

44. Benefits
44
1.Useful for legacy Databases
2.Eliminate dual writes
3.Supported by all storages
4.Custom connectors

45. Drawbac
ks
1.Additional queries
2.Indicator fields
45

46. Demo
Key
Roles:
Order
Service
Customer
Service
Frameworks
46

47. Kaf
ka
Customer
Service
Order
Service
New Order
5 order.payment.processed
4 order.processed
event handler
1 order.created
2 order.created
event handler
47

48. Apache Kafka in details
48

49. Anatomy of topic
Kafka
Topic
P0
P1
P2
msg 0
Topic is
divided
into
multiple
partitions
(P0, P1,
P2)
Each message
has an identifier
called offset
Producer
s write
message
s to the
end of
partitionPartition is a
unit of
parallelism
Consumers
can read from
any offset
Message
can have
key:value
msg 3
msg 1 msg 4 msg 7 msg 8
msg 2 msg 5
msg 6
49

50. Producers
• Publish to topics
• Round-robin
• Murmur 2
• Retry, Batching
• Async
• Broker list
50

51. Kafka Broker
0 1 2 3 4 5 6 7
0 1 2 3
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
0 1 2 3 4 5 6 7
0 1 2 3
0 1 2 3 4 5 0 1 2 3 4 5
Leader
partition
Replica
Each partition has
exactly one
leader
0 1 2 3 4 5 0 1 2 3 4 5
And >= 0 replicas.
They’re called In-
Sync Replicas
All read and
writes are made
using leader
partitionBroker 1 Broker 2 Broker 3
0 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
P
0
P
1
P
2
51

52. Consumers
• Multiple consumers
• List of partitions
• Offset management
• Offset backup to consumer_offsets
52

53. Consumer Groups
P0 P1 P2
Instance 1 Instance 1 Instance 2 Instance 1Instance 2Instance 3Instance 4
consumer group 1 consumer group 2 consumer group 3
Kafka
Cluster
53

54. Historical rewind
P0
Current Offset
Instance 1
Instance 1
Service 1
ms
g 1
msg
2
msg
3
msg
4
msg
5
54

55. Retention period
P0
P1
P2
msg
12
msg
15
msg
11
msg
14
msg
10
msg
13
msg
1
msg
4
msg
7
msg
2
msg
5
msg
8
msg
3
msg
6
msg
9
55

56. Log compaction
key 1 ->
value
key 2 ->
value
key 1 -> new value
56

57. Guarantees
• Messages are appended in the order
they send
• Ordering is guaranteed within partition
• Consumer reads a messages in the
order they are stored
• N - 1 Broker failures57

58. Delivery semantics
• at least once - by default
• at most once - configurable
• exactly once - Kafka Transactions, June
2017
58

59. Worst scenario
Unclean Leader Election
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
P
0
P
1
P
2
0 1 2 3 4 5 6 7
0 1 2 3
P
1
P
2
0 1 2 3 4 5
P
0
0 1 2 3 4 5
Leader
partition
Replica
Broker
1
Broker
2
Broker
3
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
0 1 2 3 4 5 6 7
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 30 1 2 3
Sync
0 1 2 3 4 5
0 1 2 3 4 5 6 7
0 1 2 3
Choose one of
the
unsynchronized
followers
Wait for Broker 1
to recover
59

60. Service Deployment
• No significant problems in producer side
• Consumers may have issues with blue-
green deployment
• Remember to reserve additional
partitions for blue and green stacks
60

61. Monitoring
• Broker, Producer and Consumer has
JMX support out of the box
• Additional option is to write custom
reporters
61

62. Kafka Streams
62

63. Apache Kafka
63

64. • Useful for large services
• Fault tolerant
• Consume, process and produce
messages
• Aggregations
• Stateful processing
• Distributed joins 64

65. Why Kafka ?
• Kafka doesn’t acknowledge messages
• Easy to use
• Configurable
• Super fast for reading tail messages

66. Summary
• Build an event driven backbone
• Add Service interfaces where needed
• Don’t forget about CDC and Kafka
Connect
66

67. www.slideshare.net/KennyBastani/in-the-eventual-consistency-of-succeeding-at-microservi
Do what makes sense
But don’t build microservices without events

68. Demo, Slides
ivanursul.com/microservices-interaction-apache-kafka
68

69. QA
69