During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed microservices system at Wix that can easily handle increasing traffic and storage needs with many different use-cases. In this talk I will share these patterns with you, including: * Consume and Project (data decoupling) * End-to-end Events (Kafka+websockets) * In memory KV stores (consume and query with 0-latency) * Events transactions (Exactly Once Delivery)

1. natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
Battle-tested event-driven patterns for
your microservices architecture
Natan Silnitsky
Backend Infra Developer, Wix.com

2. >200M registered users (website builders) from 190 countries
5% of all Internet websites run on Wix
1,500 Microservices
1,500M Kafka messages produced / Day
@NSilnitsky
At Wix
* Event driven uses cases

3. Agenda
Event-driven microservices
Event-driven patterns:
• Consume and Project
• Event-driven from end to end
• 0-latency KV Store
• Events in Transactions

4. @NSilnitsky
Microservices
The Monolith

5. @NSilnitsky * Separately
Monolith to Distributed Microservices System
Microservices

6. @NSilnitsky
Microservices
Decoupled Microservices
Mutable
Contention
Choice

7. @NSilnitsky
Microservices
Data Encapsulation
* flexibility, schema

8. What about microservices
communication?

9. @NSilnitsky
Request-reply
communication
Microservices Communication
Checkout
Service
User
Service
Inventory
Service
Payments
Service
* ecom example

10. @NSilnitsky
Request-reply
communication
Microservices Communication
Request-Reply Model
HTTP
RPC
HTTP RPC
HTTP RPC
Checkout
Service
Payments
Service
Inventory
Service
User
Service

11. @NSilnitsky
Request-reply
communication
* 1 by 1- slow
Microservices Communication
Request-Reply Model
Checkout
Service

12. @NSilnitsky
Request-reply
communication
Microservices Communication
Request-Reply Model
Checkout
Service
Inventory
Service

13. Broker
@NSilnitsky * Reliability
Event-driven
communications
Introduce a message broker in between services
Message
Consumer
Message
Producer

14. @NSilnitsky * Not ineteract directly
Order created
Event-driven
communications
Introduce a message broker in between services
Event-driven model
Message
Consumer
Flow control is moved from the sender
Broker
Message
Producer

15. @NSilnitsky
Order created
Event-driven
communications
Introduce a message broker in between services
Event-driven model
Message
Consumer
Flow control is moved from the sender
to the receiver.
Broker
Message
Producer
* more robust

16. Kafka Broker
Topic
Partition
Partition
Partition
@NSilnitsky * Linear scaling
Introduce a message broker in between services
Distributed log broker
Event-driven
communications
Message
Consumer
Message
Producer

17. @NSilnitsky
Append-only log
Introduce a message broker in between services
Distributed log broker
Event-driven
communications
Kafka Broker
Topic
Partition
Message
Consumer
Message
Producer
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5

18. Order Created Topic
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
@NSilnitsky
* Data retained,
Immutable
Introduce a message broker in between services
Distributed log broker
Event-driven
communications
Message
Consumer
Message
Producer
Kafka Broker

19. @NSilnitsky * eventually consumed
Distributed log broker eventually consume
Event-driven
communications
Order Created Topic
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
Message
Consumer
Message
Producer
Kafka Broker

20. Checkout
Service
Event-driven
communications
Microservices communication & data
Event driven model
User
Service
Inventory
Service
Cart
Service
Payments
Service
@NSilnitsky

21. Checkout
Service
Cart
Service
Inventory
Service
Payments
Service
User
Service
Event-driven
communications
Microservices communication & data
Event driven model
Kafka Broker
Orders
Partition
Partition
Partition
Users
Partition
Partition
Partition
Cart
Partition
Partition
Partition
Inventory
Partition
Partition
Partition

22. Agenda
Event-driven microservices
Event-driven patterns:
• Consume and Project
• Event-driven from end to end
• 0-latency KV Store
• Events in Transactions
* used daily. blog

23. Consume
and Project
01

24. MetaSite
RPC
Wix
Stores
Wix
Bookings
Wix
Restaurants
RPC
RPC
@NSilnitsky
Consume and
Project
For Popular Services

25. MetaSite
RPC
Wix
Stores
Wix
Bookings
Wix
Restaurants
RPC
RPC
@NSilnitsky
Consume and
Project
Site installed Apps?
Site version?
Site owner?

26. MetaSite
RPC
Wix
Stores
Wix
Bookings
Wix
Restaurants
RPC
RPC
@NSilnitsky
Consume and
Project
Request
overload
Site installed Apps?
Site version?
Site owner?
(~1M RPM requests)
Read +
Writes
Large
MetaSite
Object

27. Request
overload
MetaSite
RPC
Wix
Stores
Wix
Bookings
Wix
Restaurants
RPC
RPC
@NSilnitsky
Consume and
Project
Site installed Apps?
Site version?
Site owner?
Read +
Writes
Large
MetaSite
Object
(~1M RPM requests)

28. Kafka Broker
Produce
to Kafka
Producer
MetaSite
Site
Updated!
Consume and
Project
Entire MetaSite
Context

29. Kafka Broker
Filter
Site installed
Apps Updated!
Installed Apps
Context
Produce
to Kafka
Consume
and Project
MetaSite
Consumer
Reverse
lookup
writer
Consume and
Project
Producer

30. Kafka Broker
Installed Apps
Context
Produce
to Kafka
Consume
and Project
MetaSite
Reverse
lookup
writer
Reverse
lookup
reader
RPC
RPC
RPC
Split Read
from Write
Consume and
Project
Producer Consumer

31. Consume and Project
— Produce instead of serve - massive load reduction
— Project - client-query-optimized “Materialized View”
— Read/write split - only scale read-only DB. more resilient

32. Event-driven
from end to end
02

33. Kafka Broker
Consumer
Browser
Producer
Contacts
Importer
Service
Contacts
Jobs
Service
Web
Sockets
Service
@NSilnitsky
Event-driven
from end to end
Use case:
Long-running async business process

34. Subscribe for notifications
Consumer
Producer
Browser
@NSilnitsky
Event-driven
from end to end
Use case:
Long-running async business process
Kafka Broker
Web
Sockets
Service

35. Import CSVs
Consumer
Producer
Browser
@NSilnitsky
Event-driven
from end to end
Use case:
Long-running async business process
Kafka Broker
Web
Sockets
Service

36. Web
Sockets
Service
Consumer
* Distributed
Producer
Browser
done!
Event-driven
from end to end
Use case:
Long-running async business process
Kafka Broker

37. Web
Sockets
Service
Consumer
* Distributed
Producer
Browser
Cross DC!
Event-driven
from end to end
Use case:
Long-running async business process
Kafka Broker
done!

38. Event-Driven from End to End
— No polling - notify the user on any status change
— Scalable - spin up many import job workers
— Replication - cross-DC requests are easy to set up

39. 0-latency
KV Store
03

40. Kafka Broker
Configuration Topic
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
5
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
K1
V1
K2
V1
K1
V2
K3
V1
K4
V1
K5
V1
@NSilnitsky
Producer
Key - value
Consumer
0-latency
KV Store
Compacted Kafka
Topics

41. Kafka Broker
Consumer
KVStoreReader
In-Memory Map
Compacted Topic
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
K1
V1
K2
V1
K1
V2
K3
V1
K4
V1
K5
V1
Key Value
K1 V1
K2 V1
* Startup
0-latency
KV Store
@NSilnitsky
Compacted Kafka
Topics

42. 0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
K1
V1
K2
V1
K1
V2
K3
V1
K4
V1
K5
V1
@NSilnitsky
0-latency
KV Store
Kafka Broker
Consumer
KVStoreReader
In-Memory Map
Key Value
K1 V1 -> V2
K2 V1
K3 V1
Compacted Kafka
Topics
* Finished
Compacted Topic

43. Consumer
Consumer
Bookings
Business
Manager
KVStoreReader
Time Zones
Key Value
Albania UTC+1
Algeria UTC+1
Andorra UTC+1
Time Zones
KVStoreReader
Countries
Key Value
Albania .al
Algeria .dz
Andorra .ad
Countries
@NSilnitsky
0-latency
KV Store
Kafka Broker

44. Producer
Kafka Broker
Consumer
Bookings
Business
Manager
KVStoreReader
Countries
Countries
*New*
South Sudan
KVStoreWriter
Countries
@NSilnitsky
0-latency
KV Store
* Writer producer

45. Kafka Broker
Producer
Bookings
KVStoreWriter
Time Zones
Time Zones
Consumer
KVStoreReader
Time Zones
Key Value
Albania UTC+1
...
South Sudan UTC+3
*New*
Time Zone for
South Sudan
*New* Country
South Sudan
@NSilnitsky
0-latency
KV Store

46. Kafka Broker
Producer
Bookings
KVStoreWriter
Time Zones
Time Zones
Consumer
KVStoreReader
Time Zones
Key Value
Albania UTC+2
Algeria UTC+1
Andorra UTC+1 *New*
Time Zone for
Albania
Consumer
Events
KVStoreReader
Time Zones
DST in-effect for
Albania
@NSilnitsky
0-latency
KV Store

47. 0-latency KV Store
— Loading the compacted topic to memory –
a 0-latency kv store for dynamic configuration
— Compacted topic is still a topic –
other services can consume its update events
— Small Datasets - for large datasets, the topic should be
streamed to a disk-based DB + In-memory Cache

48. Events in
Transactions
04 * idempotent

49. Payment
Completed
Payments
Kafka Broker
Producer
Events in
Transactions
Classic Ecommerce Flow

50. Payments Checkout
Kafka Broker
Producer Consumer Producer
Order (Items)
Completed
Events in
Transactions
Classic Ecommerce Flow
Payment
Completed

51. Payments Checkout
Inventory
Kafka Broker
Producer Consumer Producer Consumer
Delivery
Consumer
Consumer
Invoices
Events in
Transactions
Classic Ecommerce Flow

52. Producer Consumer Producer Consumer
Consumer
Consumer Completed
Orders
Events in
Transactions
Kafka Broker
Classic Ecommerce Flow

53. Producer
Events in
Transactions
Kafka Broker
Enable.idempotence = true
Attaches offset to
message
Classic Ecommerce Flow

54. Idempotent
Producer
Events in
Transactions
Kafka Broker
Classic Ecommerce Flow

55. Events in
Transactions
Kafka Broker
Producer
0 1 2 3 4 5
0 1
duplicate!
1
Payments
Idempotent
Producer

56. Transaction Scope
Consumer Producer Consumer
Consumer
Consumer
Producer
Events in
Transactions
Kafka Broker

57. Events in Transactions
— Events transactions are quite complex –
More boilerplate, performance tuning via transaction size.
— DB updates/3rd party calls are not covered –
You can dedup by using Kafka record offset as version
— Don’t turn on transactions by default -
Use in critical flows like payments processing

59. Wix developers have employed these
event-driven patterns to make their
microservices more decoupled,
resilient and scalable.

60. A Scala/Java high-level SDK for Apache Kafka.
0.1.5 is out!
github.com/wix/greyhound

61. Thank You
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

62. Resources
6 Event Driven Architecture Patterns - Part 1 & Part 2 - by Natan Silnitsky
The Data Dichotomy: Rethinking the Way We Treat Data and Services -
by Ben Stopford
Shared Database - by Chris Richardson
Exactly once delivery - DevOpsDay TLV 2019 - By Natan Silnitsky
@NSilnitsky

63. Slides & More
slideshare.net/NatanSilnitsky
medium.com/@natansil
twitter.com/NSilnitsky
natansil.com

64. Q&A
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil