Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices. We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place. In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include: * How to increase dev velocity of event driven style code. * How to optimize working with Kafka in polyglot setting * How to support growing amount of traffic and developers. * How to tackle multiple DCs environment.

1. natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
10 Lessons Learned from using Kafka
with 1000 microservices
Natan Silnitsky
Backend Infra Developer, Wix.com
JAVA
GLOBAL
SUMMIT

2. But first…
a few Kafka terms

3. Kafka
Producer
Kafka Broker
A few
Kafka terms
@NSilnitsky

4. Kafka
Producer
Topic
Partition
Partition
Partition
Kafka Broker
Topic Topic
Partition
Partition
Partition
Partition
Partition
Partition
A few
Kafka terms
@NSilnitsky

5. Topic TopicTopic
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Kafka
Producer
Partition
0 1 2 3 4 5
append-only log
A few
Kafka terms
@NSilnitsky

6. Topic TopicTopic
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
0 1 2 3 4 5
Kafka
Consumers
6 7 8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
1
7
1
8
1
9
2
0
A few
Kafka terms
@NSilnitsky

7. Now,
Here’s how we use it at Wix.

8. >180M registered users (website builders) from 190 countries
5% of all Internet websites run on Wix
3000+ people work at Wix
>500B HTTP Requests / Day
6PB of static content
At Wix
@NSilnitsky

9. Kafka Messages per day
Microservices
(backend) Developers
300M 1075M
500 1,400
100 300*
2017 2020
At Wix

10. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?

11. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?

12. #1 Common Infra
with common
features.What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?

13. Kafka
Consumer
Kafka
Producer
Kafka Broker
Greyhound
Wraps
Kafka
Service A Service B
@NSilnitsky

14. Kafka
Consumer
Kafka
Producer
Kafka Broker
Service A Service B
Abstract
so that it is easy to
change for everyone
Simplify
APIs, with additional
features
Greyhound
Wraps
Kafka

15. Kafka
Consumer
Kafka
Producer
Kafka Broker
Service A Service B
Scala ZIO
+ Java API
Greyhound
Wraps
Kafka
@NSilnitsky

16. Greyhound
Wraps
Kafka
Simple Consumer API
- Boilerplate
@NSilnitsky

17. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

18. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

19. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

20. static void runConsumer() throws InterruptedException {
String topic = "some-topic";
MessageHandler<SomeMessage> handler =
message -> System.out.printf("Record value:%dn", message.messageValue);
GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler);
}
Greyhound
Consumer API
* No explicit commit

21. + Parallel
Consumption!
Greyhound
Wraps
Kafka
Simple Consumer API

22. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

23. static void runConsumer() {
MessageHandler<PurgeSite> handler = ...
GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler)
.withGroup("some-group")
.withMaxParallelism(40);
}
Greyhound
Consumer
@NSilnitsky

24. Kafka Broker
Site
Chat-bot-m
essages
Topic
Greyhound
Consumer
Kafka
Consumer
SCALA ZIO FIBERS
+ QUEUES
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
(THREAD-SAFE)
PARALLEL
CONSUMPTION
@NSilnitsky

25. + Retries!
...what about
Error handling?
Greyhound
Wraps
Kafka
Simple Consumer API
Thread-safe Parallel Consumption

26. static void setupConsumer() {
ConsumeRetryPolicy retryPolicy = ConsumeRetryPolicy.aRetryPolicy(
Retries.fromBackoffs(Duration.ofSeconds(1), Duration.ofMinutes(10), ...));
GreyhoundConsumer renewConsumer = GreyhoundConsumer.aGreyhoundConsumerSpec(
topic,
renewHandler)
.withGroup("some-group")
.withRetry(retryPolicy);
}
Greyhound
Consumer
@NSilnitsky

27. Kafka Broker
renew-sub-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Greyhound Consumer
Kafka Consumer
FAILS TO
READ
@NSilnitsky

28. Kafka Broker
renew-sub-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
renew-sub-topic-retry-0
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
renew-sub-topic-retry-1
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Inspired by Uber
RETRY!
Greyhound Consumer
Kafka Consumer
RETRY
PRODUCER
@NSilnitsky

29. #2 Retry Topics will
cause your cluster to
grow faster. 😐
What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#1 Common Infra

30. Kafka Broker
renew-sub-topic-retry-0
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
renew-sub-topic-retry-1
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Kafka Broker
renew-sub-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
renew-sub-topic-retry-N
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
RETRY!
Inspired by Uber
@NSilnitsky

31. Kafka Broker
renew-sub-topic-retry-0
0 1 2 3 4 5
renew-sub-topic-retry-1
0 1 2 3 4 5
Kafka Broker
renew-sub-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
renew-sub-topic-retry-N
RETRY!
Inspired by Uber
0 2 3 4 51
@NSilnitsky

32. Retries same message on failure
* lag
BLOCKING
POLICY
HANDLER
Kafka Broker
source-control-
update-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Greyhound Consumer
Kafka Consumer
build-log-service

33. BLOCKING
POLICY
HANDLER
@NSilnitsky

34. + Context Propagation
Super
cool
for us
Greyhound
Wraps
Kafka
Simple Consumer API
Thread-safe Parallel Consumption
Scheduled retries & blocking handler

35. CONTEXT
PROPAGATION
Language
User
Type
USER REQUEST METADATA
Sign up
Site-Members
Service
Browser
Geo
...
@NSilnitsky

36. CONTEXT
PROPAGATION
Sign up
Site-Members
Service
Browser
Kafka Broker
Producer
Topic/Partition/Offset
Headers
Key
Value
timestamp

37. CONTEXT
PROPAGATION
Sign up
Site-Members
Service
Browser
Kafka Broker
Producer
Contacts
Service
Consumer

38. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#3 Self-service
tooling and
documentation.
#2 Retry Topics - bigger cluster
#1 Common Infra

39. Our
team

40. Our
team
Self-service
Tools & Docs

41. 1. Kafka CLI scripts (+ flags)
2. Yahoo Kafka Manager (OSS)
3. Confluent Control Centre
4. A control plane of our own
Self-service
Tools

42. Self-service
Tools
@NSilnitsky

43. Self-service
Tools
@NSilnitsky

44. Self-service
Tools

45. Self-service
Tools

46. Self-service
Tools

47. Self-service
Tools

48. Self-service
Tools
@NSilnitsky

49. Self-service
Docs
How do I investigate this lag?
How do I add retries on errors?
1. Github Readme for Greyhound code
2. Internal StackOverflow Q&A
3. Slack bot that answers without you

50. Self-service
Docs

51. Self-service
Docs
@NSilnitsky

52. Self-service
Docs
Discover
topics and
message
structure.
@NSilnitsky

53. protobuf
internal
API site
Self-service
Docs
Discover
topics and
message
structure.
Message
structure
defined in...
And exposed
with...
@NSilnitsky

54. protobuf
internal
API site
Self-service
Docs
Discover
topics and
message
structure.
Or… avro
schema
registry
Message structure
defined in...
And exposed
with...
@NSilnitsky

55. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#4 Testing
needs to be easy.
v
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra

56. Greyhound Testkit
Kafka Broker ZooKeeper
Test Environment
Scala Test {
}
Testkit
included!
@NSilnitsky

57. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow? #5 Async event
driven monitoring
is less trivial.
#4 Easy Testing
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra

58. ALERTS
METRICS
PUBLISHING
Producer
Consumer
Slack
emailKafka Broker
Metrics Server

59. @NSilnitsky

60. #6 Proactive
broker maintenance.
What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#5 Non-trivial Monitoring
#4 Easy Testing
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra

61. ● Add brokers when needed
● Split clusters when needed
● Delete unused topics
● Avoid hard failures
As a rule of thumb, we recommend each broker to have up to 4,000 partitions
and each cluster to have up to 200,000 partitions.
https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
Don’t let brokers break...
@NSilnitsky

62. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#7 Avoid
active-active
ordered produce
(write)
#5 Non-trivial Monitoring
#4 Easy Testing
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra
#6 Proactive broker maintenance

63. Kafka Broker
Producer
Update
my site Browser
service
1
1
DC1
@NSilnitsky

64. Kafka Broker
Update
my siteBrowser
Kafka Broker
Producer
service
1 2
21
DC1 DC2
Producer
@NSilnitsky

65. Kafka Broker
Browser
Kafka Broker
Producer
1 122
2
DC1 DC2
Producer
1
REPLICATION
Out of
Order
@NSilnitsky

66. Kafka Broker
Update
my site Browser
Kafka Broker
Producer
1 2
Producer
DC1 DC2
Best
Practice
@NSilnitsky

67. Kafka Broker
Browser
1 2
Producer
Kafka Broker
1 2
DC1 DC2
REPLICATION
@NSilnitsky

68. Kafka Broker
Browser
1 2
Producer
Kafka Broker
1 2
DC1 DC2
Consumer
REPLICATION
@NSilnitsky

69. Kafka Broker
Browser
1 2
Producer
Kafka Broker
1 2
DC1 DC2
Consumer
REPLICATION
@NSilnitsky

70. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#8 Avoid using
Kafka SDK
directly in nodeJs
#7 Avoid active-active write
#5 Non-trivial Monitoring
#4 Easy Testing
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra
#6 Proactive broker maintenance

71. Greyhound
Kafka Broker
Producer Consumer
Scala
services
NodeJS
services
ConsumerProducer
Single
Event loop
@NSilnitsky

72. Greyhound
Kafka Broker
Producer Consumer
Scala
services
NodeJS
services
ConsumerProducer
Single
Event loop
✘
@NSilnitsky

73. * memory
Greyhound
Kafka Broker
Consumer
Sidecar
(Or REST Proxy)
gRPC
NodeJS
services
Single
Event loop

74. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#9 Consume and
project
#8 Avoid nodeJs SDK
#7 Avoid active-active write
#5 Non-trivial Monitoring
#4 Easy Testing
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra
#6 Proactive broker maintenance

75. MetaSite
Site
installed
Apps?
RPC
Wix
Stores
Wix
Bookings
Wix
Restaurants
Site
version?
Site owner?
The Popular
Flow
RPC
RPC
@NSilnitsky

76. Read +
Writes
Large
MetaSite
Object
Request overload
(~1M RPM requests)
MetaSite
Site
installed
Apps?
RPC
Wix
Bookings
Wix
Restaurants
Site
version?
Site owner?
RPC
RPC
Wix
Stores
@NSilnitsky

77. Read +
Writes
Large
MetaSite
Object
Request overload
(~1M RPM requests)
MetaSite
Site
installed
Apps?
RPC
Wix
Bookings
Wix
Restaurants
Site
version?
Site owner?
RPC
RPC
Wix
Stores
@NSilnitsky

78. Entire
MetaSite
Context
1. Produce
to Kafka
Kafka
Broker
Producer
MetaSite
Site
Updated!

79. Kafka
Broker
Filter
Site installed
Apps Updated!
Installed
Apps
Context
1. Produce
to Kafka
2. Consume
and Project
Producer
MetaSite
Site
Updated!
Consumer
Reverse
lookup
writer

80. Kafka
Broker
Filter
Site installed
Apps Updated!
Installed
Apps
Context
1. Produce
to Kafka
2. Consume
and Project
Producer
MetaSite
Site
Updated!
Consumer
Reverse
lookup
writer
Reverse
lookup
reader
RPC
RPC
RPC
3. Split Read
from Write

81. Kafka messaging is event driven.
It is only relevant to service-service communications,
not for browser-server interactions, where a user is waiting,
right?
@NSilnitsky

82. Kafka messaging is event driven.
It is only relevant to service-service communications,
not for browser-server interactions, where a user is waiting,
right? Wrong.
@NSilnitsky

83. What do you do
when the traffic,
meta-data, and amount of
developers and use cases
grow?
#10 WebSockets
are Kafka’s best
friend
#9 Consume and project
#8 Avoid nodeJs SDK
#7 Avoid active-active write
#5 Non-trivial Monitoring
#4 Easy Testing
#3 Self-service tooling & docs
#2 Retry Topics - bigger cluster
#1 Common Infra
#6 Proactive broker maintenance

84. Completely distributed
and event driven
Kafka
WebSockets
+
=
@NSilnitsky

85. Kafka Broker
Consumer
Browser
Producer
Contacts
Importer
Contacts
Jobs
Web
Sockets
Service
Use Case:
Long-running async
business process
@NSilnitsky

86. Kafka Broker
Subscribe for notifications
ConsumerProducer
Use Case:
Long-running async
business process
Contacts
Importer
Contacts
Jobs
Browser Web
Sockets
Service
@NSilnitsky

87. Kafka Broker
Import
CSVs
ConsumerProducer
Use Case:
Long-running async
business process
Contacts
Importer
Contacts
Jobs
Browser Web
Sockets
Service
@NSilnitsky

88. Web
Sockets
Service
Kafka Broker
Consumer
* distributed
Use Case:
Long-running async
business process
Producer
Contacts
Jobs
Browser
done!
Contacts
Importer

89. So,
What do you do
when the traffic, meta-data,
and amount of developers and
use cases grow?

90. Wix
Created an entire
ecosystem to support large-scale
Kafka-related needs.

91. A Java/Scala high-level SDK for Apache Kafka.
0.1 is out!
github.com/wix/greyhound

92. Thank You
JAVA
GLOBAL
SUMMIT
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil

93. Slides & More
slideshare.net/NatanSilnitsky
medium.com/@natansil
twitter.com/NSilnitsky
natansil.com
JAVA
GLOBAL
SUMMIT

94. Q&A
JAVA
GLOBAL
SUMMIT
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil