This talk is about the Wix ecosystem for event driven architecture on top of Kafka. I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.

1. natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil
How to build 1000 microservices
with Kafka and thrive
Natan Silnitsky
Backend Infra Developer, Wix.com

2. Wix Backend
~270 developers
~1300 micro-services
>10M LoC (Scala ,Java, Node.js)

3. Wix Backend
~270 developers
~1300 micro-services
>10M LoC (Scala ,Java, Node.js)
850M Kafka messages every day

4. Our inter-service communication Story
Or
How Wix moved to event driven architecture

5. till 2014
synchronous
rpc-json + retries
*special cases:
activeMQ
Embedded JMS broker
for the JVM

6. Kafka SDK
synchronous
rpc-json + retries
*special cases:
activeMQ
2015
many
production
operations
issues
till 2014

7. A lot of
boilerplate
and missing
features
Kafka SDK
2015
synchronous
rpc-json + retries
*special cases:
activeMQ
till 2014
many
production
operations
issues

8. Greyhound Make it simple
and abstract
Kafka SDK
2015
synchronous
rpc-json + retries
*special cases:
activeMQ
till 2014
many
production
operations
issues
A lot of
boilerplate
and missing
features

9. Greyhound
Kafka SDK
2015
synchronous
rpc-json + retries
*special cases:
activeMQ
till 2014
many
production
operations
issues
A lot of
boilerplate
and missing
features
2018
Async request-reply
* polyglot

10. Greyhound
Kafka SDK
Build 1,000 microservices and thrive
* Easily change

11. Greyhound
Kafka SDK
Kafka works for us because of its structure

12. Kafka
Producer
Kafka Broker

13. Kafka
Producer
Topic
Partition
Partition
Partition
Kafka Broker
Topic Topic
Partition
Partition
Partition
Partition
Partition
Partition

14. Topic TopicTopic
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Kafka
Producer
Partition
0 1 2 3 4 5
append-only log

15. Topic TopicTopic
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
0 1 2 3 4 5
Kafka
Consumers
6 7 8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
1
7
1
8
1
9
2
0

16. Kafka
Consumer
Kafka
Producer
Kafka Broker
*performance/scale

17. Greyhound
Producer
Greyhound
Consumer
Kafka
Consumer
Kafka
Producer
Kafka Broker
Why we wrapped Kafka

18. Greyhound wraps Kafka
Less
boilerplate
…? - Setup boilerplate

19. KafkaProducer createProducer() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
Return producer;
}
Setup
with Kafka

20. KafkaProducer createProducer(GreyhoundBuilder greyhoundBuilder) {
GreyhoundBufferedProducer unorderedProducer = greyhoundBuilder
.bufferedProducerMaker()
.unordered()
.build();
}
Setup
with Greyhound
* broker

21. KafkaConsumer createConsumer() {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.serializer", "com.foo.customserializer.UserSerializer");
KafkaConsumer<String, User> consumer = new KafkaConsumer<>(props);
Return producer;
}
Setup
with Kafka

22. GreyhoundConsumer createConsumer() {
GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler);
}
Setup
with Greyhound
* Opinionated Types

23. Greyhound wraps Kafka
Less
boilerplate
…? - Setup boilerplate
- Consumer API
boilerplate

24. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

25. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

26. static void runConsumer() throws InterruptedException {
final Consumer<Long, SomeMessage> consumer = createConsumer();
while (true) {
final ConsumerRecords<Long, SomeMessage> consumerRecords =
consumer.poll(1000);
consumerRecords.forEach(record -> {
System.out.printf("Record value:%dn", record.value().messageValue);
});
consumer.commitAsync();
}
}
Kafka
Consumer API

27. static void runConsumer() throws InterruptedException {
String topic = "some-topic";
MessageHandler<SomeMessage> handler =
message -> System.out.printf("Record value:%dn", message.messageValue);
GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler);
}
Greyhound
Consumer API
* No explicit commit

28. Greyhound wraps Kafka
- Setup boilerplate
- Consumer API boilerplate
Extra
features
…?

29. The Kafka consumer is NOT thread-safe.
All network I/O happens in the thread of the application making the call.
It is the responsibility of the user to ensure that multi-threaded access is
properly synchronized.
Un-synchronized access will result in ConcurrentModificationException.
https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

30. The Kafka consumer is NOT thread-safe.
All network I/O happens in the thread of the application making the call.
It is the responsibility of the user to ensure that multi-threaded access is
properly synchronized.
Un-synchronized access will result in ConcurrentModificationException.
https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

31. Greyhound wraps Kafka
So
instead of
a lot of
consumers...
- Setup boilerplate
- Consumer API boilerplate
+ Parallel
Consumption!

32. Kafka Broker
Topic
Greyhound
Consumer
Kafka
Consumer
WORKER THREAD
POOL
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
(THREAD-SAFE)
PARALLEL
CONSUMPTION

33. static void runConsumer() {
MessageHandler<SomeMessage> handler = ...
GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler)
.withGroup("some-group")
.withMaxParallelism(10);
}
Greyhound
Consumer

34. Greyhound wraps Kafka
...what about
Error handling?
- Setup boilerplate
- Consumer API boilerplate
+ Thread-safe Parallel Consumption
+ Retries!

35. static void setupConsumer() {
ConsumeRetryPolicy retryPolicy = ConsumeRetryPolicy.aRetryPolicy(
Retries.fromBackoffs(Duration.ofSeconds(1), Duration.ofMinutes(10)));
GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler)
.withGroup("some-group")
.withRetry(retryPolicy);
}
Greyhound
Consumer

36. Kafka Broker
site-created-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Greyhound Consumer
Kafka Consumer
FAILS TO
READ

37. Kafka Broker
site-created-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
site-created-topic-retry-0
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
site-created-topic-retry-1
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Inspired by Uber
RETRY!
Greyhound Consumer
Kafka Consumer
RETRY
PRODUCER

38. Kafka Broker
site-created-topic-retry-0
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
site-created-topic-retry-1
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Kafka Broker
site-created-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
site-created-topic-retry-N
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
RETRY!
Inspired by Uber

39. Retries same message on failure
* Handle lag
HANDLER
BLOCKING
POLICY
Kafka Broker
site-created-topic
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
Greyhound Consumer
Kafka Consumer

40. * Handle lag
HANDLER
BLOCKING
POLICY

41. Greyhound wraps Kafkaand when Kafka
brokers are
unavailable...
- Setup boilerplate
- Consumer API boilerplate
+ Thread-safe Parallel Consumption
+ Scheduled retries & blocking handler
+ Resilient Producer

42. Greyhound
Resilient
Producer
FAILS TO
PRODUCE
Kafka Broker

43. Retry on failure
Save to disk
H2
Greyhound
Resilient
Producer
Kafka Broker

44. H2H2H2
Retry on failure
H2
Greyhound
Resilient
Producer
Ordered
block until message
is produced
Kafka Broker

45. H2H2H2
Retry on failure
H2
Greyhound
Resilient
Producer
Kafka Broker
Unordered
(better throughput
and resiliency)

46. Greyhound wraps Kafka
- Setup boilerplate
- Consumer API boilerplate
+ Thread-safe Parallel Consumption
+ Scheduled retries & blocking handler
+ Resilient Producer
+ Context Propagation
Super cool
for us

47. CONTEXT
PROPAGATION
USER REQUEST
METADATA
Browser
User
action

48. Producer
Kafka Broker
Topic/Partition/Offset
Headers
Key
Value
timestamp
Browser

49. Kafka Broker
Topic/Partition/Offset
Headers
Key
Value
timestamp
Consumer

50. Greyhound wraps Kafka
- Setup boilerplate
- Consumer API boilerplate
+ Thread-safe Parallel Consumption
+ Scheduled retries & blocking handler
+ Resilient Producer
+ Context propagation
+ Metrics publishing
One last thing…
for now

51. ALERTS
METRICS
PUBLISHING
Producer Consumer
Metrics
Server
Slack
email

52. Kafka
Consumer
Kafka
Producer
Kafka Broker
- Setup boilerplate
- Consumer API boilerplate
+ Thread-safe Parallel Consumption
+ Scheduled retries & blocking handler
+ Resilient Producer
+ Context propagation
+ Metrics publishing

53. Greyhound takes care of
the mechanics of distributed systems
so YOU can focus on writing business logic.

54. OSS
SOON
Testkit
included!

55. Greyhound
Consumer
Greyhound
Producer
WIX CI +
Greyhound new features
/ updates
continuously
updated

56. Kafka Broker
ConsumerProducer
Browser
Service A Service B
IT’S A HYBRID ARCHITECTURE.
REQUEST-RESPONSE
EVENT-DRIVEN
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
REST

57. Kafka Broker
Producer
Browser
Pub/Sub
REST
use case #1
SITE
CREATED!

58. Kafka Broker
Consumer
Producer
Browser
Pub/Sub
REST
use case #1
Consumer
Consumer
SITE
CREATED!

59. Kafka Broker
Consumer + Retries
Job
Scheduler
Guarantee completion
use case #2
Producer
1
32
RENEW
SUBSCRIPTION!

60. Kafka Broker
Long running async business process
use case #3
Browser Web
Sockets
Service
1
Subscribe for notifications

61. Kafka Broker
Consumer
Long running async business process
use case #3
Browser
Producer
REST
2
3
Web
Sockets
Service
4
1
Subscribe for notifications

62. Kafka Broker
Consumer
Long running async business process
use case #3
Browser
Producer
REST
2
3
Web
Sockets
Service
4
6
Completion notification
5

63. Kafka Broker
ConsumerProducer
Service C
THE
COMPACT
OPTION
KVStore
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5

64. 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
DC A
(Kafka Cluster A)
THE
CROSS-DC
REPLICATION
OPTION
DC B
(Kafka Cluster A)

65. DC B
(Kafka Cluster A)
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
DC A
(Kafka Cluster A)
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5
dynamic consumer
COPIES
IN REAL
TIME
X20
RAVEN

66. Kafka Control Plane

67. Wix has embraced
Asynchronous, Event Driven design.
...and has all the required tools in place to support it.

68. Thank You
natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil

69. Slides & More
slideshare.net/NatanSilnitsky
medium.com/@natansil
twitter.com/NSilnitsky

70. Q&A
natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil