This document discusses Kafka deployment at online advertising company Criteo. Some key points: 1. Criteo uses Kafka to process up to 10 million messages per second and 180 TB of data per day across 13 Kafka clusters spanning multiple datacenters. 2. They define partitioning based on retaining 72GB of data per partition over a 72 hour period. This has led to topics with over 1,300 partitions. 3. Criteo developed an in-house C# Kafka client optimized for their use cases of high throughput and ability to blacklist partitions when needed. They are looking to upgrade to support new Kafka features like idempotent producers and transactions. 4. Monitoring lag is a key metric,

1. Kafka deployment @
ScaleOleksandr Nitavskyi
Criteo

2. Volume, Time and Location

3. 3 •
Criteo is a high performance online
advertising platform.
Our goal is to show the right campaign
to the right user at the right moment.

4. 4 •
How does it work?
$1.12 $0.83 $0.74 $1.03
www
1 2
AD Server
3
Criteo
RTB
567
< 100 ms

5. 5 •
All messages through the same pipeline
HDFS
Streaming
jobs
Paris

6. Show me your numbers

7. 7 •
 Up to 10 millions msgs/s
 (500 billions msgs/day)
 180 TB / day (compressed)
 Around 200 brokers
 Kafka in production for 4 years
Some Figures

8. 8 •
The Infrastructure
2
3
2
Multiple Datacenters
 Bare-metal servers
 Servers managed by Chef
 User applications running on
Mesos or YARN
 13 Kafka clusters

9. 9 •
Partition size
• How we define the number of partitions?
• 1GB/partition/hour
• 72h retention
• We try to keep 72GB per partition
• No key, no problem when increasing partitions
• We have topics with more than 1.300 partitions

10. Run, Forest, Run!
15

11. 11 •
• First C# implementation
• Open Source (https://github.com/criteo/kafka-sharp/)
• Built for high-troughput
• Ability to blacklist partitions
• It discards messages if needed
• Our use:
• No Key Partitioning
• No order per partition guarantee
In-house C# Kafka Client

12. 12 •
In-house C# Kafka Client
Broker 1
Broker 2
Broker 3
X
Slow
Application

13. 13 •
Cons
• Costly to mantain
• Difficult to keep up to date
Pros
• Highly customizable
• Optimized for our use case
• Full control during the migration
Trade-off

14. Tic tac, tic tac
20

15. 15 •
• Lag is our main metric for the clients
• We should be able to measure the lag in all conditions:
• No messages sent
• Blacklisted partitions
• New partitions added
• Offline partitions
SLA based on lag

16. 16 •
SLA based on lag

17. 17 •
Watermarks
46
6
6
6
6
5
5
partition 1
partition 2
partition 4
• Special messages sent to each partition.
• They contain a monotonic timestamp.
• They provide a clock for the stream of messages.
• If a message has a timestamp lower than the previous timestamp, it’s late.
2
3
3
2
2
3
2
2
2
1
1
1
3
3
44
4
4
44
5 4
new old

18. 18 •
{
"partition": 36,
"partition_count": 582,
"kafka_topic": "bid_request",
"timestamp": 1547394552,
"region": "par",
"cluster": "local",
"environment": "prod"
}
Watermark message

19. 19 •
Consensus
2
6
4
4
5
4
1
4
2
2
3
1
partition 1
partition 2
partition 4
3
8
5
partition 312345
• Watermarks are not aligned across
partitions.
• (At Criteo) We can have unordered
messages per partition.
• Empty partitions contain only
watermarks.
• partition_time = max(watermark)
• Global time: min(partitions_time)

20. 20 •
Watermark Injector
Consumer
Watermark Tracker
Map<partition, timestamp>
Online
Service
Watermark
Injector
Chronometer

21. Move!
30

22. 22 •
• Producers/Consumers can overload the cluster.
• Overloaded brokers may lead to losing data.
• Front Cluster
• All data, that goes to HDFS
• Application clusters
• Streaming
Front and Application cluster
Front Application
Online
Service
HDFS

23. 23 •
Mesos
• Kafka Connect application running on Mesos.
• Custom connector.
• Writes offsets on the destination.
• Replication inter or cross datacenters.
Replication

24. Stream processing first try

25. Event time vs Processing time

26. 26 •
Replication
Replicator
worker 1
Replicator
worker 2

27. 27 •
Watermark problem
P=1
P=2
P=3
P=4
Replicator
worker 1
Replicator
worker 2
P=2
P=4P=3
P=1

28. 28 •
Watermark reinjection
Watermark
Reinjection
New consensus

29. Stream processing second and last try

30. Current challenges
40

31. display
Problems with Kappa Architecture
clicks 0
0
1
1
2
2
Job state
time

32. More generic problems Kappa and Kafka
partition 0
partition 1
partition 2
partition 3
partition 4

33. 33 •
We are upgrading our C# Kafka client
We look forward for new features:
• Idempotent producers
• Transaction
• Headers
Kafka new features

34. 34 •
• Challenges:
• More and more streaming use cases.
• Multiple frameworks: Flink, Kafka Connect, Kafka Stream, Plain
Consumers.
• Clients running on Mesos, Yarn or bare-metal
• Pulsar and Pravega evaluation
• We are working on a framework to help:
• Release
• Schedule
• Scale
• Monitor
• Maintain
Streaming

35. Thank you!
criteo.com/careers