This document discusses Criteo's large Kafka infrastructure in Europe. Some key details: - Criteo uses Kafka to process up to 7 million messages per second (400 billion per day) across about 200 brokers in 13 Kafka clusters across multiple datacenters. - They have developed an in-house C# Kafka client optimized for their high-throughput use case of no key partitioning and no order guarantees. - Criteo monitors lag and message ordering using "watermark" messages containing timestamps that are tracked across partitions to measure stream processing lag. - Data is replicated between clusters for redundancy using custom Kafka Connect connectors that write offsets to the destination.

1. Ricardo Paiva
How Criteo is managing one
of the largest Kafka
Infrastructure in Europe
DevLead @ Criteo

2. 2 •
SRE Data Rivers
Hervé
Rivière
Martin
Richard
Axel
Rovillé
Ricardo
Paiva
Qinghui
Xu

3. Volume, Time and Location

4. 4 •
Criteo is a high performance online
advertising platform.
Our goal is to show the right campaign
to the right user at the right moment.

5. 5 •
How does it work?
$1.12 $0.83 $0.74 $1.03
www
1 2
AD Server
3
Criteo
RTB
567
< 100 ms

6. 6 •
All messages through the same pipeline
HDFS
Streaming
jobs
Paris

7. Show me your numbers

8. 8 •
 Up to 7 millions msgs/s
 (400 billions msgs/day)
 180 TB / day (compressed)
 Around 200 brokers
 Kafka in production for 4 years
Some Figures

9. 9 •
The Infrastructure
2
3
2
Multiple Datacenters
 Bare-metal servers
 Servers managed by Chef
 User applications running on
Mesos or YARN
 13 Kafka clusters

10. 10 •
32 brokers
2 x 12 cores / 1.8 GHz
< 5% cpu
256GB RAM
< 20GB Used memory (230GB cache)
12 disks of 8TB
RAID 10
48 TB total
~ 40% used
10GB NIC
Kafka setup per Datacenter
2.5 million msg/sec
1.1Gbps

11. 11 •
Partition size
• How we define the number of partitions?
• 1GB/partition/hour
• 72h retention
• We try to keep 72GB per partition
• No key, no problem when increasing partitions
• We have topics with more than 1.300 partitions

12. Run, Forest, Run!
15

13. 13 •
• First C# implementation
• Open Source (https://github.com/criteo/kafka-sharp/)
• Built for high-troughput
• Ability to blacklist partitions
• It discards messages if needed
• Our use:
• No Key Partitioning
• No order per partition guarantee
In-house C# Kafka Client

14. 14 •
In-house C# Kafka Client
Broker 1
Broker 2
Broker 3
X
Slow
Application

15. 15 •
Cons
• Costly to mantain
• Difficult to keep up to date
Pros
• Highly customizable
• Optimized for our use case
• Full control during the migration
Trade-off

16. Tic tac, tic tac
20

17. 17 •
• Lag is our main metric for the clients
• We should be able to measure the lag in all conditions:
• No messages sent
• Blacklisted partitions
• New partitions added
• Offline partitions
SLA based on lag

18. 18 •
Watermarks
46
6
6
6
6
5
5
partition 1
partition 2
partition 4
• Special messages sent to each partition.
• They contain a monotonic timestamp.
• They provide a clock for the stream of messages.
• If a message has a timestamp lower than the previous timestamp, it’s late.
2
3
3
2
2
3
2
2
2
1
1
1
3
3
44
4
4
44
5 4
new old

19. 19 •
{
"partition": 36,
"partition_count": 582,
"kafka_topic": "bid_request",
"timestamp": 1547394552,
"region": "par",
"cluster": "local",
"environment": "prod"
}
Watermark message

20. 20 •
Consensus
2
6
4
4
5
4
1
4
2
2
3
1
partition 1
partition 2
partition 4
3
8
5
partition 312345
• Watermarks are not aligned across
partitions.
• (At Criteo) We can have unordered
messages per partition.
• Empty partitions contain only
watermarks.
• partition_time = max(watermark)
• Global time: min(partitions_time)

21. 21 •
Watermark Injector
Consumer
Watermark Tracker
Map<partition, timestamp>
Online
Service
Watermark
Injector
Chronometer

22. Move!
30

23. 23 •
• Producers/Consumers can overload the cluster.
• Overloaded brokers may lead to losing data.
• Pipeline Cluster
• All data, that goes to HDFS
• General Purpose clusters
• Streaming
Local and Stream cluster
Pipeline General purpose
Online
Service
HDFS

24. 24 •
Mesos
• Kafka Connect application running on Mesos.
• Custom connector.
• Writes offsets on the destination.
• Replication inter or cross datacenters.
Replication

25. 25 •
Replication
Replicator
worker 1
Replicator
worker 2

26. 26 •
Watermark problem
P=1
P=2
P=3
P=4
Replicator
worker 1
Replicator
worker 2
P=2
P=4P=3
P=1

27. 27 •
Watermark reinjection
Watermark
Reinjection
New consensus

28. Current challenges
40

29. 29 •
We migrated to Kafka 2
But still using old message format :(
We are upgrading our C# Kafka client
We look forward for new features:
• Idempotent producers
• Transaction
• Headers
Kafka new features

30. 30 •
• Challenges:
• More and more streaming use cases.
• Multiple frameworks: Flink, Kafka Connect, Kafka Stream, Plain
Consumers.
• Clients running on Mesos, Yarn or bare-metal.
• We are working on a framework to help:
• Release
• Schedule
• Scale
• Monitor
• Maintain
Streaming

31. Thank you!
criteo.com/careers