This document discusses Criteo's large Kafka infrastructure in Europe. Some key details:
- Criteo uses Kafka to process up to 7 million messages per second (400 billion per day) across about 200 brokers in 13 Kafka clusters across multiple datacenters.
- They have developed an in-house C# Kafka client optimized for their high-throughput use case of no key partitioning and no order guarantees.
- Criteo monitors lag and message ordering using "watermark" messages containing timestamps that are tracked across partitions to measure stream processing lag.
- Data is replicated between clusters for redundancy using custom Kafka Connect connectors that write offsets to the destination.
8. 8 •
Up to 7 millions msgs/s
(400 billions msgs/day)
180 TB / day (compressed)
Around 200 brokers
Kafka in production for 4 years
Some Figures
9. 9 •
The Infrastructure
2
3
2
Multiple Datacenters
Bare-metal servers
Servers managed by Chef
User applications running on
Mesos or YARN
13 Kafka clusters
10. 10 •
32 brokers
2 x 12 cores / 1.8 GHz
< 5% cpu
256GB RAM
< 20GB Used memory (230GB cache)
12 disks of 8TB
RAID 10
48 TB total
~ 40% used
10GB NIC
Kafka setup per Datacenter
2.5 million msg/sec
1.1Gbps
11. 11 •
Partition size
• How we define the number of partitions?
• 1GB/partition/hour
• 72h retention
• We try to keep 72GB per partition
• No key, no problem when increasing partitions
• We have topics with more than 1.300 partitions
13. 13 •
• First C# implementation
• Open Source (https://github.com/criteo/kafka-sharp/)
• Built for high-troughput
• Ability to blacklist partitions
• It discards messages if needed
• Our use:
• No Key Partitioning
• No order per partition guarantee
In-house C# Kafka Client
15. 15 •
Cons
• Costly to mantain
• Difficult to keep up to date
Pros
• Highly customizable
• Optimized for our use case
• Full control during the migration
Trade-off
17. 17 •
• Lag is our main metric for the clients
• We should be able to measure the lag in all conditions:
• No messages sent
• Blacklisted partitions
• New partitions added
• Offline partitions
SLA based on lag
18. 18 •
Watermarks
46
6
6
6
6
5
5
partition 1
partition 2
partition 4
• Special messages sent to each partition.
• They contain a monotonic timestamp.
• They provide a clock for the stream of messages.
• If a message has a timestamp lower than the previous timestamp, it’s late.
2
3
3
2
2
3
2
2
2
1
1
1
3
3
44
4
4
44
5 4
new old
23. 23 •
• Producers/Consumers can overload the cluster.
• Overloaded brokers may lead to losing data.
• Pipeline Cluster
• All data, that goes to HDFS
• General Purpose clusters
• Streaming
Local and Stream cluster
Pipeline General purpose
Online
Service
HDFS
24. 24 •
Mesos
• Kafka Connect application running on Mesos.
• Custom connector.
• Writes offsets on the destination.
• Replication inter or cross datacenters.
Replication
29. 29 •
We migrated to Kafka 2
But still using old message format :(
We are upgrading our C# Kafka client
We look forward for new features:
• Idempotent producers
• Transaction
• Headers
Kafka new features
30. 30 •
• Challenges:
• More and more streaming use cases.
• Multiple frameworks: Flink, Kafka Connect, Kafka Stream, Plain
Consumers.
• Clients running on Mesos, Yarn or bare-metal.
• We are working on a framework to help:
• Release
• Schedule
• Scale
• Monitor
• Maintain
Streaming
Special Slide Instructions:
Customize this slide with your name and title, add an account name or logo to the slide if presenting to a client
Speaker Notes:
Good morning/afternoon I’m excited to be with you today to share how Criteo is evolving.
We have a new vision and are making some exciting changes as a result of the return of our founder JB Rudelle to lead Criteo…