Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Criteo is managing one of the largest Kafka Infrastructure in Europe

946 views

Published on

In Criteo we manage one of the largest Kafka infrastructure in Europe, with more than 7 million msgs/sec. This talk was first presented on the Kafka Meetup Paris, in January of 2019.

Published in: Technology
  • Be the first to comment

How Criteo is managing one of the largest Kafka Infrastructure in Europe

  1. 1. Ricardo Paiva How Criteo is managing one of the largest Kafka Infrastructure in Europe DevLead @ Criteo
  2. 2. 2 • SRE Data Rivers Hervé Rivière Martin Richard Axel Rovillé Ricardo Paiva Qinghui Xu
  3. 3. Volume, Time and Location
  4. 4. 4 • Criteo is a high performance online advertising platform. Our goal is to show the right campaign to the right user at the right moment.
  5. 5. 5 • How does it work? $1.12 $0.83 $0.74 $1.03 www 1 2 AD Server 3 Criteo RTB 567 < 100 ms
  6. 6. 6 • All messages through the same pipeline HDFS Streaming jobs Paris
  7. 7. Show me your numbers
  8. 8. 8 •  Up to 7 millions msgs/s  (400 billions msgs/day)  180 TB / day (compressed)  Around 200 brokers  Kafka in production for 4 years Some Figures
  9. 9. 9 • The Infrastructure 2 3 2 Multiple Datacenters  Bare-metal servers  Servers managed by Chef  User applications running on Mesos or YARN  13 Kafka clusters
  10. 10. 10 • 32 brokers 2 x 12 cores / 1.8 GHz < 5% cpu 256GB RAM < 20GB Used memory (230GB cache) 12 disks of 8TB RAID 10 48 TB total ~ 40% used 10GB NIC Kafka setup per Datacenter 2.5 million msg/sec 1.1Gbps
  11. 11. 11 • Partition size • How we define the number of partitions? • 1GB/partition/hour • 72h retention • We try to keep 72GB per partition • No key, no problem when increasing partitions • We have topics with more than 1.300 partitions
  12. 12. Run, Forest, Run! 15
  13. 13. 13 • • First C# implementation • Open Source (https://github.com/criteo/kafka-sharp/) • Built for high-troughput • Ability to blacklist partitions • It discards messages if needed • Our use: • No Key Partitioning • No order per partition guarantee In-house C# Kafka Client
  14. 14. 14 • In-house C# Kafka Client Broker 1 Broker 2 Broker 3 X Slow Application
  15. 15. 15 • Cons • Costly to mantain • Difficult to keep up to date Pros • Highly customizable • Optimized for our use case • Full control during the migration Trade-off
  16. 16. Tic tac, tic tac 20
  17. 17. 17 • • Lag is our main metric for the clients • We should be able to measure the lag in all conditions: • No messages sent • Blacklisted partitions • New partitions added • Offline partitions SLA based on lag
  18. 18. 18 • Watermarks 46 6 6 6 6 5 5 partition 1 partition 2 partition 4 • Special messages sent to each partition. • They contain a monotonic timestamp. • They provide a clock for the stream of messages. • If a message has a timestamp lower than the previous timestamp, it’s late. 2 3 3 2 2 3 2 2 2 1 1 1 3 3 44 4 4 44 5 4 new old
  19. 19. 19 • { "partition": 36, "partition_count": 582, "kafka_topic": "bid_request", "timestamp": 1547394552, "region": "par", "cluster": "local", "environment": "prod" } Watermark message
  20. 20. 20 • Consensus 2 6 4 4 5 4 1 4 2 2 3 1 partition 1 partition 2 partition 4 3 8 5 partition 312345 • Watermarks are not aligned across partitions. • (At Criteo) We can have unordered messages per partition. • Empty partitions contain only watermarks. • partition_time = max(watermark) • Global time: min(partitions_time)
  21. 21. 21 • Watermark Injector Consumer Watermark Tracker Map<partition, timestamp> Online Service Watermark Injector Chronometer
  22. 22. Move! 30
  23. 23. 23 • • Producers/Consumers can overload the cluster. • Overloaded brokers may lead to losing data. • Pipeline Cluster • All data, that goes to HDFS • General Purpose clusters • Streaming Local and Stream cluster Pipeline General purpose Online Service HDFS
  24. 24. 24 • Mesos • Kafka Connect application running on Mesos. • Custom connector. • Writes offsets on the destination. • Replication inter or cross datacenters. Replication
  25. 25. 25 • Replication Replicator worker 1 Replicator worker 2
  26. 26. 26 • Watermark problem P=1 P=2 P=3 P=4 Replicator worker 1 Replicator worker 2 P=2 P=4P=3 P=1
  27. 27. 27 • Watermark reinjection Watermark Reinjection New consensus
  28. 28. Current challenges 40
  29. 29. 29 • We migrated to Kafka 2 But still using old message format :( We are upgrading our C# Kafka client We look forward for new features: • Idempotent producers • Transaction • Headers Kafka new features
  30. 30. 30 • • Challenges: • More and more streaming use cases. • Multiple frameworks: Flink, Kafka Connect, Kafka Stream, Plain Consumers. • Clients running on Mesos, Yarn or bare-metal. • We are working on a framework to help: • Release • Schedule • Scale • Monitor • Maintain Streaming
  31. 31. Thank you! criteo.com/careers

×